Systems and methods for road sign ground truth construction with knowledge graphs and machine learning

Abstract

The present disclosure relates to machine learning and road sign recognition. A method of road sign classification with a knowledge graph includes detecting and selecting a representation of a sign across multiple frames, outputting a prompt that initiates a request for a classification of the sign representation, classifying one or more images that include the sign, querying a knowledge graph for a plurality of road sign classes that have at least one attribute in common with the sign, and classifying the sign across the multiple frames in response to a confidence level that exceeds a threshold.

Description

Technical Field

[0001] This disclosure relates to machine learning and road sign recognition. Background Technology

[0002] Recognizing and understanding road signs is a crucial feature of Advanced Driver Assistance Systems (ADAS), which are provided in modern vehicles via technologies such as Road Sign Recognition (RSR) or Intelligent Speed ​​Adaptation (ISA). These features may be mandatory depending on the specific guidelines of each country. Recent RSR and ISA solutions heavily utilize machine learning methods and require comprehensive, high-quality road sign annotation datasets as ground truth. To be ready for real-world use, ground truth datasets must be established from test drives around the world. The number of road sign images to be annotated can be enormous, exceeding ten million per year, as is the case with ADAS development. Any representative sample of these images covering a sufficient number of countries and conditions would be of considerable size. Therefore, optimizing the annotation task and minimizing the time spent by the annotator in each session is critical.

[0003] Road and traffic signs vary significantly across multiple countries. While each country's signs follow one of several conventions (e.g., the Vienna Convention, the SADC Convention, the SIECA Convention, and the MUTCD Convention), variations exist. No universal machine classifier is currently available. Training such a universal classifier for signs across all countries requires a sufficiently large dataset, including as many annotated signs as possible. Even understanding all road signs within a single country is extremely challenging for human annotators. Therefore, a steep learning curve and a thorough review process are essential for generating high-quality annotations. Human-machine collaborative annotation methods address efficient and scalable annotation. Summary of the Invention

[0004] According to one embodiment, a system includes a knowledge graph. The system also includes a controller configured to detect and select road signs identified across one or more frames of an image, output a prompt initiating a request for road sign classification, classify the one or more images including road signs, query the knowledge graph to obtain multiple road sign classes having the same attributes as candidate classes for the next classifier, and classify the road signs across the multiple frames.

[0005] According to a second embodiment, a method for road sign detection and classification using knowledge graphs in human-machine collaboration is disclosed. The method includes detecting and selecting sign representations across multiple frames, outputting a prompt initiating a request for classification associated with the sign representation, classifying road signs and visual attributes on one or more images, querying a knowledge graph to obtain multiple road sign classes having at least one identical attribute to the sign, classifying the sign in response to a confidence level exceeding a threshold from the model, and tracking the same signs across the multiple frames.

[0006] According to a third embodiment, a system for classifying road signs using a machine learning model is disclosed. The system includes a display configured to output a user interface and a processor communicating with the display. The processor is programmed to detect and select representations of signs across one or more images using a machine learning model, output a prompt initiating a request for classification associated with the sign representations, classify the one or more images including the signs, obtain multiple road sign classes having at least one identical attribute to the signs, and classify the signs across the one or more images in response to a confidence level exceeding a threshold. Attached Figure Description

[0007] Figure 1 The illustration shows an example annotation system for capturing and annotating image data; Figure 2 The illustration shows a visualization of the road sign itself; Figure 3 The illustration shows an example of aligning the concept of road signs in RSO with the use of other terms; Figure 4 The illustration shows an example image of a crowdsourcing task utilizing a user interface according to an embodiment of the present disclosure; Figure 5 A road sign workflow solution for the machine learning training and annotation process has been made public. Figure 6A The illustrations show fully automated road sign annotations in different countries; Figure 6B The illustration shows annotations for semi-automated road signs in different countries; Figure 6C The illustration depicts a human-initiated semi-automated tagging system; Figure 7 The illustration shows an example of tracking road signs across multiple frames; Figure 8A The diagram illustrates an example of a user interface associated with a knowledge graph; Figure 8B The illustration shows an example of a user interface when road signs are improperly categorized; Figure 8CThe illustration shows an example of a user interface associated with a knowledge graph when a road sign is incorrectly categorized and input is received at the user interface. Detailed Implementation

[0008] This document describes embodiments of the present disclosure. However, it is to be understood that the disclosed embodiments are merely examples, and other embodiments may take various and alternative forms. The figures are not necessarily to scale; some features may be enlarged or minimized to show details of particular components. Therefore, the specific structural and functional details disclosed herein should not be construed as limiting, but merely as a representative basis for teaching those skilled in the art to employ the embodiments in different ways. As will be understood by those skilled in the art, various features illustrated and described with reference to any of the figures may be combined with features illustrated in one or more other figures to produce embodiments not explicitly illustrated or described. The combinations of illustrated features provide representative embodiments of typical applications. However, various combinations and modifications of features consistent with the teachings of this disclosure may be desired for particular applications or implementations.

[0009] A comprehensive, high-quality road sign annotation dataset is crucial for the success of AI-based Road Sign Recognition (RSR) systems. In practice, annotators often face difficulties when learning about traffic systems from different countries; therefore, these tasks are typically time-consuming and produce poor results. The following embodiment describes a system utilizing knowledge graphs and machine learning models to classify various road signs and / or their visual attributes. This system calculates the distance between real signs on an image and their prototype representations in a latent space to provide ranked prototypes that match the real signs in the image, thereby assisting annotators in effectively classifying road signs. Annotators can query the road sign knowledge graph using visual attributes and receive the best-matching candidates suggested by the model. The model uses candidate and real sign image patches from the knowledge graph as input. Such an embodiment utilizing a knowledge graph approach can reduce the sign search space. Furthermore, by leveraging machine learning models, our system can propose generally correct individual candidates, thus significantly reducing the human search workload.

[0010] There are two main challenges in performing road sign annotation tasks. First, there are many road signs to search before finding a match (the US alone has more than 800 federally approved road signs, and more than 10 states have their own state conventions that differ from the federal convention. This makes manually classifying each sign instance against the entire sign palette impractical. One solution is to have machine learning systems limit the number of candidates a human annotator can search from (e.g., limit to 5 signs). The second challenge lies in the fact that different countries follow different road sign conventions. For example, the US follows the MUTCD, while European countries use the Vienna Convention. Some countries use multiple conventions, and some introduce different variations in features such as color, font, and size. No annotator is fully familiar with all road sign systems and may select the wrong road sign, especially when the instance is unclear (e.g., grayscale images, nighttime images, etc.).

[0011] The following embodiments may be intended to reduce the workload of human annotators by providing machine-guided annotations, and to reduce the cognitive load on humans by utilizing a smaller number of road sign candidates in the detection and classification stages, and to apply human-machine collaboration to track the same road signs across frames.

[0012] The system described below can utilize images as input to a machine learning model with label detection. If label detection fails or a label cannot be identified in the image, the system can utilize human annotators to draw bounding boxes around the image. The system can retrieve specific inputs from the annotators to improve the machine learning model and the system. Such inputs can include information about visual attributes (e.g., shape, color, etc.). For example, the system can utilize a knowledge graph to identify incorrect visual attributes about labels for correction. The system can also work by receiving visual attributes if the classifier identifies incorrect label classifications. Therefore, the retrieved input about visual attributes can be used to improve the machine learning model.

[0013] The system described below can use various machine learning models to obtain optimal estimates of signs (detection, category, shape, color, sign text, etc.). Additionally, the system can leverage abductive reasoning to infer road sign candidates using a knowledge graph. Furthermore, the system can dynamically create micro-tasks for human annotators to also utilize machine interaction for annotation (e.g., obtaining missing information, contextual information, validating machine learning predictions, etc.). The same sign objects can be tracked automatically (via machine learning algorithms), manually (via humans), and semi-automatically (via human-computer interaction) across frames.

[0014] The problem can be addressed by combining knowledge graphs and machine learning to assist annotators and accelerate benchmark ground truth annotation. A fundamental assumption can be made that all road signs possess some basic visual features, and the system can navigate through a knowledge graph of these visual features (focusing on country-specific submaps using GPS data associated with such images) to locate candidate signs and support meaning construction by human annotators. The reduction in the search space translates to a reduction in the search effort and time required for annotators (e.g., human annotators) to locate the correct signs. To further reduce the search space, the system can leverage a machine learning model that utilizes one-shot learning to find matching signs, even those not seen in previous training data. Therefore, the system can introduce a Road Sign Ontology (RSO) to represent salient features of road signs, employ a proposed crowdsourcing technique to cross-country and state-level road sign knowledge graphs, and build a machine learning model combined with the knowledge graph to further rank road sign candidates using (normalized) prediction scores.

[0015] Figure 1 An example annotation system 100 for capturing and annotating image data 132 is illustrated. Annotation system 100 includes a server 102 hosting an annotation web application 124 accessible via network 122 by client device 104. Server 102 includes a processor 106 operably connected to storage device 110 and network device 118. Server 102 further includes an image data input source 130 for receiving image data 132. Client device 104 includes a processor 108 operably connected to storage device 112, display device 114, human-machine interface (HMI) control 116, and network device 120. It should be noted that the example annotation system 100 is an example, and other systems 100 can be used. For example, although only one client device 104 is shown, a system 100 including multiple client devices 104 is envisioned. As another possibility, although the example implementation is shown as a web-based application, alternative systems can be implemented as standalone systems or client-server systems with thick client software.

[0016] Each of the processor 106 of server 102 and the processor 108 of client device 104 may include one or more integrated circuits that implement the functionality of a central processing unit (CPU) and / or a graphics processing unit (GPU). In some examples, processors 106, 108 are system-on-a-chip (SoC) integrating the functionality of a CPU and a GPU. The SoC may optionally include other components, such as, for example, storage device 110 and network devices 118 or 120, into a single integrated device. In other examples, the CPU and GPU are connected to each other via peripheral connectivity devices, such as PCI express or another suitable peripheral data connection. In one example, the CPU is a commercially available central processing unit that implements an instruction set, such as one of the x86, ARM, Power, or MIPS instruction set families.

[0017] Regardless of the specific circumstances, during operation, processors 106 and 108 execute stored program instructions retrieved from storage devices 110 and 112, respectively. Accordingly, the stored program instructions include software that controls the operation of processors 106 and 108 to perform the operations described herein. Storage devices 110 and 112 may include both non-volatile memory and volatile memory devices. Non-volatile memory includes solid-state memory, such as NAND flash memory, magnetic and optical storage media, or any other suitable data storage device that retains data when annotation system 100 is deactivated or loses power. Volatile memory includes static and dynamic random access memory (RAM), which stores program instructions and data during operation of annotation system 100.

[0018] The GPU of client device 104 may include hardware and software for displaying at least two-dimensional (2D) and optionally three-dimensional (3D) graphics to the client's display device 114. Display device 114 may include an electronic display screen, projector, printer, or any other suitable device for reproducing the graphics display. In some examples, the processor 108 of client 104 uses the hardware functionality in the GPU to execute software programs to accelerate the performance of machine learning or other computational operations described herein.

[0019] The HMI control 116 of client 104 may include any of a variety of devices that enable client device 104 of annotation system 100 to receive control input from workers or other users. Examples of suitable input devices for receiving human-machine interface input may include keyboards, mice, trackballs, touchscreens, voice input devices, and graphic tablet devices.

[0020] Network devices 118 and 120 may each include any of a variety of devices that enable server 102 and client device 104 to send and / or receive data from external devices via network 122, respectively. Examples of suitable network devices 118 and 120 include network adapters or peripheral interconnect devices that receive data from another computer or external data storage device, which can be useful for receiving large datasets in an efficient manner.

[0021] The annotation web application 124 is an example of a software application executed by server 102. When executed, the annotation web application 124 can use various algorithms to perform aspects of the operations described herein. In one example, as discussed above, the annotation web application 124 may include instructions executable by processor 106 of server 102. Computer-executable instructions can be compiled or interpreted from computer programs created using various programming languages ​​and / or technologies, including, without limitation and individually or in combination, JAVA, C, C++, C#, Visual Basic, JavaScript, Python, Perl, PL / SQL, etc. Generally, processor 106 receives instructions, for example from storage device 110, computer-readable media, etc., and executes these instructions to perform one or more processes, including one or more of the processes described herein. Such instructions and other data can be stored and transmitted using various computer-readable media.

[0022] Web client 126 may be a web browser or other web-based client executed by client device 104. When executed, web client 126 may allow client device 104 to access annotation web application 124 to display the user interface of annotation web application 124. Web client 126 may further provide input received via HMI control 116 to annotation web application 124 on server 102 via network 122.

[0023] In artificial intelligence (AI) or machine learning systems, model-based reasoning refers to inference methods based on the operation of a machine learning model 128 operating on the worldview being analyzed. Generally, the machine learning model 128 is trained to learn a function that provides a precise correlation between input and output values. At runtime, the machine learning engine uses the knowledge encoded in the machine learning model 128 to compare against observed data to derive conclusions such as diagnoses or predictions. An example machine learning system may include the TensorFlow AI engine, available from Alphabet Inc. in Mountain View, California, although other machine learning systems may be used additionally or alternatively. As discussed in detail herein, the annotation web application 124 and the machine learning model 128 can be configured to identify and annotate features of image data 132 for use in efficient and scalable benchmark ground truth generation systems and methods to produce high-precision (pixel-level accuracy) annotations for developing object detection / localization and object tracking.

[0024] Image data source 130 can be a camera, for example, mounted on a moving object such as a car, wall, or utility pole, or mounted in a mobile device configured to capture image data 132. In another example, image data input 132 can be an interface, such as a network device 118 or an interface to storage device 110, for retrieving previously captured image data 132. Image data 132 can be video, such as an image sequence. Each image in image data 132 may be referred to herein as a frame. For privacy reasons, faces and license plates may be blurred from image data 132 for certain annotation tasks.

[0025] The annotation system 100 may also include a knowledge graph that can be utilized. A knowledge graph (KG) can represent a collection of interconnected descriptions of entities—real-world objects, events, situations, or abstract concepts—where the descriptions utilize a formal structure that allows both humans and computers to process them in an efficient and clear manner. A knowledge graph can be a type of database. Entity descriptions can contribute to each other, forming a network where each entity represents a portion of the entity description associated with it.

[0026] Figure 2 The diagram illustrates a visualization of the road sign ontology. This system can leverage the Road Sign Ontology (RSO) and its consistent knowledge graph (described in more detail below) to assist the data annotation process and the training of machine learning models for road sign classification. The RSO seeks salient visual features representing road signs that are discernible through sight or imagination, and is modeled using a Web Ontology language. Figure 2 This can be an example of visualizing the main ontology concepts.

[0027] RSO ontology can represent road sign features that are beneficial to the performance of machine learning algorithms. Furthermore, the ontology should represent concepts at an appropriate level of granularity, allowing annotators to effectively identify road signs and their visual features when reviewing or viewing images. Some key features of road signs can be represented as being included in their shape, color, text, and printed icons. RSO distinguishes between two types of shapes associated with road signs. The most obvious is the shape of the physical sign. For example, in the United States, stop signs have an octagonal shape, yield signs have a downward-pointing triangle shape, and speed limit signs have a rectangular shape. There are 11 different shapes that physical signs for road signs can take. The second type of shape includes the geometry printed on the physical sign. Common printed geometries include arrows, circles, and diagonals. RSO represents 9 different printed shapes.

[0028] RSO can also differentiate between the colors of road signs. Similar to shapes, RSO distinguishes between several different types of colors associated with road signs. Specifically, road signs can have a foreground color, a background color, and a border color. For example, 11 common colors can be enumerated within the body.

[0029] Icons are a special type of shape printed on road signs to depict various objects. Commonly depicted object types include vehicles, people, animals, and a wide variety of traffic infrastructure (such as traffic lights). Given the large number of possible different icons, the RSO defines only a few common categories, including: animals, infrastructure, nature, people, vehicles, and others.

[0030] RSO can also recognize various types of text on road signs. Many road signs include printed text. Stop signs print the word STOP, yield signs print the word YIELD, and speed limit signs include both the word Speed ​​Limit and a number. RSO does not enumerate all possible text that could be printed on a sign, but instead allows annotation of the text on a particular sign using OWL data type properties. While RSO may not define an enumeration for all possible text on a road sign, it does make it possible to categorize text into various types based on the intended meaning or use. Text categories can include, but are not limited to, speed, height, weight, time, name, and quantity. As an example, the text of a speed limit sign could be identified using the speed category, while the text of a sign announcing entry into town would be identified using the name category.

[0031] Road signs can comply with conventions and allow people to detect and identify the meaning of signs with only a brief glance, thus reducing the cognitive load on drivers. Such conventions can define rules and constraints regarding how road signs can be printed and / or displayed, and various types of conventions. Some road sign conventions may include the Vienna Convention (primarily in Europe and China), the MUTCD (Uniform Manual for Traffic Control Facilities) Convention (primarily used in the United States), and the SADC (Southern African Development Community) Convention (primarily used in Africa). Variations of these conventions can be defined and used for more specific geographic areas. For example, each state in the United States may comply with a federal version of the MUTCD, or may define its own state-specific version. Each road sign represented by an RSO can be associated with the convention it complies with.

[0032] The convention can also provide standard images depicting the signs. These standard images are often referred to as prototypes and serve as templates for sign design, construction, and instructions in manuals. Prototypes typically appear in two versions—a full-color version and a grayscale version. RSO allows road signs to link to these prototype images on the web.

[0033] Figure 3 The illustration shows an example of a road sign in a road sign ontology with other lexical concepts in a knowledge graph. Knowledge graphs may be necessary to enable the identification of road signs and different applications. However, manually building knowledge graphs can be both time-consuming and difficult due to a lack of comprehensive domain knowledge from annotators. The following system can work on a two-step system: 1) relying on a crowdsourced (e.g., crowdsourcing) approach to build a large-scale graph with basic properties; and 2) aligning and expanding the graphs to "fine-tune" them to domain-specific data and vocabulary.

[0034] Figure 4An example image of a crowdsourcing task utilizing a user interface according to an embodiment of this disclosure is illustrated. In such a task, the system can create a web-based application for crowdsourcing workers or applications to extract sign features for constructing a road sign knowledge graph. As shown on the user interface, if applicable, the interface can display features associated with a road sign template by querying inputs related to sign shape, background color, border color, additional shapes inside the sign (e.g., a left arrow), icons (e.g., vehicles), text, and variables (e.g., street names). The micro-task can be completed by any crowdsourcing effort and does not require road sign knowledge, such as the meaning of a given road sign template. All answers, except for the text that should be entered into a text field, can be selected from the provided options. Therefore, the system does not specify any qualifications or require any training, but simply provides instructions with examples. The system can model a "gold standard" road sign for which it knows ten road sign templates of a baseline truth value to filter out fraudsters who may intentionally attempt to deceive various micro-tasks. Each individual signature template can be presented to a task, and an internal expert can review the responses from the task, followed by an additional review with another internal expert, if necessary, for further clarification. In this context, a road sign template can refer to a prototype image that can be extracted from a convention document (e.g., MUTCD). A road sign can refer to a real physical sign on an image captured by a camera. Icons can be symbols found in road signs or road sign templates (e.g., people, vehicles, etc.).

[0035] Because the number of localized road signs becomes quite large and requires people to know the different languages ​​used in different locations, the system can leverage a large pool of people recruited from various crowdsourcing marketplaces with globally registered or internal labor forces.

[0036] The system can first work to identify road sign templates for various countries. In the first task, the system can ask the public to find websites that provide official road sign documents containing road sign templates for the target country or state. Then, the system can ask at least three public workers to reach a consensus on recommended sources. After identifying sources, manual template extraction may be required if the identified resources do not support the individual image file format used for each road sign template.

[0037] Verified attributes can be extracted from the tag template and transformed into RDF facts of the corresponding entities in the general knowledge graph. The system can then be refined to produce different domain-specific graphs. To obtain domain-specific knowledge graphs, the system can first create separate graphs for each domain from relevant subgraphs of the general knowledge graph. For example, a subgraph containing all facts about a target country or state. The system can then perform alignments to expand and refine the domain-specific graphs.

[0038] One alignment can include automated reasoning. Since the RSO may conform to OWL-DL, the system can perform semantic reasoning to add more facts, such as adding categories to a sign via its color and shape. Reasoning can also create facts at different granularities. For example, a property might have a foreground color and can be refined using subsumption to have icon color, text color, shape, and sign color, or mapped to a more general property with color.

[0039] Another alignment could include the automatic transformation of individual triples. If the contents of the triples in the general graph are transformable, the system can apply rules to obtain more facts. For example, the text “SPEED LIMIT 30” can be transformed into two triples: “SPEED LIMIT” as text and 30 as a value in a domain-specific graph.

[0040] Another type of alignment can be simply manual alignment. It's also possible to suggest that ontology alignment experts add new vocabulary to the domain-specific graph as needed. For example, category / class names that might be used in the domain are often acronyms that cannot be automated without additional input. Therefore, annotators may need to add manual alignment or input.

[0041] As a result of alignment (or various forms of alignment), a general road sign knowledge graph and multiple domain-specific graphs can be obtained. These graphs can be periodically reviewed by domain experts using other web-based applications (including internal applications). The knowledge graphs can be stored and processed within an enterprise cluster. The system can leverage a database to store intermediate annotations and perform multiple authentications before storing them in the cluster. The system can leverage various applications to transform web application data models into RSO-compliant triples. The system can leverage the built-in semantic reasoning and regular expression capabilities within the enterprise cluster to perform graph alignment.

[0042] The task for human annotators could be to draw bounding boxes around road signs on an image and select matching sign prototypes from a small sign canvas. The system may not expect human annotators to possess knowledge about traffic systems, which often varies by country. Furthermore, simple tasks do not require separate training courses on road signs. Instead, annotators can interact with the system by providing road sign features visible within bounding boxes they draw on the WebUI. Annotators can provide sign shape and background color as public attributes, and icons, text, and additional shapes as optional attributes. Annotators can then be asked to perform visual matching based on previously... K Choosing a flag template from among the candidates is something an average person can do. The task execution time and quality may depend on the search space—that is, the number of candidates they must visually compare—and the image quality.

[0043] The system's tool can support the annotator by providing a small number of road sign candidates that match the attributes given by the knowledge graph search. If the number of sign candidates exceeds a threshold... K Then, machine learning models can be applied to further reduce the number of candidates.

[0044] This system can integrate machine learning models such as single-sample classifiers to predict similarities to cropped image patches containing road signs. K There are 10 road sign template candidates. The input to the model can be (1) a cropped image patch around a bounding box drawn by the annotator on a real road image, and (2) a sign template filtered by a road sign knowledge graph. Both of these inputs can be encoded into a latent space, and nearest neighbor classification can be used to rank the road sign templates. The system can then generate a list of the top 10 road sign templates. K Several candidates are returned to the human annotator. Due to the rarity of road signs in the long tail, predicting unseen classes can be crucial in road sign annotation. Model predictions with regular classifications of known classes can effectively assist human annotators unless the model can be trained on a large dataset that includes rare classes. During the training phase of such a model, the encoder encodes real images into latent distributions, and the decoder reconstructs the encoded distributions back to the prototypes corresponding to the input images. By doing so, the trained encoder can be used as a feature extractor, and the model learns image similarity and prototype concepts, rather than learning the classification itself. Therefore, a pre-trained model can predict novel classes unknown during training time.

[0045] Figure 5The illustration shows an example of a human-machine collaborative road sign detection and classification process to address the use of machine learning training and annotation processes. Input frame 501 can be received from one or more cameras or sensors located on a vehicle collecting data on the road. The received images and videos can be retrieved from the cameras and sent to detector 503. The detector can be automated to utilize various algorithms with recognition capabilities to identify bounding boxes around all notified road signs.

[0046] Given an image, system 500 can first run road sign detector 503, and then a human can verify and correct the location of all signs in the image. Human verification can be done at the first instance or at another different subsequent instance. Detector 503 can provide partial bounding boxes around the various road signs identified in input frame 501. System 500 can then run classifier 507 (such as a naive classifier) ​​on each road sign. Classification can be done by a single classifier 507 or a combination of classifiers 507 to predict partial attributes of the road sign, such as shape, color, and sign text (e.g., speed limit value). If the classification is incorrect (verified by a human), the system can use the incorrect classification to query knowledge graph 509 to obtain the missing attributes of the sign. User 511 can be used to identify whether the sign is correct, or machine learning can be used to automate this process. The system can also query knowledge graph 509 if the confidence level is below a threshold or if the sign is unidentifiable. User 511 can be used to correct attributes identified by system 500 (such as by a knowledge graph or machine learning model). Using these attributes, the system can query Knowledge Graph 513 again to obtain a set of all road sign classes that have the same attributes as the candidate classes of the next classifier.

[0047] The next classifier, 515, could be a one-shot classifier based on an encoder (such as a vibration prototype encoder) that learns a similarity metric to classify images at test time using a single prototype of each road sign within a candidate class. This method is ideal for unseen signs, where data is unavailable but prototype signs are available. Classification accuracy is improved by reducing the number of candidate classes through a knowledge graph. However, there may still be a large number of candidate classes with the same attributes. For example, in the United States, there are more than 50 diamond-shaped signs with a black foreground color and a yellow background color. To improve the one-shot classifier, we introduce metric learning, such as triplet loss, during training to further separate the different classes in the embedding space. The top k outputs from this classifier are finally presented to the crowdwork worker, which significantly reduces his workload compared to existing annotation pipelines.

[0048] Figure 6AThe diagram illustrates automated road sign annotations from different countries. Raw data 601 can be collected in one country (e.g., the UK). However, the machine learning detector 603 can be trained on Chinese signs. Thus, the system may not precisely understand what the sign 605 represents. The detector 603 can look for specific attributes to be identified in the sign, such as the circle shape. Next, the system can identify the corresponding classifier when it is trained on signs from a neighboring country or another European country (e.g., German signs). The system can classify the identified sign with a high confidence level for German bicycle signs, even though the target country is the UK. Accordingly, the system can utilize a knowledge graph to help classify German signs.

[0049] Therefore, as shown, automatic pre-labeling can be performed. Countries adopting the same convention have the same sign representation, but can have different sign classes in an instantaneous system. The system uses inference from a knowledge graph to provide automatic classification for the target country. The example above shows a machine learning model trained on a dataset of German road signs predicting the class of bicycle lane indicator signs in German road signs. The knowledge graph inference engine can then find the corresponding sign class in UK road sign 611.

[0050] Figure 6B The illustration depicts semi-automated road sign annotation in different countries. In such an example, the dataset might be in one country (Ireland), but is utilized on a machine learning model in China. The raw data 620 can be collected in Ireland. The ML detector 621 can be trained on Chinese signs and identify the signs 623. The ML road sign classifier 625 can be used for German signs, as it is closer to Ireland and Europe. The classifier can classify the signs with low confidence and understand that the target country is Ireland. The system can utilize a knowledge graph 627 to identify the signs; however, it may be an incorrect sign 629. Annotators (which can be human annotators) can provide the properties 631 of the signs, including color, shape, content, etc. The knowledge graph can provide sign candidates to choose from based on the annotator's input. Therefore, human annotators can provide the closest guess instead of sifting through hundreds (if not thousands) of possible candidates. Thus, a human annotation task with a knowledge graph can reduce the search space for road sign candidates when one or more machine classifiers are unable to predict the road sign class with high confidence. The system can include a threshold confidence level to provide sign candidates, such as a 40% confidence level. However, any percentage confidence level can be used as a threshold to define whether additional input from the annotator is needed and whether possible candidates should be output. This method requires less cognitive load for human annotators compared to brute-force methods that find the correct road sign class among hundreds of road sign candidates in the target country.

[0051] Figure 6C The illustration depicts a human-initiated semi-automated labeling system. In such an embodiment, road signs not used in machine training can be predicted as the top K candidates. This example shows that if the machine cannot detect a road sign in a given image 650, a human draws geometry (e.g., a bounding box) to indicate the road sign, followed by the road sign's attributes. If a sign cannot be predicted, the system can output a request to draw a bounding box. The system can receive input (e.g., certain pixels) associated with the bounding box around the image. The system can also seek input associated with the sign's properties, such as shape, color, content, etc., requesting it to receive such input. A knowledge graph can be used to find matching road sign candidates and their prototype images (road sign templates) based on the human annotator's input on attributes. The system can output a road sign icon along with an image patch (cropped based on the human annotator's input on the geometry), which is used to generate the top K candidates. K A few-shot learning classifier can be developed. Knowledge graphs can reduce the search space (number of candidates) and provide better predictions in few-shot learning classifiers. Input can be saved for future reference when the annotator confirms the correct image, thus eliminating the need for human input and annotators.

[0052] Figure 7 The illustration shows an example of tracking road signs across multiple frames. Figure 7 As shown, road signs can be explored over time through video recordings. The system can first select the best representation of a sign across frames and ask an annotation agent to classify it. The annotation agent can then move through other frames in which the signal appears and designate those sign instances as the same sign. By doing so, the classification of a sign can be performed only once, and the same ID will be assigned to all instances of that sign across frames. Depending on the performance of the annotation agent, detection (identifying the location of the sign), classification (specifying what kind of sign it is), and tracking (giving the same ID to all sign instances across frames) can be performed by a human annotator, a machine (e.g., a machine learning model or AI), or via a hybrid machine-human approach.

[0053] In the first frame, road sign A is tracked backward or forward across keyframes until no identical sign is found. Both machine tracking algorithms and human annotators can track the same sign. The system can annotate road sign B in another frame. This can be based on the assumption that the machine can detect road sign B in frame X with a higher confidence level than the same sign in frame _k8. Road sign B can be tracked backward or forward across keyframes until no identical sign can be found.

[0054] Figure 8AThe diagram illustrates an example of a user interface associated with a knowledge graph. For example... Figure 8A As shown, the road sign located on image 801 may differ from the road sign template 803 shown as identified and classified by the system. This interface can request input from the user to confirm whether the bounding box 801 matches the template 803 identified according to a machine learning model. Figure 8B The diagram illustrates an example of a user interface when road signs are improperly categorized. As shown, this input can identify that the categorization is incorrect. Accordingly, the system and interface can output several issues. Figure 8C The diagram illustrates an example of a user interface associated with a knowledge graph when a road sign is incorrectly categorized and input is received at the user interface. For example... Figure 8C As shown, the road sign located on image 801 is different from the template 803 shown as identified and categorized by the system. As shown, the system can output questions related to: Does the sign image shown in the attached figure match the sign image categorized in question 1 and identified in template 803? If the received input claims no, additional questions can be added. For example, if the image is grayscale, the system can determine whether the background color of the road sign is "light" or "dark". Question 2 can ask whether the "foreground color" is light or dark. Question 3 can ask whether the "border color" is light or dark. Question 4 can ask whether the border color of the road sign inside the bounding box is dark or light. Question 5 can ask whether the shape of the road sign inside the bounding box is "circle", "diamond", "rectangle", "lower triangle", or "upper triangle". Finally, for question 5, the interface can ask whether the content of the road sign inside the selected bounding box is a car, arrow, bicycle, pedestrian, or other item. The interface can also require the selection of an icon similar to the icon closest to the inside of the bounding box when all questions are answered. This interface can receive input from each questionnaire and use each input from the machine learning model to further help identify the future training and classification of the machine learning system.

[0055] The processes, methods, or algorithms disclosed herein may be deliverable to / implemented by a processing device, controller, or computer, which may include any existing programmable electronic control unit or dedicated electronic control unit. Similarly, processes, methods, or algorithms may be stored in various forms as data and instructions executable by a controller or computer, including but not limited to information permanently stored on non-writable storage media such as ROM devices and information changeably stored on writable storage media such as floppy disks, magnetic tapes, CDs, RAM devices, and other magnetic and optical media. Processes, methods, or algorithms may also be implemented in a software executable object. Alternatively, suitable hardware components (such as application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), state machines, controllers) or other hardware components or devices, or a combination of hardware, software, and firmware components, may be used to embody processes, methods, or algorithms, wholly or partially.

[0056] While exemplary embodiments have been described above, they are not intended to describe all possible forms covered by the claims. The language used in this specification is descriptive rather than limiting, and it should be understood that various changes may be made without departing from the spirit and scope of this disclosure. As previously stated, features of various embodiments may be combined to form further embodiments of the invention that may not be explicitly described or illustrated. While various embodiments may have been described as providing advantages over or preferred over other embodiments or prior art implementations in one or more desired characteristics, those skilled in the art will recognize that one or more features or characteristics may be compromised to achieve desired overall system properties depending on the specific application and implementation. These properties may include, but are not limited to, cost, strength, durability, lifecycle cost, merchantability, appearance, packaging, size, suitability, weight, manufacturability, ease of assembly, etc. Accordingly, any embodiment described as less desirable than other embodiments or prior art implementations in one or more characteristics is not outside the scope of this disclosure and may be desirable for a particular application.

Claims

1. A system comprising: The input interface is configured to receive one or more images; The controller communicates with the input interface and is configured as follows: Detect and select road signs identified across multiple frames from the one or more images; Output a prompt indicating that a request has been made to classify road signs; Utilize machine learning models to detect and classify one or more images, including road signs; When the classification result is incorrect or the confidence level of the classification is lower than the threshold, the attributes of the road sign corrected by the user are obtained, and the knowledge graph is queried to obtain multiple road sign classes with the same attributes as the candidate classes of the next classifier. Classify road signs across the multiple frames; and Track road signs across the multiple frames.

2. The system according to claim 1, wherein, The controller is further configured to identify one or more attributes associated with the road sign and output multiple road sign templates in response to the one or more attributes.

3. The system according to claim 1, wherein, The controller is further configured to classify the road signs in response to input received at the interface.

4. The system according to claim 1, wherein, The controller is further configured to output a first classification and, in response to a misclassification of the first classification, receive reclassification input from the annotator.

5. The system according to claim 1, wherein, The controller is further configured to classify the road signs across the plurality of frames in response to a confidence level exceeding a threshold, and to output a request for attribute input when the confidence level is below the threshold.

6. The system according to claim 1, wherein, The controller is further configured to output a classification associated with the road sign at the user interface.

7. A method for classifying road signs using knowledge graphs, comprising: Representation of detection and selection flags across multiple frames; Output a prompt to initiate a request for a classification associated with the representation of the flag; Classify one or more images including the aforementioned logo; When the classification result is incorrect or the confidence level of the classification is lower than the threshold, the attributes of the user-corrected sign are obtained, and the knowledge graph is queried to obtain multiple road sign classes that have at least one attribute with the sign. and In response to a confidence level exceeding a threshold output by the next classifier, the flags are classified across the multiple frames.

8. The method according to claim 7, wherein, The classification assignment is associated with an identifier that includes the one or more images.

9. The method according to claim 7, wherein, The classification utilizes one or more machine learning models for road sign detection, road sign classification, and road sign property prediction.

10. The method according to claim 7, wherein, The method includes using metric learning to separate different classes in the embedding space.

11. The method according to claim 7, wherein, The method includes outputting a request for attribute input associated with the representation of the flag when the confidence level is below a threshold.

12. The method according to claim 11, wherein, The method includes outputting multiple potential flag templates in response to attribute input.

13. A system for classifying road signs using a machine learning model, comprising: The display is configured to output a user interface; and A processor that communicates with the display, the processor being programmed to: Utilize machine learning models to detect and select representations of symbols across one or more images; A prompt is displayed at the user interface, which initiates a request for a category associated with the representation of the flag; Classify one or more images including the aforementioned logo; When the classification result is incorrect or the confidence level of the classification is below a threshold, the attributes of the user-corrected sign are obtained, and multiple road sign classes associated with candidates including at least one attribute that is the same as the sign are obtained. and In response to a confidence level exceeding a threshold output by the next classifier, the label is classified across the one or more images.

14. The system according to claim 13, wherein, The processor is further programmed to output a request for attribute input associated with the representation of the flag when the confidence level is below a threshold.

15. The system according to claim 14, wherein, The processor is further programmed to output multiple flag templates in response to attribute inputs.

16. The system according to claim 13, wherein, The processor is further programmed to output one or more attributes associated with the flag.

17. The system according to claim 13, wherein, The processor is further programmed to receive the geometry associated with the flag as input.

18. The system according to claim 13, wherein, The processor is further programmed to perform classification using a single-sample classifier or a few-sample classifier.

19. The system according to claim 13, wherein, The processor is further programmed to obtain multiple road sign classes having at least one attribute that is the same as the sign.

Images