Training systems and methods

Abstract

A training system and method identifies pixels in an image that are within a desired uncertainty range and uses them to train a machine learning model. The predicted pixel classification probability values ​​may be normalized. For each classification label, a count of normalized predicted pixel classification probability values ​​that are within the uncertainty range may be obtained. For each classification label, the count may be divided by the total number of normalized predicted pixel classification probability values ​​to generate an associated uncertainty ratio. Pixels with poorly represented classification labels whose uncertainty ratios are within the desired range of uncertainty can be used to train a machine learning model.

Description

[Technical field]

[0001] [CROSS-REFERENCE TO RELATED APPLICATIONS] This application claims the benefit of priority to U.S. Provisional Application No. 63 / 264,409, "Training Systems and Methods" filed on November 22, 2021 (Attorney Docket No. AA649), the entire contents of which are incorporated herein by reference.

[0002] This application relates generally to computer vision, and more particularly to techniques for training machine learning engines onboard autonomous vehicles. [Background technology]

[0003] Machine learning engines generally require training to improve their classification capabilities. Training typically involves running a large dataset through the machine learning engine so that it can learn. For example, a machine learning engine can be trained to recognize images of cats among other images by passing a sufficient number of images of cats to the machine learning engine. Here, the number of images of cats must be substantial. For these reasons, it is desirable to improve the efficiency of training models in machine learning engines.

[0004] The above background art is intended to provide a brief overview of the background of some of the current problems and is not intended to be exhaustive. [Brief description of the drawings]

[0005] Non-limiting and non-exhaustive features of the present invention are described with reference to the following figures, in which like reference numerals refer to like parts in the various figures unless otherwise specified.

[0006] [Figure 1] FIG. 1 is a high-level block diagram of an autonomous vehicle in various contexts of the present invention.

[0007] [Diagram 2] FIG. 2 is a schematic block diagram of components of an embodiment of the system of the present invention.

[0008] [Diagram 3] 1 is a cropped visual representation of an exemplary image of the present invention.

[0009] [Figure 4A-C] 1 is a visual representation of the ground truth image.

[0010] [Diagram 5] FIG. 5 is a visual representation of probabilistic layers of various features of the representations of FIGS. 4A-4C.

[0011] [Figure 6] 2 is a flow chart illustrating the flow and operation of various aspects of the present invention; DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0012] In the following description, numerous specific details are set forth to provide a thorough understanding of various features and processes. However, those skilled in the relevant art will recognize that the techniques described herein can be practiced without one or more of the specific details, or with other methods, components, materials, etc. In other instances, well-known structures, materials, or operations may not be shown or described in detail so as not to obscure certain features.

[0013] References throughout this specification to an "aspect," "arrangement," or "configuration" indicate that particular features, structures, or characteristics are being described. Thus, the appearances of phrases such as "in one aspect," "in one arrangement," "in one configuration," etc. in various places throughout this specification do not necessarily refer to the same aspect, feature, configuration, or arrangement. Moreover, particular features, structures, and / or characteristics being described may be combined in any suitable manner.

[0014] As used in this disclosure and in the claims, terms such as "component", "system", "platform", "layer", "selector", "interface", and the like are intended to refer to a computer-related entity or an entity associated with an operating device having one or more specific functions, which entity may be either hardware, a combination of hardware and software, software, or software in execution. By way of example, a component may be, but is not limited to, a process running on a processor, a processor, an object, an executable file, a thread of execution, a program, and / or a computer. By way of example, but not limitation, both an application running on a server and the server itself may be a component. One or more components may reside within a process and / or thread of execution, and a component may be localized on one computer and / or distributed across two or more computers. Furthermore, a component may execute from various computer-readable media, device-readable storage devices, or machine-readable media having various data structures stored thereon. A component may communicate via local and / or remote processes according to signals that include one or more data packets (e.g., data from another component in a local system, a distributed system, and / or another component that interacts with another system via signals over a network such as the Internet). As another example, a component may be a device with a particular function provided by mechanical parts operated by electrical or electronic circuits, and operated by a software or firmware application executed by a processor, the processor being internal or external to the device, and executing at least a portion of the software or firmware application.As yet another example, a component may be a device that provides a particular functionality without mechanical parts through electronic components, and the electronic components may include an embedded processor that executes software or firmware that provides, at least in part, the functionality of the electronic component.

[0015] As used herein, terms such as "store," "storage," "data store," "data storage," "database," and the like refer to memory components, entities incorporated in memory, or components that make up memory. It should be understood that the memory components described herein can be either volatile memory or non-volatile memory, or can include both volatile and non-volatile memory.

[0016] Moreover, the term "or" refers to an inclusive "or" rather than an exclusive "or." That is, unless otherwise specified or clear from the context, "X employs A or B" is intended to mean any of the natural inclusive permutations. That is, if X employs A, if X employs B, or if X employs both A and B, then "X employs A or B" is satisfied in each of the foregoing cases. Furthermore, the articles "a" and "an," as used in the subject disclosure and claims, should generally be construed to mean "one or more," unless otherwise specified or clear from the context to refer to the singular form.

[0017] The words "exemplary" and / or "demonstrative" as used herein mean to serve as an example, instance, or illustration. For the avoidance of doubt, the subject matter disclosed herein is not limited by the disclosed examples. Further, any aspect or design described herein as "exemplary" and / or "demonstrative" should not necessarily be construed as preferred or advantageous over other aspects or designs, and is not intended to exclude equivalent exemplary structures and techniques known to those skilled in the art. Furthermore, to the extent that terms such as "including," "having," "containing," and the like are used in either the detailed description or the claims, such terms are intended to be similar to the term "comprising" as an open conjunction, without excluding additional or other elements.

[0018] The terms “infer” or “inference” as used herein generally refer to the process of inferring or inferring states of a system, environment, user, and / or intent from a set of observations captured via events and / or data. The captured data and events may include user data, device data, environmental data, data from sensors, application data, implicit data, explicit data, etc. Inference may be used to identify specific contexts or actions or to generate a probability distribution over states of interest given the data and events, for example.

[0019] The disclosed subject matter can be implemented as a method, apparatus, or article of manufacture using standard programming and / or engineering techniques to generate software, firmware, hardware, or any combination thereof that controls a computer to implement the disclosed subject matter. As used herein, the term "article of manufacture" is intended to encompass any computer-readable device, machine-readable device, computer-readable carrier, computer-readable medium, or computer program accessible from a machine-readable medium. For example, computer-readable media include, but are not limited to, magnetic storage devices such as hard disks, floppy disks, magnetic tapes, optical disks (compact disks (CDs), digital video disks (DVDs), Blu-ray Discs (BDs), etc.), smart cards, flash memory devices (e.g., cards, sticks, key drives), virtual devices that emulate storage devices, and / or any combination of the above computer-readable media.

[0020] Generally, program modules include routines, programs, components, data structures, etc. that perform particular tasks or implement particular abstract data types. The disclosed and illustrated embodiments of the present invention may be practiced in distributed computing environments where certain tasks are performed by remote processing devices that are linked through a communications network. In a distributed computing environment, program modules may be located in both local and remote memory storage devices.

[0021] A computing device includes at least a computer-readable storage medium, a machine-readable storage medium, and / or a communication medium. A computer-readable storage medium or a machine-readable storage medium is any storage medium that can be accessed by a computer, and includes both volatile and non-volatile media, removable and non-removable media. By way of example, and not limitation, a computer-readable storage medium or a machine-readable storage medium may be implemented in conjunction with any method or technology for storing information, such as computer-readable or machine-readable instructions, program modules, structured data, unstructured data, or the like.

[0022] A computer readable storage medium includes, but is not limited to, random access memory (RAM), read only memory (ROM), electrically erasable programmable read only memory (EEPROM), flash memory or other memory technology, compact disc read only memory (CD-ROM), digital versatile disc (DVD), Blu-ray disc (BD) or other optical disk storage, magnetic cassette, magnetic tape, magnetic disk storage or other magnetic storage device, solid state drive or other solid state storage device, or other tangible and / or non-transitory media that can be used to store the required information. In this regard, the terms "tangible" or "non-transitory" as applied to storage, memory, or computer readable media in this specification should be understood as qualifiers to exclude only the propagating transitory signal itself, and not to exclude standard storage, memory, or computer readable media that do not merely convey the propagating transitory signal.

[0023] The computer-readable storage medium can be accessed by one or more local or remote computing devices, for example, via an access request, query, or other data retrieval protocol, to perform various operations on the information stored on the medium.

[0024] The system bus as used herein can be any type of bus structure that can be further interconnected to a memory bus (with or without a memory controller), a peripheral bus, and a local bus using various bus architectures available commercially. The database as used herein can include a basic input / output system (BIOS) that can be stored in non-volatile memory such as ROM, EPROM, EEPROM, etc., and the BIOS contains basic routines that help transfer information between elements in the computer, such as during startup. The RAM can also include a high-speed RAM, such as a static RAM, for caching data.

[0025] As used herein, a computer may operate in a networked environment using logical connections via wired and / or wireless communications to one or more remote computers. The remote computers may include workstations, servers, routers, personal computers, portable computers, microprocessor-based entertainment equipment, peer devices, or other common network nodes. The logical connections shown may include wired / wireless connections to larger networks, such as a local area network (LAN) and / or a wide area network (WAN). Such LAN and WAN networking environments are commonplace in offices and companies and facilitate the creation of extensive computer networks, such as a corporate intranet, either of which may connect to a global communications network, such as the Internet.

[0026] When used in a LAN networking environment, the computer can be connected to the LAN through a wired and / or wireless communication network interface or adapter. The adapter can facilitate wired or wireless communication to the LAN, and a wireless access point (AP) can also be located on the LAN for communicating with the adapter in a wireless mode.

[0027] When used in a WAN network environment, the computer may include a modem or may be connected to a communications server on the WAN via other means for establishing communications over the WAN, such as the Internet. The modem may be internal or external, may be a wired or wireless device, and may be connected to the system bus via an input device interface. In a networked environment, the program modules depicted herein in association with the computer or portions thereof may be stored in the remote memory / storage device.

[0028] When used in a LAN or WAN network environment, a computer can access a cloud storage system or other network-based storage system in addition to or instead of an external storage device. Typically, the connection between the computer and the cloud storage system can be established over a LAN or WAN, respectively, via an adapter or modem, etc. Upon connecting the computer to the relevant cloud storage system, the external storage interface, with the aid of the adapter and / or modem, can manage the storage provided by the cloud storage system in the same way as other types of external storage. For example, the external storage interface can be configured to provide access to cloud storage sources as if the cloud storage source were physically connected to the computer.

[0029] The term "processor" as used herein may refer to substantially any computing processing unit or device, including, but not limited to, single-core processors, single-core processors with software multithreaded execution, multi-core processors, multi-core processors with software multithreaded execution, multi-core processors with hardware multithreading technology, vector processors, pipeline processors, parallel platforms, and parallel platforms with distributed shared memory. Additionally, a processor may refer to an integrated circuit, an application specific integrated circuit (ASIC), a digital signal processor (DSP), a field programmable gate array (FPGA), a programmable logic controller (PLC), a complex programmable logic device (CPLD), a state machine, discrete gate or transistor logic, discrete hardware components, or combinations thereof designed to perform the functions described herein. A processor may utilize nanoscale architectures, such as, but not limited to, molecular-based and quantum dot-based transistors, switches, and gates, to optimize space usage and improve the performance of user equipment. A processor may also be implemented as a combination of computing processing units. For example, a processor may be implemented with one or more processors together, tightly coupled, loosely coupled, or remotely located from each other. Multiple processing chips or multiple devices may share the performance of one or more functions described herein; similarly, storage may be performed across multiple devices.

[0030] By way of overview, various configurations are described herein. For ease of explanation, the methods are expressed and described as a series of steps or actions. It should be understood and appreciated that the various arrangements are not limited to the illustrated acts and / or order of acts. For example, actions may occur in various orders and / or simultaneously, and may occur simultaneously with other actions not shown or described herein. Moreover, not all actions shown are required to implement the methods. In addition, the methods may also be represented as a series of interrelated states via a state diagram or events. Furthermore, the methods described below may be stored on an article of manufacture (e.g., a machine-readable storage medium), which facilitates transferring and transferring such methods to a computer.

[0031] With reference to FIG. 1, the system and method of the present invention rely on input data to improve model predictions for relatively rare features. The system and method of the present invention can be applied to any data set, whether collected and stored, artificially generated, or collected and evaluated in real time. For example, data can be collected by a variety of collection means, such as moving vehicles, sensors attached to traffic lights, sensors embedded in roads, sensors attached to drones, etc. Moving vehicles can be operated manually, autonomously, or a combination of both. In some aspects, data collected in real time can be used to improve model predictions while also being used to navigate the vehicle. Sensors include cameras that generate short-range or long-range image data. Data from other sensors, such as LIDAR, radar, ultrasonic sensors, etc., when converted to image data, can be used to generate data suitable for inspection by the system of the present invention. In at least one arrangement, an autonomous vehicle 10 or bot 10 includes a movable body 12 supported by a number of wheels 14. As will be readily appreciated by one of ordinary skill in the art, at least one of the wheels 14 is a caster wheel, which allows the bot 10 to more efficiently traverse various terrains. The body 12 may include at least one sensor 16 capable of receiving data regarding an environment proximate to the bot 10. The at least one sensor 16 may be configured to receive at least optical, infrared, LIDAR, radar, ultrasonic, or other related forms of data regarding the environment surrounding the bot 10. The body 12 may include at least one processor 18 connected to the at least one sensor 16 to receive and process data from the at least one sensor 16 and use the processed data to navigate the bot 10 according to the environment surrounding the bot 10. The at least one processor 18 may include and / or execute a machine learning engine 20 to assist in recognizing, classifying, or categorizing received image data regarding the environment surrounding the bot 10.The sensors 16 can be deployed to provide any kind of data useful for traffic management. These data can also be accessed to improve the predictive capabilities of machine learning models. For example, the sensors 16 can be deployed on traffic poles 2 or embedded in features 23 of the road 25.

[0032] With reference to FIG. 2, a system 60 for improving the predictive capabilities of a machine learning model for relatively rare features can include one or more processors capable of executing instructions to perform the data processing required to create data that can be used to train a machine learning model. The system 60 of the present invention includes at least one processor that executes instructions to receive data, predict the probability of a classification of the data, evaluate the total number of predictions for each classification, filter the predictions, evaluate the number of predictions for each classification in the filtered predictions, select data using a comparison of the evaluations, and train the machine learning model. The system 60 includes, but is not limited to, a machine learning model 53 that receives the data 51 and generates a prediction probability matrix 55. As described herein, the data 51 may include many forms of sensor data collected in multiple ways by any of multiple types of sensors. The machine learning model 53 can include models known to those skilled in the art, and can include, as non-limiting and non-exhaustive examples, clustering, dimensionality reduction, ensemble methods, neural nets (such as convolutional neural network models), and deep learning, transfer learning, reinforcement learning, natural language processing, and word embedding. Many techniques suitable for object detection and recognition will be readily apparent to those skilled in the art. In the art, non-limiting examples include Region-based Convolutional Neural Networks (R-CNN), Faster R-CNN, Faster R-CNN, Region-based Fully Convolutional Networks (R-FCN), Histogram of Oriented Gradients (HOG), Single Shot Detector (SSD), Spatial Pyramid Pooling (SPP-net), and You Only Look Once (YOLO).

[0033] With continued reference to FIG. 2, the machine learning model 53 may include instructions for creating a prediction probability matrix 55. The machine learning model 53 may include a machine learning pipeline in which at least two operations are typically performed in sequence, the first being the extraction of features from the data 51 and the second being classification. Feature extraction may require several convolution, activation, and pooling functions for each layer of the artificial neural network. The convolution function may downsample the image into features using information from neighboring pixels. A prediction layer may then be used to predict the target value. Convolution typically involves sliding a kernel matrix across the image data, processing one or more pixels at a time, and generating a dot product with each cell of the feature matrix to define the extracted feature. Those skilled in the art will appreciate that suitable techniques other than convolution may be used for feature extraction. Activation may be used to introduce nonlinearity into the machine learning model 53. Activation may be performed after one or more convolution stages. Activation may be performed before or after pooling. Activation can be performed using any of a variety of known functions such as Rectified Linear Unit (ReLU), Sigmoid, Tanh, as non-limiting examples. ReLU is a piecewise linear function that outputs a positive value if it is positive and zero otherwise. Pooling can be used to reduce the spatial size of the convolved feature matrix. Pooling may include max pooling or average pooling, as examples. Max pooling reduces the size of the image by downsampling. For example, in a typical convolutional network, the height and width of the image are gradually reduced (downsampling by pooling), which makes it easier for filters in deeper layers to focus on a larger receptive field (context). However, the number of channels / depth (number of filters used) gradually increases, allowing more complex features to be extracted from the image.Downsampling allows a model (such as the U-Net algorithm) to better understand what is in the image, but may lose information about where it is (hence the need for upsampling as well). Pooling functions work similarly to convolutions, but instead of a dot product, the maximum value (or average value) of the image regions overlapped by the kernel is taken. Those skilled in the art should appreciate that suitable architectures other than sliding window convolutional networks may be used, such as, but not limited to, the published U-Net deep learning architecture for semantic segmentation. The U-Net algorithm is a convolutional network architecture for fast and accurate segmentation of images, and outperforms methods such as sliding window convolutional networks. In some arrangements, a U-Net neural network architecture may be used, in which a reduction pass that performs multiple convolutions, ReLU activations, and max pooling actions is used to reduce spatial information while feature information is being generated, and an augmentation pass is used to upsample the spatial and feature information, and finally generate the output prediction.

[0034] With continued reference to FIG. 2, as will be appreciated by those skilled in the art, static machine learning pipelines generally operate on large data sets. The large data sets may be subsampled, annotated, and used to train machine learning models. In at least some configurations, data may be collected from routes that an autonomous vehicle or bot has traveled or will travel. The collected data may be labeled and used to train machine learning modules executed by a processor onboard the bot. In contrast, dynamic machine learning pipelines may require less supervision for training. Rather than using a random subset of the collected data or collecting random data, an algorithmic component may be added to the training pipeline to allow a human operator or machine to carefully select specific examples, reducing the effort. As the machine learning engine begins to build a model of how to make predictions, the machine learning engine may participate in estimating uncertainty, for example by flagging certain cases that are uncertain. In some configurations, for example, in the case of surface detection, a softmax average is computed for each image, or additionally or alternatively for each class of images with a threshold salience or importance to the bot (e.g., curbs), and if the score of any image falls into a range of interest, which covers cases where the model becomes unsure of its prediction, the image may be provided to an annotation entity for annotation. Experts will appreciate that there are companies that provide such data annotation services. The annotated data can be used to train a machine learning engine. As the "annotate and train" cycle is repeated, the error rate of predictions may begin to decrease. Training with examples that the machine learning engine is unsure of, rather than examples that it understands, exponentially decreases the prediction error rate, allowing for optimized (threshold-defined) classification at reduced cost (i.e., with significantly less human or machine oversight).

[0035] With continued reference to FIG. 2, in some configurations, for example, in situations where multiple redundant data sources are available (e.g., when multiple classification models are used, e.g., a neural network-based classification model and a deterministic classification model), a discrepancy-based active learning approach may be implemented. For example, image data from each source where there is a discrepancy in the classification of a particular image in the data may be retained. Images for which recurring discrepancies occur may be annotated and the machine learning engine may be retrained using the annotated data.

[0036] With continued reference to FIG. 2, at least one arrangement may improve the efficiency of training a deep learning model to aid in the navigation of an autonomous vehicle or bot. The training data helps the deep learning model to correctly interpret and classify each image it receives. Each image includes pixel data. In some configurations, the classification may be performed at the pixel level rather than the image level. For example, for a route taken or to be taken by a bot, pixels may be classified as background, drivable, extended drivable (e.g., grass), non-drivable (e.g., poles, people), and discontinuous surface features (DSF) (e.g., curbs). DSF pixels may account for approximately 2% of the set of data images. In some configurations, an active learning approach to training is performed, where the deep learning model is trained using only data that is uncertain up to a predefined uncertainty threshold or that is within a predefined range of uncertainty. Images with sufficiently uncertain pixel data may be sent to an annotation entity for annotation. Additionally or alternatively, annotation may occur in real time when sufficiently uncertain pixels are encountered. In some arrangements, classification may be captured for only a portion of the image, e.g., the portion of the image closest to the bot, as shown in Figure 3. As will be appreciated by those skilled in the art, portion 203 (Figure 3) of image 205 (Figure 3) includes a depiction of space relatively close to the bot standing upright. For example, in some arrangements, classification may be captured for only the nearest two-fifths of each image. Those skilled in the art will appreciate that the percentage of the image for which data is captured may vary based on at least the height of the bot, and possibly other factors, and may be an adjustable parameter, or hyperparameter.

[0037] Continuing with reference to FIG. 2, various criteria can be used to filter the values ​​of the prediction probability matrix 55. For example, the elements 55 of the matrix can include values ​​that are a function of the probability of classification of the data. The elements 55 of the matrix can be filtered to remove negative values, normalize the values, clip the data, etc. Activation functions described herein can be used to remove negative values ​​and normalize the data set. The data can be clipped according to any desired criteria. For example, the range, accuracy, and field of view of the sensor can be used to clip the data to collect data that best suits the desired results. In one example, the relatively rare features may include discontinuous features of a surface. In this case, the sensors collecting the data can be limited to those that collect surface data, and the data can be clipped to exclude data that does not feature a surface or that is more than a preselected distance from the sensor. For example, FIG. 3 illustrates data cropping, where a cropped percentage 201 (FIG. 3) of sensor data 205 (FIG. 3) (e.g., an image from a camera) is cropped because it is closer to the uncropped percentage 203 (FIG. 3) and is therefore considered more reliable than the cropped percentage 201 (FIG. 3). The filtered data is subject to evaluation. For example, a first parameter A can be calculated by summing 57 filtered matrix values ​​for each classification. Similarly, a second parameter B can be calculated by summing 59 filtered matrix values ​​within a preselected range of values ​​for each classification. Depending on the desired outcome, other functions can be applied to the filtered matrix values. Parameters A and B can be used to filter the data 51 to provide the machine learning model with a relatively rare set of data for training 63. For example, the ratio of A to B 61 can be used along with other criteria to determine which data to provide to train the model on relatively rare features. In some configurations, such data can be provided to the machine learning model 53 to rapidly improve the machine learning model 53.In some configurations, the improved models can be refined so that they can be used for navigation.

[0038] Now, referring to Figs. 4A-4C, examples of ground truth images, model predictions, and their comparisons are shown to illustrate the matrix values ​​of an exemplary classification of curb features. Fig. 4A is an actual ground truth classification provided by an annotation vendor for training a machine learning model for a road surface semantic segmentation task. In this example, the red parts of the image represent the undriveable surface, the green parts represent the standard driveable surface, and the grey parts represent the discontinuous features of the surface. Fig. 4B shows the machine learning model prediction for the image of Fig. 4A. Fig. 4C shows the conflict between the true and predicted labels. The blue parts of the image represent correctly classified pixels, and the yellow parts represent incorrectly classified pixels. The machine learning model needs more data on relatively rare features that are misclassified in the image. The system of the present invention can provide relatively rare data for training the model.

[0039] FIG. 5 shows an example of what the filtered images of FIGS. 4A-4C might look like. In this example, the class probability for each vector element is set to 1, and the model predicts the pixel classification with 100% certainty. Metal grid 643 (FIG. 4C) and discontinuous surface features 645 (FIG. 4C), such as non-drivable areas, are shown in the matrix of FIG. 5 with less than 100% classification certainty. With respect to metal grid 643 (FIG. 4C), the system and machine learning model of the present invention determined that metal grid 643 (FIG. 4C) is a non-drivable surface with about 50% average certainty, determined that metal grid 643 (FIG. 4C) is a standard drivable surface with about 30% average certainty, determined that metal grid 643 (FIG. 4C) is an extended drivable surface with about 15% average certainty, and determined that metal grid 643 (FIG. 4C) is a curb with about 25% average certainty. When filtering includes selecting data with matrix values ​​between 0.4 and 0.7, the non-drivable classification of metal grid 643 (FIG. 4C) includes such data and is selected for providing to the machine learning model. With respect to surface discontinuity feature (DSF) 645 (FIG. 4C), the system and machine learning model of the present invention determined that DSF 645 (FIG. 4C) is a standard drivable surface with about 57% average certainty and that DSF 645 (FIG. 4C) is a curb with about 43% average certainty. When filtering includes selecting data with matrix values ​​between 0.4 and 0.7, the standard drivable and curb classification of DSF 645 (FIG. 4C) includes such data and is selected for providing to the machine learning model.

[0040] Now referring to FIG. 6, in at least one configuration, a machine learning model of an autonomous vehicle or bot can estimate uncertainty regarding the classification of an image in image data related to an environment proximate to the bot and / or uncertainty regarding the classification of at least one pixel in such an image. The surface detection model can be implemented using a semantic segmentation engine that can classify each pixel of the camera image into various classifications including background, bot standard drivable, bot augmented drivable, bot non-drivable, and DSF. The semantic segmentation model can be a U-Net model. The model learns a reduced representation of the image through at least one downsampling pass through the neural network while simultaneously preserving local information about desired properties through at least one upsampling pass with skip connections through the neural network. The upsampling pass can be used to make predictions of image / pixel classifications for unclassified images provided to the model.

[0041] Continuing with reference to Figure 6, various performance metrics of a segmentation engine can be used, including but not limited to, Intersection over Union (IoU) and Jaccard Index. The IoU score can take into account false positives and / or false negatives in each classification, facilitating accurate reporting of imbalanced classes. Class imbalance occurs when the total number of images in one class of data is much smaller than the total number of images in another class of data. A limitation of the IoU metric is that it does not report the accuracy of classification boundaries, but it is still a commonly used performance metric.

[0042] With further reference to FIG. 6, the camera image (short-range camera image as a non-limiting example) may include the road surface. A large amount of road image data may result in classification such that the drivable area occupies about 80% of the image and the DSF occupies about 2% of the image, which may result in an inherent bias in the machine learning classifier and poor classification of the minority class (i.e., DSF). In other words, the machine learning model can better recognize and accurately classify drivable image data than DSF image data because the machine learning model recognizes far more drivable pixels than DSF pixels. The non-drivable area and the extended drivable area may each occupy about 10% of the image. The performance of the machine learning model on the less represented classes (e.g., DSF) may be improved by focusing data collection on obtaining more samples of the less represented classes and fine-tuning the model training on this data. For example, poorly represented features such as DSFs (e.g., curbs) may not be classified as reliably as other features that are relatively prevalent in the image, so poorly represented features (as a non-limiting example, DSFs) may be provided to a representation learning model that learns representations for the data without having an a priori list of features to work from.

[0043] With continued reference to FIG. 6, we now describe a technique for evaluating the confidence of classification predictions and, based on such evaluation, preserving under-represented data that is required to train a machine learning model to improve its predictive capabilities. This technique may be useful for interpreting the confidence of machine learning models in many applications. Employing this technique to increase confidence may be particularly important in data-driven active learning environments where the goal is to achieve a certain threshold level of accuracy with minimal labeling effort. In such settings, machine learning models can learn to select the most informative unlabeled samples for annotation based on the techniques of the present invention. Predictions with very high uncertainty are considered more informative for improving the performance of the machine learning model. Thus, the system of the present invention can be designed to capture data that the machine learning model cannot understand (i.e., cannot interpret or classify correctly) and train the machine learning model only on the captured data. Such a system may seek to find the level of uncertainty in the segmentation model's predictions and determine the features in the input that influenced the predictions.

[0044] With continued reference to FIG. 6, as will be appreciated by those skilled in the art, a deep learning model is a type of machine learning model based on artificial neural networks. When deploying a deep learning model, it may be beneficial to determine whether high-level features are learned by the model, whether the model is legitimately adequate, and whether the model is able to capture “expected features.” In traditional machine learning models, in contrast to deep learning models, a feature may be understood to be a specification of an attribute and the value of that attribute. For example, “color” is an attribute and “color is blue” is a feature. As will be readily appreciated by those skilled in the art, there are various ways to transform a set of attributes without changing the set of features. Non-limiting examples include regrouping attribute values, converting multi-valued attributes to binary attributes, etc. Traditional machine learning models typically rely on feature engineering in the modeling pre-processing stage. For computer vision tasks, e.g., detecting buildings in an image, features considered in traditional machine learning include at least, but are not limited to, pixel color (e.g., RGB), histogram of oriented gradients (HOG), scale-invariant feature transform (SIFT), bag-of-visual-words (BOV), textons (e.g., corner detector, edge detector), and / or pixel values ​​transformed by some dimensionality reduction method. A skilled practitioner would know how to derive a semantic representation for a given feature. Deep learning typically employs a different approach, called representation learning. A skilled practitioner would understand that representation learning involves reasoning on data, defining an architecture for a model that learns to represent the data. In contrast to predefining features that describe the data, “raw” or minimally preprocessed data is provided to the deep learning model. The deep learning model is expected to learn representations for data that are not known a priori and may not be explicitly associated with semantic concepts.

[0045] With continued reference to FIG. 6, in at least one configuration, the system can perform the techniques of the present invention using a short-range U-Net segmentation model operating on raw unlabeled images. The system can also store depth visualization images generated from the raw one-channel depth images. In some configurations, the system can be scripted using a Python computer program. Those skilled in the art will appreciate that other suitable computer programs can also be used. A configuration file can be used to compile the input settings required for instructions implementing the system to run. The configuration file can include paths to unlabeled RGB and depth images, checkpoints for the machine learning model (e.g., weighting files to be applied to various data), directory paths where results are saved, and hyperparameters for tuning the system. In a configuration in which a Python computer program is used, a Python file is created that executes the system's instructions on the raw images, separates the raw images into preferred and normal images, and calculates depth visualizations for the preferred and normal images. The depth visualizations may be calculated in parallel using multiple cores or all cores of a central processing unit (CPU). Other files can include architecture scripts for the machine learning model, helper functions, etc.

[0046] With continued reference to FIG. 6, in at least one configuration, the system runs in real time and can predict the classification uncertainty for each frame in the camera image, as follows: The raw RGB images are fed as input to the trained U-Net model, and the model generates unnormalized predicted probabilities (called "logits") of the classification predictions. Such unnormalized values ​​may be positive or negative and may be of unlimited magnitude. The activation function, for example, but not limited to, binary, linear, sigmoid, tanh, reLU, softplus, softsign, softmax, swish, etc., are well known in the art and define the output given the input and move the neuron / node to the next layer of the neural network. Since the unnormalized values ​​may include negative numbers, applying an activation function to the values ​​can convert all the values ​​to non-negative numbers. Another activation function can normalize the predicted probabilities so that each predicted probability takes a value between 0 and 1. Other activation functions can be applied to the data depending on the desired format of the matrix values. The U-Net classification output may be in the form of channels x height x width, where a channel may be one of five values ​​(or classification labels), for example, but not limited to, background, non-drivable (e.g., obstacles), standard drivable, extended drivable (e.g., grass), and DSF (e.g., curb). A value is assigned to each semantic label, resulting in a five-channel matrix, or NumPy array. As the skilled artisan will appreciate, NumPy is a library of array processing routines used in Python computer programs. The probability assigned to each classification label may be based, at least in part, on an analysis of the dataset provided to the machine learning model. In some arrangements, a portion of the softmax NumPy array may be cropped to focus only on the uncropped portion of a given image. Cropping may be performed for a variety of reasons. For example, in one arrangement, the portion cropped may be 3 / 5 of the softmax NumPy array representing the region furthest from the bot, and the focus is on the remaining 2 / 5 of the array (closest to the bot).Such clipping can help remove outlier predictions that occur based on data obtained beyond a preselected distance from the short-range camera. The clipping percentage depends on the desired prediction accuracy at a particular distance from the camera and need not be limited to a particular value. The clipping percentage can be a tunable value or a hyperparameter. The model predictions that are considered uncertain can be configured in a linear space (an evenly spaced sequence in the interval) defined by start=0.4, end=0.7, and interval=0.05 for predictions per channel. As the skilled artisan will appreciate, linspace is a function in the NumPy library. The normalized predicted probability value for each pixel is between 0 and 1 and can take values ​​in 0.05 intervals (i.e., 0, 0.05, 0.1, 0.15, ..., 0.95, 1). For example, but not limited to, pixels with normalized predicted probabilities between the lower and upper bounds (which may include the endpoints) are considered sufficiently uncertain. For example, the lower bound can be 0.4 and the upper bound can be 0.7. The range of predicted probabilities selected need not be limited to the 0.4-0.7 window, nor need it include either or both endpoints. The range of predicted probabilities selected may depend at least in part on biases in the machine learning model, such that the model produces systematically biased results. The range of predicted probabilities may be an adjustable range, or a hyperparameter. Probabilities beyond the range of predicted probabilities may not have enough uncertainty, and probabilities below the range of predicted probabilities may leave a larger probability range than desired for other (e.g., more representative) classification labels. The model iterates through each pixel in the selected portion of the image and obtains the maximum probability ("argmax") of the pixel. If the maximum probability (i.e., the argmax prediction) is within the range of predicted probabilities, the channel index (representing the predicted class label) and predicted probability value are saved. This process is repeated for each pixel in the image. For all pixels with predicted probability values ​​within the predicted probability range, pixel counts are performed for all classification labels.The pixel count for each classification label is the number of pixels in the selected image that belong to that classification label. For each of the five classification labels, an uncertainty ratio is calculated by dividing the pixel count for that classification label by the total number of pixels with that classification label. Images with associated uncertainty ratios and classification label counts are filtered (i.e., sifted) according to predefined hyperparameters and saved. As a non-limiting example, images with an uncertainty ratio for the DSF classification label greater than 0.4 and a total number of pixels classified as DSF greater than 1500 may be saved. Filtering according to the metrics allows the selection of data that can be used to train machine learning models on high-priority unsuccessful cases. Height and width may refer to the size of the image in the U-Net algorithm.

[0047] 6, stored data that falls within a predefined uncertainty ratio range or exceeds a threshold uncertainty ratio value may then be used to train a machine learning model. Additionally or alternatively, such stored data may be used by the machine learning model to make inferences about images it receives.

[0048] With continued reference to FIG. 6, in at least one configuration, the system of the present invention may perform according to the operations illustrated in FIG. 6. Although the following description and FIG. 6 refer specifically to images, the system of the present invention may process any data. The system is not limited to processing image data. With reference to FIG. 6, a processor with a receiver receives a raw RGB image and passes the image to a machine learning segmentation model running in the processor at action 100. Control flow proceeds to action 102. At action 102, a portion of the image is selected for further processing. In some configurations, the portion selected for further processing may include a portion of the image closest to the receiver. In some arrangements, the selected portion may match a proportion of the image, such as, but not limited to, 2 / 5 of the image. Control flow proceeds to action 104, where the model generates a set of predicted probability values ​​for each pixel in the selected portion, and in some arrangements, a softmax function is applied to generate an array of normalized predicted probability values ​​(i.e., a set of predicted probability values ​​each having a value between 0 and 1), and in some arrangements, the array is a 3D matrix with dimensions of channel, height, and width, where the channel can take any of five classification labels: background, non-drivable, standard drivable, extended drivable, and DSF, and each of the five classification labels is assigned a predicted probability for each pixel in the array, and the array may in some arrangements be a NumPy array. Control flow proceeds to action 106, where the model iterates through each pixel in the selected portion of the image and obtains the maximum probability for each pixel (e.g., using an argmax function). Control flow proceeds to action 108, where it is determined whether the maximum probability for each pixel is within a preselected uncertainty range (e.g., 0.4 to 0.7) and counts the number of pixels that fall within that range for each classification. Control flow proceeds to action 110. In action 110, for each classification, an uncertainty ratio is calculated by dividing the count of each classification label within a preselected range by the total number of pixels of that classification in the cropped image.Control flow proceeds to action 112 to train the machine learning model using images that meet preselected criteria based at least on the uncertainty ratio. The criteria include, for example, a classification with a preselected threshold for the uncertainty ratio (e.g., 0.4) or, for example, a total number of pixels greater than another preselected threshold (e.g., 1500). The criteria threshold is adjustable.

[0049] In at least one configuration, the depth visualization may include a 2D to 3D conversion to facilitate annotation, such as when different terrain features in an image are the same color (e.g., when a road and a sidewalk are both red). In an arrangement, the depth image may be a one-channel image that includes depth information for each RGB pixel. The machine learning model takes as input the depth image and the camera position (e.g., left, center, right), converts the depth pixels of the depth image from millimeters to meters if the pixel depth value is less than 7,000 mm, and back-projects the depth pixel values ​​from 2D to 3D x, y, z coordinates (z represents depth) of each pixel in the camera frame using certain camera-specific parameters. In one arrangement, the back-projected pixels in the camera coordinate frame may be passed to a transformation function to convert the back-projected pixels to a base link coordinate frame of the autonomous vehicle or bot. As a non-limiting example, the base link coordinate frame may be a reference frame projected from between the rear wheels of the autonomous vehicle or bot. The transformation function takes the rotation transformation and the camera transformation and uses them to generate a depth value z for the base link frame.

[0050] Those skilled in the art will appreciate that information and signals may be represented using any of a variety of existing technologies. For example, data, instructions, commands, information, signals, bits, symbols, or chips that may be referred to in the specification may be represented by voltages, currents, electromagnetic waves, magnetic fields or particles, optical fields or light particles, ultrasound, projected capacitance, or any combination thereof.

[0051] Those skilled in the art will also appreciate that the various exemplary logic blocks, modules, circuits, and method steps described in connection with the configurations disclosed herein can be implemented as electronic hardware, computer software, or a combination of both. To clearly illustrate this interchangeability of hardware and software, the various exemplary components, blocks, modules, circuits, and procedures have been described in terms of their functionality. Whether such functionality is implemented as hardware or software depends on the particular application and design constraints imposed on the overall system. Skilled artisans may implement the described functionality in various ways for each particular application, but such determined implementations should not be interpreted as departing from the scope of the appended claims.

[0052] The various example logic blocks, modules, and circuits described in connection with the configurations disclosed herein may be implemented or performed using a general purpose processor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic, discrete hardware components, or any combination thereof designed to perform the functions described herein. A general purpose processor may be a microprocessor, but alternatively, the processor may be any conventional processor, controller, microcontroller, or state machine. A processor may also be implemented as a combination of computing devices, such as a combination of a DSP and a microprocessor, multiple microprocessors, one or more microprocessors in combination with a DSP core, or other similar configurations.

[0053] The operations of the methods described in relation to the configurations disclosed herein may be embodied directly in hardware, in software modules executed by a processor, or in a combination of the two. The software modules may be stored in RAM memory, flash memory, ROM memory, EPROM memory, EEPROM memory, registers, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art. The storage medium is coupled to the processor such that the processor can read information from the storage medium and write information to the storage medium. Alternatively, the storage medium may be integrated into the processor. The processor and the storage medium may reside in an ASIC. The ASIC may reside in a functional device such as, for example, a computer, a robot, a user terminal, a mobile phone or tablet, an automobile, an IP camera, etc. Alternatively, the processor and the storage medium may reside as separate components within such functional device.

[0054] The above description is not exhaustive and is not intended to limit the features to the precise forms disclosed. Those skilled in the art may devise various alternatives and modifications without departing from the disclosure, and the general principles defined herein may be applied to other aspects without departing from the spirit or scope of the appended claims. The disclosure is therefore intended to embrace all such alternatives, modifications, and variations. Moreover, while several configurations of the disclosure have been shown in the drawings and / or described herein, the disclosure is not intended to be limited thereto, as the disclosure is intended to be made as broad as the art will permit, and the specification is intended to be interpreted accordingly. Thus, the above description should not be construed as limiting, but merely as examples of specific configurations. Those skilled in the art will also envision other modifications within the scope and spirit of the claims appended hereto. Other elements, steps, actions, methods, and techniques that are not substantially different from those described above and / or in the appended claims are also intended to be within the scope of the disclosure. Thus, the appended claims are not intended to be limited to the configurations shown and described herein, but are to be accorded the widest scope consistent with the principles and novel features disclosed herein.

[0055] The arrangements shown in the drawings are presented only to illustrate certain examples of the disclosure. Also, the drawings described are merely illustrative and not limiting. In the drawings, the size of some elements may be exaggerated and not drawn to a particular scale for illustrative purposes. Furthermore, elements shown in the drawings with the same number may be identical or similar elements, depending on the context.

[0056] When the term "comprises" is used in the present specification and claims, it does not exclude other elements or steps. When an indefinite or definite article is used to refer to a singular noun (e.g., "a", "an", "the"), the plural of that noun is also included unless otherwise specified. Thus, the term "comprises" should not be interpreted as being limited to the items listed therein, and does not exclude other elements or steps, and thus the scope of the expression "a device including items A and B" should not be limited to a device consisting of only components A and B. Furthermore, to the extent that terms such as "comprise", "have", "possess", etc. are used in the present specification and claims, such terms are intended to be as inclusive as the term "comprises", and "comprises" is to be interpreted as if it were used as a transitional term in the claims.

[0057] Moreover, terms such as "first," "second," "third," etc., whether in the description or claims, are provided to distinguish between similar elements and do not necessarily describe a sequential or chronological order. It will be understood that terms so used are, under appropriate circumstances, interchangeable (unless expressly disclosed otherwise) and may operate in the disclosure set forth herein in other orders and / or arrangements than those described or illustrated.

[0058] A method for improving recognition of preselected characteristic categories during autonomous driving, the method comprising the steps of determining preselected categories that a machine learning model was unable to accurately predict, collecting specific sensor data associated with the preselected categories, and training the machine learning model to recognize the preselected categories based on training data obtained from the collected specific sensor data.

[0059] In the method of the preceding paragraph, the determining step includes calculating uncertainty data associated with the particular collected sensor data, and identifying a preselected category based at least in part on the calculated uncertainty data.

[0060] The method of the preceding paragraph further comprises the steps of: providing at least a portion of the collected sensor data to a machine learning model; receiving unnormalized predictions for the preselected categories from the machine learning model; normalizing the unnormalized predictions to generate an array of elements representing the normalized predictions and truncating the array; determining a maximum probability for each of the preselected categories for each element of the array; storing an identifier associated with each element having a maximum probability within the preselected probability range; defining, for each of the preselected categories, the normalized elements having a value within a maximum probability value within the preselected probability range as uncertainty data for the preselected category; calculating a ratio of uncertainty data for each of the preselected categories, the ratio being the number of uncertainty data in the preselected category divided by the sum of elements in the category; and selecting training data based at least in part on the uncertainty ratio and the sum of elements of the preselected classes.

[0061] The method according to any preceding clause, wherein the training comprises real-time training of a machine learning model during autonomous driving.

[0062] A method for classifying pixel data includes the steps of establishing a plurality of classes corresponding to relative degrees of terrain drivability; acquiring images of the pixel data using a sensor; assigning, by a processor, a pixel prediction uncertainty label to pixels that are within a predefined range of uncertainty; and calculating, by the processor, an uncertainty ratio value for each of the plurality of classes, wherein the uncertainty ratio value is equal to the number of pixels in the class having an uncertainty label divided by the total number of pixels in the class.

[0063] The method of the preceding paragraph further comprises discarding a portion of the image data that is relatively far from the sensor.

[0064] The method according to any of the preceding clauses, wherein the portion is 3 / 5.

[0065] The method according to any of the preceding clauses, further comprising determining a maximum probability value for the depth of each pixel, and wherein the step of assigning an uncertainty value to each pixel is based on the maximum probability value.

[0066] The method of any of the preceding clauses, further comprising providing a set of pixels having uncertainty ratio values ​​within a predetermined range for a given class for machine learning training.The method of any of the preceding clauses, further comprising providing a set of pixels having uncertainty ratio values ​​within a predetermined range for a given class for machine learning inference.

[0067] An autonomous vehicle comprising: a body; a plurality of wheels coupled to the body and configured to support the body; at least one sensor coupled to the body; and a processor housed in the body and coupled to the at least one sensor, the processor configured to execute a machine learning model trained to recognize preselected categories based at least in part on a subset of data received from the at least one sensor, the subset of data having an associated range of prediction uncertainty.

[0068] In the autonomous vehicle of the preceding paragraph, the processor is coupled to a memory that stores a machine learning model, the machine learning model including instructions executable by the processor to generate unnormalized predictions for preselected categories, normalize the unnormalized predictions, generate an array of elements representing normalized predictions, prune the array, determine a maximum probability for each of the preselected categories for each array element of the array, store an identifier associated with each of the array elements having a maximum probability within a range of preselected probabilities, define, for each of the preselected categories, a normalized element having a value within the maximum probability within a range of preselected probabilities as uncertainty data for one of the preselected categories, calculate an uncertainty ratio of the uncertainty data for each of the preselected categories, where the uncertainty ratio is a count of the uncertainty data for the preselected category divided by a sum of the array elements for the category, and select training data based at least in part on the uncertainty ratio of the preselected category and the sum of the array elements.

[0069] An autonomous vehicle comprising: a body; a plurality of wheels coupled to the body and configured to support the body; at least one sensor coupled to the body; and a processor housed in the body and coupled to the at least one sensor, the processor configured to receive image data from the at least one sensor, the image data including a first plurality of pixels, the processor further configured to execute a machine learning model to recognize pixels of the first plurality of pixels and assign them to a drivability class, the machine learning model further configured to determine a maximum probability value for a depth of each pixel of preselected topographical pixel data, the machine learning model being trained on a second plurality of pixels of an under-represented class of topographical drivability having an uncertainty ratio value having an uncertainty within a predefined uncertainty range, the uncertainty ratio value comprising a quotient of a number of pixels having a predicted probability within the predefined uncertainty range and a total number of the second plurality of pixels.

[0070] In the autonomous vehicle of the preceding paragraph, the processor is coupled to a memory that stores a machine learning model, the machine learning model including instructions executable by the processor to discard portions of image data that are relatively far from the at least one sensor.

[0071] In the autonomous vehicle referred to in the preceding paragraph, the proportion of said portion is three-fifths.

[0072] The autonomous vehicle of any of the preceding clauses, wherein the processor is further configured to determine a maximum probability value for a depth of each pixel of the pixel data.

Claims

1. 1. A method for improving recognition of a category of preselected characteristics during automated driving, comprising: determining the preselected categories that the machine learning model was not able to accurately predict; collecting sensor data associated with the preselected categories; training a machine learning model to recognize the preselected categories based on training data obtained from the collected sensor data; A method comprising the steps of:

2. 2. The method of claim 1 , wherein the determining step comprises: calculating uncertainty data associated with the collected sensor data; and identifying the preselected category based at least in part on the calculated uncertainty data.

3. providing at least a portion of the collected sensor data to the machine learning model; receiving unnormalized predictions for the preselected categories from the machine learning model; normalizing the unnormalized prediction; generating an array of elements representing the normalized prediction; truncating said sequence; determining a maximum probability for each of said preselected categories for each array element; storing an identifier associated with the array element having the maximum probability within a preselected range of probabilities; defining, for each of said preselected categories, a normalization element having a value within said maximum probability value within said preselected probability range as uncertainty data for said preselected category; calculating an uncertainty ratio of the uncertainty data for each of the preselected categories, the uncertainty ratio being the number of the uncertainty data in the preselected category divided by the sum of the array elements in the category; selecting the training data based at least in part on the uncertainty ratio and a sum of the array elements of a preselected class; 2. The method of claim 1, further comprising:

4. 2. The method of claim 1, wherein the training comprises real-time training of the machine learning model during the autonomous driving.

5. 1. A method for classifying pixel data, comprising the steps of: establishing a plurality of classes corresponding to relative degrees of terrain drivability; acquiring an image of pixel data using a sensor, said image having image data; assigning, by a processor, pixel prediction uncertainty labels to pixels that are within a predefined range of uncertainty; calculating, by a processor, an uncertainty ratio value for each of said plurality of classes; Equipped with the uncertainty ratio value being equal to a first number of pixels in the one of the plurality of classes having an uncertainty label divided by a total number of pixels in the one of the plurality of classes.

6. 6. The method of claim 5, further comprising discarding portions of the image data that are relatively far from the sensor.

7. 7. The method of claim 6, wherein the portion is 3 / 5.

8. 6. The method of claim 5, further comprising determining a maximum probability value for a depth of each pixel of the pixel data, and wherein assigning an uncertainty value to each pixel of the pixel data is based on the maximum probability value.

9. 6. The method of claim 5, further comprising providing a set of pixels having uncertainty ratio values ​​within a predetermined range for a given class for machine learning training.

10. 6. The method of claim 5, further comprising providing a set of pixels having uncertainty ratio values ​​within a predetermined range for a given class to a machine learning inference.

11. An autonomous vehicle, Body and a plurality of wheels coupled to the body and configured to support the body; at least one sensor coupled to the body; a processor housed in the body and coupled to the at least one sensor; Equipped with The processor is configured to execute a machine learning model trained to recognize preselected categories based at least in part on a subset of data received from the at least one sensor, the subset of data having an associated range of prediction uncertainty.

12. 12. The autonomous vehicle of claim 11, wherein the processor is coupled to a memory that stores the machine learning model, the machine learning model including instructions executable by the processor to generate unnormalized predictions for the preselected categories, normalize the unnormalized predictions, generate an array of elements representing the normalized predictions, prune the array, determine a maximum probability for each of the preselected categories for each array element of the array, store an identifier associated with each of the array elements having the maximum probability within a range of preselected probabilities, define, for each of the preselected categories, a normalized element having a value within the maximum probability within a range of preselected probabilities as uncertainty data for one of the preselected categories, calculate an uncertainty ratio of the uncertainty data for each of the preselected categories, where the uncertainty ratio is a count of the uncertainty data for the preselected category divided by a sum of the array elements for the category, and select training data based at least in part on the uncertainty ratio for the preselected categories and the sum of the array elements.

13. An autonomous vehicle, Body and a plurality of wheels coupled to the body and configured to support the body; at least one sensor coupled to the body; a processor housed in the body and coupled to the at least one sensor; Equipped with 11. The autonomous vehicle of claim 1, wherein the processor is configured to receive image data from the at least one sensor, the image data including a first plurality of pixels, the processor is further configured to execute a machine learning model to recognize and assign pixels of the first plurality of pixels to a preselected topographical drivability class, the machine learning model being trained on a second plurality of pixels of an under-represented class of topographical drivability having an uncertainty ratio value having an uncertainty within a predefined uncertainty range, the uncertainty ratio value comprising a quotient of a number of pixels having a predicted probability within the predefined uncertainty range and a total number of the second plurality of pixels.

14. 14. The autonomous vehicle of claim 13, wherein the processor is coupled to a memory that stores the machine learning model, the machine learning model including instructions executable by the processor to discard portions of the image data that are relatively far from the at least one sensor.

15. 15. The autonomous vehicle according to claim 14, wherein the proportion of the part is three-fifths.

16. 14. The autonomous vehicle of claim 13, wherein the processor is further configured to determine a maximum probability value for a depth of each pixel of the pixel data.

Images