Method and system for performing monocular depth estimation, and computer readable medium
By using multiple small machine learning models instead of large models in monocular depth estimation and selecting small models with the highest accuracy and lowest loss for processing, the problem of large model resource occupancy and long training time is solved, achieving high efficiency estimation of depth.
Patent Information
- Application Number
- CN202310845356.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Priority Date
- 2023-05-30
- Filing Date
- 2023-07-11
- Publication Date
- 2024-12-03
AI Technical Summary
Large deep neural networks occupy a lot of resources, consume a long time, and have a large amount of training data in monocular depth estimation, making it difficult to be suitable for real-time operations.
Multiple small machine learning models are used instead of large models to process the input dataset and the small model with the highest accuracy and lowest loss is selected through the model selector for processing.
It realizes the reduction of resource usage and training time while maintaining accuracy, improves processing efficiency, and is suitable for real-time operations.
Smart Images

Figure CN119068033A_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates generally to model selection, and more particularly to automatic model selection for monocular depth estimation, and more particularly to a method, system, and computer-readable medium for performing monocular depth estimation. Background Art
[0002] The learning ability of a deep neural network may be positively correlated with the number of parameters and the relative accuracy of the parameters. Therefore, for general or non-specific input data sets, a large deep neural network (e.g., a deep neural network with a number of parameters greater than a threshold number) can achieve high accuracy. However, large deep neural networks may be accompanied by many disadvantages. For example, large deep neural networks may take up a lot of memory, consume a lot of processing resources, take time to process data sets (which may be a problem for real-time operation), use a large amount of training data sets to train large deep neural networks to the necessary accuracy, take a long time to train, and so on. Therefore, although large deep neural networks can be trained to acceptable accuracy for certain data sets, large deep neural networks may not be applicable in many cases. Summary of the invention
[0003] A method for model selection for monocular depth estimation is described herein. The method may include: receiving a plurality of images; selecting one or more images from the plurality of images; processing the one or more images using a first machine learning model to generate a first prediction result; processing the one or more images using a plurality of machine learning models to generate at least a second prediction result and a third prediction result, wherein the first machine learning model is larger than the plurality of machine learning models; selecting a second machine learning model from the plurality of machine learning models based on a comparison of the first prediction result with the at least second prediction result and the third prediction result; and processing the plurality of images using the second machine learning model.
[0004] A system for model selection for monocular depth estimation is described herein. The system may include: one or more processors; and a non-transitory computer readable medium storing instructions that, when executed by the one or more processors, cause the one or more processors to perform any of the methods described previously.
[0005] The non-transitory computer-readable medium set forth herein may store instructions that, when executed by one or more processors, cause the one or more processors to perform any of the previously set forth methods.
[0006] These illustrative examples are mentioned not to limit or define the present disclosure, but to aid in understanding the present disclosure.Additional embodiments are discussed in the detailed description, and further description is provided in the detailed description. BRIEF DESCRIPTION OF THE DRAWINGS
[0007] The features, embodiments and advantages of the present disclosure will be better understood when the following detailed description is read with reference to the accompanying drawings.
[0008] Figure 1 A block diagram illustrating an exemplary system for selecting a machine learning model configured to process disparate data sets in accordance with aspects of the present disclosure;
[0009] Figure 2 An exemplary media handling system according to aspects of the present disclosure is shown;
[0010] Figure 3 A block diagram illustrating an exemplary distributed data processing network according to aspects of the present disclosure;
[0011] Figure 4 A block diagram illustrating an exemplary model selection process for selecting a small machine learning model for processing a data set in accordance with aspects of the present disclosure;
[0012] Figure 5 A flowchart diagram illustrating an exemplary process for model selection for monocular depth estimation according to aspects of the present disclosure;
[0013] Figure 6 An exemplary computing device architecture is shown that illustrates an exemplary computing device that can implement the various techniques set forth herein in accordance with aspects of the present disclosure.
[0014] Explanation of symbols
[0015] 104, 104-1, 104-2, 104-3, 104-n: computing device
[0016] 108: Central Processing Unit (CPU)
[0017] 112: Memory
[0018] 116: Input / Output Interface
[0019] 120: Network interface
[0020] 124: Data Processor
[0021] 128, 208: Network
[0022] 132: Installation
[0023] 136: Model Selector
[0024] 140: Feature Extractor
[0025] 144: Training Data
[0026] 148, 408: Machine Learning Models / Large Machine Learning Models
[0027] 152: Machine Learning Model / Small Machine Learning Model / Small ML Model 1 / Small Machine Learning Model 1
[0028] 156: Machine Learning Model / Small Machine Learning Model / Small Machine Learning Modeln / Small ML Modeln
[0029] 204: Client device
[0030] 212: Content Delivery Network
[0031] 412: Machine Learning Model / Small Machine Learning Model / Small Model 1 / Small Machine Learning Model 1
[0032] 416: Machine Learning Models / Small Machine Learning Models / Small Models 2
[0033] 420: Machine Learning Models / Small Machine Learning Models / Small Models 3
[0034] 424: Machine Learning Model / Small Machine Learning Model / Small Machine Learning Modeln / Small Modeln
[0035] 428: Model output / output / truth model output / truth depth map
[0036] 432: Model Output / Output / Model 1 Output
[0037] 436, 440: Model output / output
[0038] 444: Model Output / Output / Model n Output
[0039] 452: Benchmark Selection
[0040] 456: Model Selection
[0041] 504, 508, 512, 516, 520, 524: Blocks
[0042] 600: Computing device / computing device architecture
[0043] 602: Cache
[0044] 604: Processor
[0045] 606: Connectors
[0046] 608: Storage device
[0047] 610: Service 1
[0048] 612: Service 2
[0049] 614: Service 3
[0050] 616: Random Access Memory (RAM)
[0051] 618: Read-only memory (ROM)
[0052] 620: Memory
[0053] 622: Input Device
[0054] 624: Output device
[0055] 626: Communication interface DETAILED DESCRIPTION
[0056] The learning ability of a machine learning model may be related to the number of parameters or the number of layers of the model. Enhancing the learning ability (e.g., by increasing the number of parameters or the number of layers) may enable the machine learning model to learn from a wider range of data sets. For example, increasing the number of parameters of a classifier may increase the number of classifications that the classifier can reliably distinguish. Increasing the number of parameters or the number of layers of a machine learning model may also increase the processing cost of executing the model (e.g., processing load, execution time, training time, etc.), which may render the machine learning model inoperable under certain conditions (e.g., such as real-time operation, etc.).
[0057] Methods and systems for machine learning model selection for discrete processing tasks are described herein. Multiple small machine learning models can be instantiated and trained to process input data sets instead of large machine learning models. Since each small machine learning model may include fewer parameters or layers than a large machine learning model, the small machine learning model can be configured to achieve the same degree of accuracy as the large machine learning model for a portion of a given input data set. Each small machine learning model can be configured to process a specific input data set (e.g., a data set including specific characteristics, etc.), or can be configured to produce a specific output (e.g., a subset of possible outputs that a large machine learning model can be configured to produce, etc.). The multiple small machine learning models can be configured together to process the same input data set that the large machine learning model is configured to process with similar accuracy and / or loss as the large machine learning model. However, a small machine learning model with fewer parameters or layers than a large machine learning model can operate more efficiently (e.g., using fewer processing resources for storage and / or execution, using a smaller training data set for training, faster training time, etc.).
[0058] For example, a large classifier may be configured to classify an input image according to a plurality of different categories based on objects within the input image. A first small machine learning model may be instantiated to classify an input image according to a subset of the different categories, and a second machine learning model may be instantiated to classify an input image according to the remaining different categories. Alternatively or additionally, a first machine learning model may be instantiated to classify input images characterized by natural lighting (e.g., daylight, etc.), and a second machine learning model may be instantiated to classify input images characterized by synthetic lighting (e.g., flash, incandescent, fluorescent, etc.).
[0059] In some examples, a large machine learning model can be compressed into small machine learning models. Compressing the machine learning model can reduce the number of parameters or the number of layers, which can make the compressed machine learning model suitable for processing a portion of the input data set that the large machine learning model can process. Once compressed, the multiple small machine learning models can be instantiated by training the compressed machine learning model using different training data sets. Each small machine learning model can be trained to process a range of input data sets that the corresponding large machine learning model is expected to process.
[0060] Once trained, the input data set can be processed using a large machine learning model and multiple small machine learning models. For any input data set, the model selector can determine which machine learning model (among the large machine learning model and the multiple small machine learning models) should process a specific input data set. In some examples, the model selector can sample the input data set to generate a test feature vector, which can be used as an input and passed into the machine learning model to generate a corresponding test output. For a deep neural network (DNN), one or more initial layers of the DNN can operate as a feature extractor. The test output from the large machine learning model can be marked as a pseudo ground truth (e.g., assumed to be true). The model selector can then compare the test output from each small machine learning model with the test output from the large machine learning model. In some examples, the model selector may use an accuracy metric or loss function (e.g., accuracy, precision, area under the curve, logarithmic loss, F1 score, weighted human disagreement rate, cross entropy, mean absolute error, mean square error, etc.). The model selector may identify a specific small machine learning model with the highest accuracy metric or lowest loss (according to the loss function) among the multiple small machine learning models. The model selector may then use the specific small machine learning model to process the rest of the specific input data set.
[0061] In an illustrative example, monocular depth estimation may be performed by one or more machine learning models (e.g., DNN or similar models) for various computer vision operations (e.g., but not limited to, classification, semantic segmentation, object detection, instance segmentation, depth estimation, etc.) (e.g., such as, for autonomous driving of self-driving cars, virtual reality, augmented reality, three-dimensional (3D) simulation, target acquisition, etc.). A computing device may instantiate (define and train) a large DNN to process various images, video frames, or video clips, thereby generating a depth map or an inverse depth map using monocular depth estimation. The depth map may represent each pixel of the image as a distance (e.g., a real number) between a location in the environment represented by the pixel and a camera. Multiple small DNNs may also be trained to process various input images, video frames, and / or video clips. In some instances, each small DNN may be generated by compressing the large DNN and training the compressed large DNN.
[0062] The computing device may receive a plurality of images. The plurality of images may be different images, images extracted from video frames, images extracted from video clips, or the like. The images may be received from a content delivery network, a client device, another computing device, a camera (e.g., such as a live camera stream or previously stored images captured from a camera), a server, etc. Alternatively, the computing device may receive the images by extracting the images from a video clip stored in a memory of the computing device.
[0063] The computing device may select one or more images from the plurality of images. In some examples, the computing device samples the plurality of images to derive the one or more images.
[0064] The computing device may process the one or more images using the large DNN to generate a first prediction result corresponding to the output of the large DNN. For example, the computing device may generate a feature vector using the one or more images, and the feature vector may be passed as an input to the large DNN. The large DNN may process the feature vector and output a first prediction result (e.g., a depth map or a reverse depth map, etc.). In some examples, the computing device may regard the first prediction result as a false true value.
[0065] The computing device may process the one or more images using the multiple small DNNs to generate additional prediction results. For example, a first small DNN may process the one or more images to generate a second prediction result, a second small DNN may process the one or more images to generate a third prediction result, and so on. Each of the small DNNs may be smaller than the large DNN (e.g., fewer parameters and / or layers, etc.).
[0066] The computing device may select a small DNN from the plurality of small DNNs based on a comparison of the first prediction result with the second prediction result, the third prediction result, and the like. In some instances, the computing device may use one or more accuracy metrics and / or loss functions to compare the second prediction result, the third prediction result, and the like relative to the first prediction result. For example, since the prediction result includes a depth map or a reverse depth map (e.g., representing each pixel as a real distance from the camera), a loss function may be used to determine the difference between the first prediction result (which is marked as a false true value) and the second prediction result, the first prediction result, and the third prediction result, and the like. Examples of loss functions include, but are not limited to, adaptive robust loss, mean squared error, mean absolute error, cross entropy, weighted human disagreement rate (WHDR), combinations thereof, or similar functions. The computing device may select a specific small DNN with the highest accuracy metric, lowest error, lowest loss, and the like.
[0067] The computing device may then process the plurality of images using a specific small DNN to generate a depth map or a reverse depth map from the plurality of images. In some examples, the computing device may process each of the plurality of images. In other examples, the computing device may process a portion of the plurality of images by sampling the plurality of images. For example, the computing device processes every nth image of the plurality of images.
[0068] In some examples, the model selection process may be repeated to ensure that the particular small DNN remains the most efficient small DNN for processing the plurality of images. The model selection process may be re-executed at regular time intervals upon detection of an event, upon detection of user input, after a predetermined number of instances of a particular small DNN are executed therein, upon detection of a change in one or more characteristics of the plurality of images (e.g., such as a change in average pixel value, etc.), in a combination of the above, or in similar circumstances. The computing device may continuously ensure that the most efficient small DNN is used to process the plurality of images.
[0069] The model selection process can be applied to a variety of machine learning models to determine the most efficient way to process disparate data sets. In this way, the techniques described in this article can be applied to deep neural networks (as described previously) as well as any other type of machine learning model.
[0070] Figure 1 A block diagram of an exemplary system for selecting a machine learning model configured to process disparate data sets according to various aspects of the present disclosure is shown. The computing device 104 may be configured to process disparate data sets for adjacent devices (e.g., such as devices operating within the same network) and / or remote devices (e.g., such as devices operating within other networks, etc.). The computing device 104 may include a central processing unit (CPU) 108, a memory 112 (e.g., volatile memory (e.g., random access memory, etc.) and non-volatile memory (e.g., flash memory, hard disk, etc.) connected via a bus or similar device, an input / output interface 116, a network interface 120, and a data processor 124. In some embodiments, the computing device 104 may include additional components or fewer components.
[0071] The input / output interface 116 may include one or more hardware and / or software interfaces configured to receive data from and / or transmit data to one or more devices 132 connected to the computing device 104, such as, but not limited to, a display device, a keyboard and mouse, a sensor, a peripheral device, a media streaming device, an augmented reality device, a virtual reality device, and / or the like. In an illustrative example, a first device of the one or more devices 132 may be a virtual reality display device configured to project a three-dimensional representation of a medium (e.g., a video, a video game, one or more images, etc.). If the medium does not include three-dimensional data (e.g., the medium is two-dimensional, etc.), the computing device 104 may perform monocular depth estimation using the data processor 124 to generate a depth map and generate a three-dimensional representation of the medium from the depth map. The computing device 104 may then transmit the three-dimensional representation of the medium to the virtual reality display via the input / output interface 116. One or more devices 132 may be connected via a wired connection (e.g., universal serialbus (USB) type A, USB type B, or USB type C; high-definition multimedia interface (HDMI); digital visual interface (DVI); DisplayPort; etc.) or a wireless connection (e.g., such as but not limited to wireless fidelity (Wi-Fi), Bluetooth, Zigbee, Z-wave, infrared, ultra-wideband, etc.).
[0072] The network interface 120 may enable connection to one or more remote devices via a network 128 (e.g., the Internet, a local area network, a wide area network, a cloud network, etc.). In some examples, the computing device 104 may receive a request to process data using the data processor 124 via the network interface 120. Once the request is received, the computing device 104 may store the data in the memory 112, process the data using the data processor 124, and transmit the output to the requesting device (or one or more other devices) via the network 128. Alternatively or additionally, the output may be presented via one or more devices 132. In some examples, the data processor 124 may process the received data in real time. In these examples, the data processor 124 may process the streamed data as it is received (streamed data is received via the network interface 120 or the input / output interface 116), or may store a portion of the stream in a buffer in the memory 112 and process a portion of the streamed data stored in the buffer whenever the buffer is full.
[0073] In some embodiments, the data processor 124 may be a separate component of the computing device 104 connected to the CPU 108, the memory 112, the input / output interface 116, and the network interface 120 via a bus. The data processor 124 may be configured to operate within the computing device 104 or may operate independently of the computing device 104. For example, the data processor 124 may be an application specific integrated circuit (ASIC), a field programmable gate array, a mask programmable gate array, a microcontroller, or the like configured to process instructions stored in the memory of the data processor 124. Alternatively, the data processor 124 may be a non-volatile memory (as a separate component connected to the bus or a subcomponent of the memory 112) configured to process various data sets, the non-volatile memory storing instructions. The instructions may be executed by the CPU 108 (and / or other components of the computing device 104).
[0074] The data processor 124 may include: a model selector 136, configured to select a particular machine learning model for processing a particular data set; a feature extractor 140, configured to generate an input feature vector for a selected machine learning model (e.g., a model that does not require an additional feature extractor); training data 144, storing training data for the machine learning model; a large machine learning model 148; and one or more small machine learning models (e.g., such as small ML model 1 152 to small ML model n 156, where n can be any integer greater than 1).
[0075] The data processor 124 may use two or more machine learning models to process various types of data sets. The two or more machine learning models may have different sizes, which allow the data processor 124 to dynamically select the most efficient machine learning model for processing a given data set based on the current state of the data processor 124 and / or the computing device 104, or dynamically switch to a different machine learning model. The two or more machine learning models may include a large machine learning model (e.g., a machine learning model with a number of parameters or a number of layers greater than a threshold) and one or more small machine learning models (e.g., a machine learning model with a number of parameters or a number of layers less than a threshold).
[0076] The size of a machine learning model (e.g., the number of parameters of a neural network, the number of layers, etc.) may indicate the learning potential of the machine learning model. Large machine learning models may be trained to process general data sets (e.g., data sets that may not correspond to any taxonomy or may not have any specific shared characteristics). For example, a large image classifier trained to classify objects within an image may classify randomly sampled input images (e.g., daylight, indoors, at night or low light, objects to be classified are occluded or far from the camera, objects to be classified are clear and close to the camera, etc.). A small machine learning model may have lower accuracy and / or higher loss when classifying specific types of images. For example, a small image classifier trained to classify objects within an image may classify images that share specific characteristics (e.g., such as images taken during the day or under a lot of lighting) and may have lower accuracy or higher loss when classifying images with different characteristics (e.g., such as images captured at night or in low light conditions, etc.).
[0077] Large machine learning models may have a larger memory footprint and may use more processing resources (e.g., CPU 108, cache or volatile memory, non-volatile memory, bandwidth, etc.) than corresponding small machine learning models. Large machine learning models may also have different training intervals than small machine learning models and execute over longer intervals, making the use of large machine learning models more complex for time-sensitive operations.
[0078] The machine learning models 148 to 156 may be any type of machine learning model, including but not limited to a neural network, a deep neural network, a transformer, a classifier, a support vector machine, a decision tree, etc. In some examples, the machine learning models 152 to 156 may be generated by compressing the large machine learning model 148 (before, during, or after the large machine learning model 148 is trained). In these examples, the large machine learning model 148 may be compressed by pruning (e.g., removing unnecessary parameters or layers, etc.), quantization (e.g., reducing the memory footprint of parameters, etc.), knowledge distillation (e.g., training a small machine learning model to simulate a large machine learning model), low-rank factorization, etc.
[0079] The large machine learning model 148 may be trained and / or compressed by the data processor 124. One or more small machine learning models 152 to 156 may be generated and / or trained by the data processor 124 by compressing the large machine learning model 148. The model selector 136 may determine a machine learning model of the type of processing task to be performed for the selected processing task. The model selector 136 may pass a training request to the feature extractor 140. The feature extractor 140 may generate a training data set to train a machine learning model of the type of processing task to be performed. The data processor 124 may train a machine learning model using training data stored in the training data 144, generated (e.g., generated by the feature extractor 140 programmatically or received from user input), or received from one or more remote devices (e.g., one or more devices 132, one or more remote devices connected via the network 128, etc.) to perform one or more operations. The training data 144 may store data configured to train a machine learning model to process a specific type of input data. For example, the training data 144 may store image data so that the machine learning model can be trained to process the image (e.g., generate a depth map, classify an image, detect an object, etc.). The training data 144 may also store historical data (e.g., data associated with historical executions of the machine learning models 148 to 156, etc.), generated data, received data, etc. If one or more small machine learning models 152 to 156 are to be trained independently of the large machine learning model 148, the feature extractor 140 may generate a training set for the one or more small machine learning models 152 to 156 based on the type of machine learning model and the size of the machine learning model to be trained. The training data set used to train the one or more small machine learning models 152 to 156 may be similar or identical to the training data set used to train the large machine learning model 148.
[0080] The feature extractor 140 may use the training data set to train the machine learning models 148 to 156. The machine learning models 148 to 156 may be trained for a predetermined number of iterations at predetermined time intervals, until a target accuracy metric is reached, until a target loss value (according to one or more loss functions) is reached, and so on.
[0081] The data processor 124 may receive a data set for processing using one or more of the trained machine learning models 148 to 156. The data set may be received via the input / output interface 116, the network interface 120, or stored in the memory 112. The data set may be a discrete data set (e.g., a well-defined size and / or length, etc.) or may be a continuous stream (e.g., such as a broadcast medium, a video game, or other medium of uncertain size or length, etc.). The model selector 136 may determine which of the machine learning models 148 to 156 will most efficiently process the received data set (or a portion of the received data set) by sampling the data set, processing the sample using a machine learning model, and determining which machine learning model should process the data set by comparing the results.
[0082] The model selector 136 may sample the data set by extracting a portion of the data set. The model selector 136 may sample an initial portion of the data set (e.g., a first number of bits, a first number of images or video frames, a first predetermined second of audio, etc.). Alternatively or additionally, the model selector 136 may obtain a random sample of the data set by randomly selecting a portion of the data set, for example, using a random number generator. The model selector 136 may send an identification of the sample and an indication of which machine learning model to be utilized to the feature extractor 140.
[0083] The feature extractor 140 may generate a feature vector for the selected machine learning model (e.g., the large machine learning model 148 and one or more machine learning models 152 to 156 that do not include an internal feature extraction function, etc.). The large machine learning model 148 may be executed using the feature vector from the feature extractor 140 and generate a first output (e.g., a first prediction result). Small machine learning models 1 152 to small machine learning models n 156 may also be executed using the same feature vector (or a feature vector customized by the feature extractor 140 for the corresponding small machine learning model) to generate a second output (e.g., a prediction result from small machine learning model 1 152) to the nth output (from small machine learning model n 156, etc.).
[0084] The model selector 136 may compare the first output with the second to nth outputs to determine which of the small machine learning models should be processing the data set. For example, the model selector 136 may mark the first output as true, and then use the first output to measure the accuracy and / or loss of the second to nth outputs to determine the accuracy (e.g., using an accuracy metric, etc.) and / or loss (e.g., using a loss function, etc.) of each small machine learning model 152 to 156 relative to the large machine learning model 148 for a particular data set. The model selector 136 may select the small machine learning model 152 to 156 with the highest accuracy and / or lowest loss. Alternatively, the model selector 136 may measure the second to nth outputs relative to the first output and / or relative to the second to nth outputs, thereby generating a distribution of relative outputs (e.g., the second output relative to the third output, the second output relative to the fourth output, the second output relative to the nth output, etc.), and a specific output may be selected as a preferred output over other outputs based on the relative outputs. Then, the small machine learning model corresponding to the particular output may be selected to process the data set. Alternatively, the model selector 136 may measure each of the second to nth outputs independently of the other outputs to determine the small machine learning model that should process the data set. In this example, the large machine learning model 148 may not be used (e.g., the first output may not be generated, etc.). The model selector 136 may measure the output relative to the other outputs using any accuracy metric and / or loss function.
[0085] Alternatively, model selector 136 may determine to use large machine learning model 148 instead of small machine learning models 152-156 to process the data set. Model selector 136 may determine that small machine learning models 152-156 may have an accuracy metric less than a first threshold and / or a loss function greater than a second threshold. Therefore, model selector 136 may select large machine learning model 148 to be the most efficient machine learning model for processing the data set. Model selector 136 may select a machine learning model by balancing processing efficiency (e.g., small machine learning models 152-156 may be more efficient using fewer processing resources) and accuracy (large machine learning model 148 may sometimes be more accurate than small machine learning models 152-156).
[0086] The model selector 136 may select a small machine learning model as long as the accuracy of the selected small machine learning model relative to the large machine learning model 148 is: 1) greater than other small machine learning models under consideration; and 2) greater than a first threshold. Examples of accuracy metrics and / or loss functions include, but are not limited to, accuracy, precision, area under the curve, log loss, F1 score, weighted human disagreement rate, cross entropy, mean absolute error, mean squared error, or similar functions. The model selector 136 may use the small machine learning model with the highest accuracy and / or lowest loss to begin processing the rest of the data set.
[0087] In some instances, the model selector may perform the machine learning model selection process again during processing of the data set to ensure that the selected small machine learning model is still the most efficient machine learning model for processing the data set. For example, the data processor 124 may process the video stream to generate an estimated depth map for each video frame (or every nth frame, etc.). The first few video frames of the video stream may correspondingly include high-light conditions, for which the small machine learning model 1 152 is shown to be the most efficient (based on the execution of the aforementioned model selection process). Subsequent portions of the video stream may include video frames including low-light conditions, for which the small machine learning model 1 152 may not be the most efficient (e.g., when processing low-light video frames, the small machine learning model 1 152 may have lower accuracy and / or higher loss). The model selector 136 may use one or more recent inputs to the small machine learning model 1 152 to re-execute the model selection process and select one of the large machine learning model 148 and / or the small machine learning model 2 (not shown) to the small machine learning model n 156 to take over processing of the video stream.
[0088] The model selection process may be re-executed at regular intervals (e.g., every n video frames, every n seconds, etc.) upon detecting an event, upon receiving user input, upon detecting a change in one or more characteristics of the portion of the data set input to the selected small machine learning model (during a particular iteration) and / or an output from the selected small machine learning model (e.g., such as a change in average pixel value in the previous example, etc.), an accuracy metric and / or a loss function, a combination of the above, or in similar circumstances. The model selector 136 may continuously monitor the execution of the selected small machine learning model for a given data set to ensure that the most efficient small machine learning model is being executed.
[0089] Figure 2An exemplary media processing system according to aspects of the present disclosure is shown. In some instances, computing device 104 may operate as a load balancer by providing processing services to one or more client devices (e.g., client device 204). For example, client device 204 may be any processing device, such as, but not limited to, a desktop or laptop computer, a mobile device (e.g., such as a smartphone, tablet, etc.), a video game console, a server, etc. Client device 204 may operate a processing intensive application. Client device 204 may use the resources of computing device 104 by transmitting and / or streaming a data set to computing device 104. Computing device 104 may use data processor 124 to select a small machine learning model that is configured to process the data set to produce an output (or output stream). Computing device 104 may transmit (or stream) the output back to client device 204.
[0090] In other instances, computing device 104 may process data sets that may not be locally processed by client device 204. For example, computing device 104 may operate a virtual reality application that is configured to present three-dimensional representations of various media (e.g., movies, video games, simulations, etc.). Computing device 104 may receive content associated with the virtual reality application from content delivery network 212 via network 208. In another instance, computing device 104 may receive an image from a live camera feed (or an image gathered from a live camera feed), and if the content is not already a three-dimensional representation, computing device 104 uses monocular depth estimation to convert the content into a three-dimensional representation. Monocular depth estimation is a process for determining the approximate distance between a surface represented in an image (or video frame) and the camera that captured the image. In some instances, monocular depth estimation may be performed on each pixel in an image (or video frame) to generate a depth map. The distance may be used to generate a three-dimensional representation of a two-dimensional image. The three-dimensional representation may be used in computer vision applications such as augmented reality, virtual reality, 3D television, video games, geography, and the like. Figure 3 dimensional environments, simulations, vehicle automation (e.g., driverless cars), etc.
[0091] Computing device 104 may receive a request for three-dimensional content from client device 204. Computing device 104 may request content from content delivery network 212 and process the content (in real time). In some examples, computing device 104 may transmit content and a depth map generated by a data processor (e.g., such as data processor 124) of computing device 104 to client device 204. Client device 204 may use the content and the depth map to generate a three-dimensional representation of the content for a virtual reality application. Alternatively, client device 204 may receive content directly from content delivery network 212 and receive a depth map from computing device 104. Each depth map may be associated with metadata indicating the location of the content corresponding to the depth map. In other examples, computing device 104 may generate a three-dimensional representation of the content and transmit or stream the three-dimensional representation of the content to client device 204. For example, client device 204 may connect to computing device 104 and stream various three-dimensional representations of content from a content delivery network generated by computing device 104.
[0092] Figure 3 A block diagram of an exemplary distributed data processing network according to various aspects of the present disclosure is shown. The computing device 104 may operate in a distributed network that is configured to provide processing services to one or more devices (e.g., client devices 204, other devices, servers, networks, etc.). The computing device 104 may include a data processor 124 configured to process various data sets. The data processor 124 may use a model selection process for a large machine learning model and one or more small machine learning models to determine the most efficient machine learning model to be used when processing a particular data set. The small model selection process may balance or reduce the processing load of the computing device 104 and achieve overall accuracy when selecting a machine learning model to be processed for a particular data set. In some examples, the computing device 104 may operate on multiple large machine learning models and corresponding one or more small machine learning models to achieve parallel processing of similar and / or completely different data sets.
[0093] In some examples, computing device 104 may operate as a node in a distributed data processing network. Any number of additional computing devices (e.g., computing device 104-1, computing device 104-2, computing device 104-3, computing device 104-n, etc.) may also operate in the distributed data processing network. Computing devices 104 and each of computing devices 104-1 to 104-n may include a data processor (e.g., such as data processor 124) having a large machine learning model, one or more small machine learning models, and a model selector configured to identify the most efficient small machine learning model that can process a given data set at a threshold accuracy and / or loss.
[0094] The computing device 104 may also include a load balancer configured to identify a specific computing device capable of processing a specific data set. For example, the client device 204 may transmit a request with an identification of a specific data set to be processed to the computing device 104. The load balancer may select a computing device capable of processing the specific data set from the computing device 104 and the computing devices 104-1 to 104-n. The load balancer may select a computing device using one or more characteristics of the specific data set and the computing device 104 and the computing devices 104-1 to 104-n, such as, but not limited to, the processing load of each respective computing device, the data type of the specific data set, the expected output, the data type capable of being processed by the respective computing device, the network bandwidth, the transmission path (e.g., for transmitting the specific data set to each respective computing device and for transmitting the output back to the client device 104, etc.), the accuracy and / or loss of the machine learning model configured to process the specific data set (as determined using the model selection process as previously described), a combination thereof, or the like. In some examples, the one or more features may be weighted, where the weights are continuously adjusted based on the state of the distributed data processing network. For example, features corresponding to the capabilities of a computing device based on a particular data set may be given a high weight to ensure that the selected computing device is capable of processing the particular data set (e.g., if the particular data set includes image data, the small machine learning model of the selected computing device is trained to process image data, etc.). Other features may be weighted to balance the processing load on the distributed data processing network.
[0095] In some instances, a data set may be processed by more than one computing device. For example, a large data set or media stream may be processed in discrete segments (e.g., each image or video frame, each n-second chunk, each n-bit data, etc.). The computing device 104 may generate a sequence of feature vectors. Alternatively, for real-time operation, the computing device may generate feature vectors as the computing device receives the data. Each feature vector may be associated with a sequence identifier that indicates a portion of the data set from which the feature vector originates. The computing device 104 may then transmit the feature vector to a computing device selected by a load balancer to process the feature vector and generate an output. The output from the selected computing device and the identification and / or sequence identifier of the feature vector may be received by the computing device 104. The computing device 104 may then assemble an output sequence (when processing non-real-time data) from the output received from the computing device that processes the data set, or transmit each output to the client device 204 when the output is generated. By distributing data sets across computing devices in a distributed data processing network, computing device 104 may reduce the processing load of the distributed data processing network, reduce processing latency by processing portions of a data set in parallel, maintain the accuracy of the data set being processed, and so on.
[0096] In other examples, computing device 104 may transmit the selected small machine learning model to client device 204 to enable client device 204 to process the data set locally. In these examples, a data processor of computing device 104 may use a sample of the data set to perform a model selection process to identify a specific small machine learning model that can process the data set with a threshold accuracy or loss (as previously described). Computing device 104 may then transmit the selected small machine learning model to client device 204. The client device may process the remainder of the data set using the selected small machine learning model.
[0097] Figure 4A block diagram of an exemplary model selection process for selecting a small machine learning model for processing a data set according to aspects of the present disclosure is shown. The model selection process may identify a specific machine learning model to be used to process a given data set. A large machine learning model 408 (e.g., having a number of parameters and / or a number of layers greater than a threshold) may be trained to process a general data set. The large machine learning model 408 may be compressed to produce one or more small machine learning models (e.g., small model 1 412, small model 2 416, small model 3 420, small model n 424, etc.). The large machine learning model 408 may be compressed before, during, or after training. The large machine learning model 148 may be compressed by pruning (e.g., removing unnecessary parameters or layers, etc.), quantization (e.g., reducing the memory footprint of parameters, etc.), knowledge distillation (e.g., training a small machine learning model to simulate a large machine learning model), low-rank factorization, or by any other compression algorithm. Alternatively, the one or more small machine learning models may be defined and trained independently. Any number of small machine learning models can be generated (either by compression or by independent training), where n is any integer greater than 1.
[0098] When a request is received for processing a particular data set, the model selection process may begin. A particular data set may be sampled to generate one or more discrete portions of the data set that may be processed by machine learning models 408 to 424. One or more feature vectors may be derived from each of the one or more discrete portions of the data set. In some instances, a single feature vector may be derived for machine learning models 408 to 424. In other instances, a feature vector may be derived for each machine learning model, which may be customized for the machine learning model (e.g., based on the number of parameters and / or number of layers of the model, etc.). Each machine learning model 408 to 424 may be executed using the one or more feature vectors to generate a corresponding model output. The large machine learning model 408 may process the feature vector to generate a model output 428. The small machine learning model 1 412 may process the feature vector to generate a model 1 output 432, and the small machine learning model n 424 may process the feature vector to generate a model n output 444, and so on.
[0099] At benchmark selection 452, one or more benchmarks may be selected to evaluate model outputs 428 to 444. Benchmark selection 452 may designate a model output (e.g., model output 428) from large machine learning model 408 as a true value, and model outputs (e.g., model outputs 432 to 444) from small machine learning models 412 to 424 may be compared relative to model output 428.
[0100] In some examples, the benchmark selection may determine the benchmark based on the data type of the model outputs 428-444. For example, the benchmark for a classifier may be an accuracy metric or error metric that evaluates the output according to a Boolean value (e.g., true / false or correct / incorrect, etc.). The benchmark for a machine learning model that outputs a numerical output (e.g., such as a depth estimation machine learning model that may output a depth map or an inverse depth map, etc.) may be a loss function (e.g., that determines the difference between a control value and the output). The benchmark selection 452 may use one or more benchmarks when evaluating the outputs 428-444.
[0101] In some examples, for outputs including depth maps, the benchmark selection 452 may use a weighted human disagreement rate, mean absolute relative error, robust function loss, or a similar function. The weighted human disagreement rate uses equal weights (e.g., set to 1) and identifies for each pixel of the output depth map whether the pixel is closer or farther away from the corresponding pixel of the ground truth model output 428. Each pixel of the model output may be replaced with 0 (indicating that the pixel of the model output is closer to the corresponding pixel of the ground truth depth map) or 1 (indicating that the pixel of the model output is farther away from the corresponding pixel of the ground truth depth map). A distribution of 0 and 1 may be used to evaluate the degree to which the model output depth map deviates from the ground truth. The mean absolute relative error may be calculated using The error is evaluated, where z i corresponds to the value of pixel I on the depth map being evaluated, corresponds to the ground truth value of pixel i of the ground truth depth map 428, and M corresponds to the total number of pixels in the depth map.
[0102] The robust loss function can be represented by d, where d represents the predicted disparity (between the true value of a given pixel and the model output), d* represents the true value of the pixel from the ground-truth model output, and M represents the number of pixels in the depth map.
[0103]
[0104] In other examples, any accuracy metric, loss function, error rate, or similar function can be used to evaluate the model output relative to the true model output. Examples of accuracy metrics and / or loss functions include, but are not limited to, accuracy, precision, area under the curve, log loss, F1 score, weighted human disagreement rate, cross entropy, mean absolute relative error, mean squared error, or similar functions.
[0105] Baseline selection 452 may then identify the model output 432 to 444 with the highest accuracy or lowest loss. Model selection 456 may then select a small machine learning model corresponding to the identified model output from the small machine learning models 412 to 424. The selected small machine learning model may be used to process the remainder of the particular data set.
[0106] Figure 5 A flowchart diagram illustrating an exemplary process of model selection for monocular depth estimation according to aspects of the present disclosure. Monocular depth estimation may be performed by one or more machine learning models (e.g., deep neural networks or similar models) used for various computer vision operations (e.g., but not limited to, classification, semantic segmentation, object detection, instance segmentation, depth estimation, etc.) (e.g., such as for autonomous driving of self-driving cars, virtual reality, augmented reality, three-dimensional simulation, target acquisition, etc.). The model selection process may be performed to select an efficient machine learning model based on the processing resources consumed by the selected machine learning model and the accuracy of the machine learning model when processing a particular data set.
[0107] For example, at block 504, the computing device may receive a plurality of images. The plurality of images may be independent images (independent of other images in the plurality of images, etc.), images extracted from video frames, images extracted from video clips, or the like. The images may be received from a camera, a content delivery network, a client device, another computing device, a server, etc. Alternatively, the computing device may receive the images by extracting the images from a video clip stored in a memory of the computing device or from a live camera stream. For a live camera stream, images may be received continuously as they are captured by the camera. As additional images of the live camera stream will be received over time, the computing device is considered to have received the plurality of images once the first image from the live camera stream is received.
[0108] At block 508, the computing device may select one or more images from the plurality of images. For example, the computing device may sample the plurality of images to derive the one or more images. For a live camera stream, the one or more images may correspond to the one or more images received from the live camera stream. In some instances, the computing device may randomly sample the plurality of images. In other instances, the computing device may select the one or more images from the plurality of images according to one or more parameters. The one or more parameters may be based on the number and / or order of images in the plurality of images, characteristics of the images (e.g., pixel values (e.g., average red value, green value, blue value, and / or pixel brightness value, etc.)), metadata associated with the plurality of images, combinations thereof, or the like. For example, the computing device may sample the plurality of images by uniformly selecting images (e.g., such as the first image and the last image when two images are sampled from the plurality of images, etc.) over the distribution of the plurality of images based on the number of images to be included in the sampling.
[0109] At block 512, the computing device may process the one or more images using a first machine learning model to generate a first prediction result. For example, the computing device may use the one or more images to generate a feature vector. The feature vector may be passed as an input to the first machine learning model. The first machine learning model may process the feature vector and output a first prediction result. The first machine learning model may be a large machine learning model. The large machine learning model may be a machine learning model having a number of parameters and / or a number of layers greater than a threshold. In some examples, the computing device may mark the first prediction result as a pseudo-true value that can be used to compare the output of other machine learning models with the first prediction result.
[0110] At block 516, the computing device may process the one or more images using a plurality of machine learning models to generate a second prediction result (e.g., using a first machine learning model from the plurality of machine learning models), a third prediction result (e.g., using a second machine learning model from the plurality of machine learning models), and the like. The plurality of machine learning models may be small machine learning models. A small machine learning model may include a number of parameters and / or a number of layers that is less than a threshold. The first machine learning model may include more parameters and / or layers than the plurality of machine learning models. The plurality of machine learning models may be generated by compressing the first machine learning model (e.g., using pruning, quantization, knowledge extraction, low-rank factorization, etc.) before, during, or after training the first machine learning model. In some examples, the computing device determines whether to train the plurality of machine learning models. The computing device may determine to train one or more of the plurality of machine learning models using the same training data used to train the first machine learning model, training data similar to that used to train the first machine learning model, or training data different from that used to train the first machine learning model. Alternatively, the plurality of machine learning models may be defined and trained independently (e.g., separately from the first machine learning model). In these instances, the multiple machine learning models may be of the same, similar, or different types as the first machine learning model (e.g., different models, different parameters, different types of layers, different algorithms, different training processes or iterations, etc.).
[0111] The computing device may use the same or similar feature vectors (derived from the one or more images) passed as input to the first machine learning model as input to the multiple machine learning models to produce a second prediction result, a third prediction result, etc. In other examples, the computing device may customize the feature vectors for the multiple machine learning models. Because the multiple machine learning models have fewer parameters and / or layers, the machine learning models may accept fewer features in the input feature vectors. The computing device may compress the feature vectors (e.g., using any of the aforementioned compression techniques) to reduce the number of input features.
[0112] At block 520, the computing device may select a second machine learning model from the plurality of machine learning models based on a comparison of the first prediction with the second prediction and the third prediction, etc. In some instances, the computing device may use one or more accuracy metrics and / or loss functions to compare the second prediction, the third prediction, etc. relative to the first prediction. For example, a depth map may represent each pixel of an image as a distance (e.g., a real number) between a location in an environment represented by the pixel and a camera. The computing device may perform a pixel-by-pixel comparison of each distance value of each pixel of the second prediction relative to each corresponding pixel in the first prediction (e.g., considered as a true value for comparison purposes). A loss function may be used to determine the comparison of the second prediction with the first prediction, the third prediction with the first prediction, etc. Examples of loss functions include, but are not limited to, adaptive robust loss, mean squared error, mean absolute error, cross entropy, weighted human disagreement rate (WHDR), combinations thereof, or similar functions. The computing device may select a machine learning model with the highest accuracy metric, the lowest error rate, the lowest loss, etc. from the plurality of machine learning models to process the plurality of images.
[0113] At block 524, the computing device may process the plurality of images using a second machine learning model (e.g., the machine learning model with the highest accuracy or lowest loss selected from block 520, etc.). In some instances, the computing device may process each of the plurality of images. In other instances, the computing device may process a portion of the plurality of images by sampling the plurality of images (e.g., such as every nth image, etc.). Returning to the monocular depth estimation example, the computing device may use a second machine learning model and the plurality of images to generate a sequence of depth maps. Because the second machine learning model is a small machine learning model, monocular depth estimation can be performed in near real time (e.g., using a live camera stream, dynamically or programmatically generated images from a video game, etc.).
[0114] In some examples, the model selection process may be repeated to ensure that the second machine learning model remains the most efficient machine learning model for processing the plurality of images. The model selection process may be re-executed at regular time intervals upon detection of an event, upon detection of user input, after execution of a predetermined iteration of the second machine learning model, upon detection of a change in one or more characteristics of the plurality of images (e.g., such as a change in average pixel value, etc.), a combination of the above, or in similar circumstances. The computing device may continually ensure that the most efficient machine learning model is used to process the plurality of images (e.g., a machine learning model with the highest accuracy, lowest error rate, lowest loss, etc. when processing the plurality of images).
[0115] The model selection process can be applied to various machine learning models to determine efficient ways to process completely different data sets. In this way, the techniques described in this article can be applied to deep neural networks (as previously described) as well as any other type of machine learning model or data set.
[0116] Figure 6 An exemplary computing device according to various aspects of the present disclosure is shown. For example, computing device 600 may implement any of the systems or methods described herein. In some instances, computing device 600 may be a component of a media device or included in a media device. The components of computing device 600 are shown as being in electrical communication with each other using a connector 606 such as a bus. Exemplary computing device 600 includes a processor (e.g., a CPU, a processor, or a similar device) 604 and a connector 606 (e.g., such as a bus or a similar device), and connector 606 is configured to couple components of computing device 600 (e.g., but not limited to, memory 620, read only memory (ROM) 618, random access memory (RAM) 616, and / or storage device 608) to processor 604.
[0117] The computing device 600 may include a cache 602 of high-speed memory that is directly connected to, adjacent to, or integrated within the processor 604. The computing device 600 may copy data from the memory 620 and / or storage device 608 to the cache 602 for faster access by the processor 604. In this way, the cache 602 may provide a performance boost that avoids delays while the processor 604 waits for data. Alternatively, the processor 604 may access data directly from the memory 620, ROM 618, RAM 616, and / or storage device 608. The memory 620 may include various types of homogeneous or heterogeneous memory (e.g., such as, but not limited to, magnetic memory, optical memory, solid-state memory, etc.).
[0118] The storage device 608 may include one or more non-transitory computer-readable media, such as volatile memory and / or non-volatile memory. The non-transitory computer-readable media may store instructions and / or data that may be accessed by the computing device 600. The non-transitory computer-readable media may include, but are not limited to, a magnetic cassette, a hard-disk drive (HDD), a flash memory, a solid-state memory device, a digital versatile disk, a cartridge, an optical disk, a random access memory (RAM) 616, a read-only memory (ROM) 618, a combination thereof, or the like.
[0119] The storage device 608 may store one or more services (e.g., service 1 610, service 2 612, and service 3 614) that may be executed by the processor 604 and / or other electronic hardware. The one or more services include instructions that may be executed by the processor 604 to perform the following operations: perform operations, such as any of the techniques, steps, processes, blocks, and / or operations described herein; control the operation of devices that communicate with the computing device 600; the operation of the processor 604 and / or any special purpose processor; a combination thereof; or the like. The processor 604 may be a system on a chip (SOC) that includes one or more cores or processors, a bus, a memory, a clock, a memory controller, a cache, other processor components, and / or the like. Multi-core processors may be symmetric or asymmetric.
[0120] The computing device 600 may include one or more input devices 622, which may represent any number of input mechanisms, such as a microphone, a touch-sensitive screen for graphic input, a keyboard, a mouse, motion input, voice, a media device, a sensor, a combination thereof, or the like. The computing device 600 may include one or more output devices 624 that output data to a user. Such output devices 624 may include, but are not limited to, a media device, a projector, a television, a speaker, a combination thereof, or the like. In some examples, a multimodal computing device may enable a user to provide multiple types of input to communicate with the computing device 600. The communication interface 626 may be configured to manage user input and computing device output. The communication interface 626 may also be configured to manage communications with a remote device (e.g., establishing a connection, receiving / transmitting communications, etc.) via one or more communication protocols and / or via one or more communication media (e.g., wired, wireless, etc.).
[0121] The computing device 600 is not limited to Figure 6The computing device 600 may include other components not shown and / or may omit components shown.
[0122] The following examples illustrate various aspects of the present disclosure. As used below, any reference to a series of examples will be understood as a separate reference to each of these examples (e.g., "Example 1 to Example 4" should be understood as "Example 1, Example 2, Example 3, or Example 4").
[0123] Example 1 is a computer-implemented method, which includes: receiving multiple images; selecting one or more images from the multiple images; processing the one or more images using a first machine learning model to produce a first prediction result; processing the one or more images using multiple machine learning models to produce at least a second prediction result and a third prediction result, wherein the first machine learning model is larger than the multiple machine learning models; selecting a second machine learning model from the multiple machine learning models based on a comparison of the first prediction result with the at least second prediction result and the third prediction result; and processing the multiple images using the second machine learning model.
[0124] Example 2 is a computer-implemented method described in any of Examples 1 and Examples 3 to 8, wherein selecting a second machine learning model from the multiple machine learning models includes: generating a first accuracy value for the first prediction result, a second accuracy value for the second prediction result, and a third accuracy value for the third prediction result; comparing the first accuracy value, the second accuracy value, and the third accuracy value, wherein the second machine learning model is selected based on the second accuracy value being higher than the first accuracy value and the third accuracy value.
[0125] Example 3 is a computer-implemented method described in any of Examples 1 to 2 and Examples 4 to 8, wherein selecting a second machine learning model includes: using a loss function to generate a first value for a first prediction result, a second value for a second prediction result, and a third value for a third prediction result; comparing the first value, the second value, and the third value, wherein the second machine learning model is selected based on the second value being higher than the first value and the third value.
[0126] Example 4 is the computer-implemented method of any of Examples 1 to 3 and 5 to 8, wherein the second machine learning model is configured to generate a depth estimate map for an image in the plurality of images.
[0127] Example 5 is the computer-implemented method of any of Examples 1 to 4 and Examples 6 to 8, wherein the second machine learning model is configured to perform semantic segmentation on an image in the plurality of images.
[0128] Example 6 is the computer-implemented method of any of Examples 1 to 5 and Examples 7 to 8, wherein the second machine learning model is configured to perform instance segmentation using an image from the plurality of images.
[0129] Example 7 is the computer-implemented method of any of Examples 1 to 6 and 8, wherein the plurality of machine learning models are deep neural networks.
[0130] Example 8 is the computer-implemented method of any of Examples 1 to 7, wherein the first machine learning model includes more layers than each of the multiple machine learning models.
[0131] Example 9 is a system comprising: one or more processors; and a non-transitory computer-readable medium storing instructions that, when executed by the one or more processors, cause the one or more processors to perform the operations described in any of Examples 1 to 8.
[0132] Example 10 is a non-transitory computer-readable medium storing instructions that, when executed by one or more processors, cause the one or more processors to perform the operations described in any of Examples 1 to 8.
[0133] The term "computer-readable medium" includes, but is not limited to, portable or non-portable storage devices, optical storage devices, and various other media capable of storing, containing or carrying instructions and / or data. Computer-readable media may include non-transitory media in which data may be stored in a form that does not include carrier waves and / or electronic signals. Examples of non-transitory media may include, but are not limited to, magnetic disks or tapes, optical storage media (such as compact disks (CDs) or digital versatile disks (DVDs)), flash memory, memory, or memory devices. Computer-readable media may store code and / or machine-executable instructions on it, and the machine-executable instructions may represent any combination of processes, functions, subroutines, programs, routines, subroutines, modules, software packages, classes, or instructions, data structures, or program statements. Code fragments may be coupled to another code fragment or hardware circuit by passing and / or receiving information, data, independent variables, parameters, or memory contents. Information, independent variables, parameters, data, etc. may be passed, forwarded, or transmitted by any suitable means (including memory sharing, message passing, token passing, network transmission, or the like).
[0134] Some portions of this specification describe examples in terms of algorithms and symbolic representations of operations on information. These operations, although described functionally, computationally, or logically, can be implemented by computer programs or equivalent circuits, microcodes, etc. In addition, arrangements of operations may be referred to as modules without loss of generality. The described operations and their associated modules may be implemented in software, firmware, hardware, or any combination thereof.
[0135] Any of the steps, operations or processes described herein may be implemented or implemented using one or more hardware or software modules alone or in combination with other devices. In some examples, a software module may be implemented using a computer-readable medium storing computer program code, which may be executed by a processor to implement any or all of the steps, operations or processes described.
[0136] Some examples may involve devices or systems for implementing any or all of the steps, operations, or processes described. The devices or systems may be specially constructed for the desired purpose, and / or the devices or systems may include a general-purpose computing device that is selectively activated or reconfigured by a computer program stored in a memory of the computing device. The memory may be or include a non-transitory tangible computer-readable storage medium, or any type of medium suitable for storing electronic instructions (which may be coupled to a bus). In addition, any computing system mentioned in the specification may include a single processor or multiple processors.
[0137] Although the subject matter has been described in detail for specific examples, it should be understood that those skilled in the art can easily generate changes, modifications and equivalents of such embodiments based on the understanding of the foregoing. Many specific details are set forth herein to provide a thorough understanding of the claimed subject matter. However, those skilled in the art will understand that the claimed subject matter can be practiced without these specific details. In other examples, methods, devices or systems known to those of ordinary skill in the art have not been described in detail to avoid obscuring the claimed subject matter. Therefore, the present disclosure has been presented for purposes of example and not limitation, and does not exclude the inclusion of such modifications, modifications and / or additions to the subject matter, which will be apparent to those of ordinary skill in the art.
[0138] For the purpose of explanation, in some examples, the present disclosure may be presented as including separate functional blocks, which include functional blocks including devices, device components, steps or routines in methods implemented with software, or combinations of hardware and software. Additional functional blocks other than those shown in the figures and / or described herein may also be used. For example, circuits, systems, networks, processes, and other components may be shown as components in block diagram form to avoid obscuring the embodiments with unnecessary details. In other examples, in order to avoid obscuring the embodiments, the well-known circuits, processes, algorithms, structures, and techniques shown may not have unnecessary details.
[0139] Individual examples may be described herein as processes or methods, which may be depicted as flow charts, flow diagrams, data flow diagrams, structure diagrams, or block diagrams. Although a flow chart may describe an operation as a sequential process, many of the operations may be performed in parallel or simultaneously. In addition, the order of the operations may be rearranged. When the operations of a process are completed, the process is terminated, but may have additional steps that are not shown. A process may correspond to a method, function, process, subroutine, subprogram, etc. When a process corresponds to a function, its termination may correspond to returning the function to a calling function or a main function.
[0140] The processes and methods according to the above examples may be implemented using computer executable instructions stored in or otherwise obtained from a computer readable medium. Such instructions may include, for example, instructions and data that cause or otherwise configure a general purpose computer, a special purpose computer, or a processing device to perform a particular function or group of functions. Portions of the computer resources used may be accessed via a network. Computer executable instructions may be, for example, binary, intermediate format instructions, such as assembly language, firmware, source code, etc.
[0141] The apparatus for implementing the methods and systems described herein may include hardware, software, firmware, middleware, microcode, hardware description language, or any combination thereof, and may take any of a variety of form factors. When implemented in software, firmware, middleware, or microcode, program code or code fragments (e.g., computer program products) for performing the necessary tasks may be stored in a computer-readable or machine-readable medium. The program code may be executed by a processor, which may include one or more processors, such as but not limited to one or more digital signal processors (DSPs), general-purpose microprocessors, application-specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), or other equivalent integrated or discrete logic circuit systems. Such processors may be configured to implement any of the techniques described in this disclosure. The processor may be a microprocessor, a conventional processor, a controller, a microcontroller, a state machine, or a similar device. The processor may also be implemented as a combination of computing components (e.g., a combination of a DSP and a microprocessor, multiple microprocessors, one or more microprocessors combined with a DSP core, or any other such configuration). Thus, the term "processor" as used herein may refer to any of the foregoing structures, any combination of the foregoing structures, or any other structure or device suitable for implementing the techniques described herein. The functionality described herein may also be implemented in a peripheral device or add-in card. As a further example, such functionality may also be implemented in different chips or different processes executed in a single device on a circuit board.
[0142] In the foregoing description, various aspects of the present disclosure are described with reference to specific examples of the present disclosure, but those skilled in the art will recognize that the present disclosure is not limited thereto. Therefore, although the illustrative examples of the present disclosure have been described in detail herein, it should be understood that the inventive concept can be implemented and adopted in different ways, and the appended claims are intended to be interpreted as including such variations. The various features and aspects disclosed above may be used alone or in any combination. In addition, without departing from the broader spirit and scope of the present disclosure, the examples may be used in any number of environments and applications outside of the environments and applications described herein. Therefore, the present disclosure and the figures should be regarded as illustrative rather than restrictive.
[0143] Various exemplary logic blocks, modules, circuits and algorithmic steps described in conjunction with the disclosed embodiments herein may be implemented as electronic hardware, computer software, firmware or a combination thereof. In order to clearly illustrate this interchangeability of hardware and software, various exemplary components, blocks, modules, circuits and steps have been generally described above according to the functions of various exemplary components, blocks, modules, circuits and steps. Whether such functions are implemented as hardware or software depends on specific applications and the design constraints to the entire system. A skilled person may implement the described functions in different ways for each specific application, but such implementation details decisions should not be interpreted as causing deviations from the scope of the present application.
[0144] Unless otherwise specifically stated, it should be understood that discussions throughout this specification utilizing terms such as "processing," "computing," "calculating," "determining," and "identifying," or similar terms, refer to actions or processes of a computing device (e.g., one or more computers or similar one or more electronic computing devices) that manipulate or transform data represented as physical electronic or magnetic quantities within a memory, register, or other information storage device, transmission device, or media device of a computing platform. As used herein, "suitable for" or "configured to" is intended to be open and inclusive language, and does not exclude devices that are suitable for or configured to perform additional tasks or steps. In addition, the use of "based on" is intended to be open and inclusive, because a process, step, calculation, or other action that is "based on" one or more referenced conditions or values may actually be based on additional conditions or values beyond those referenced conditions or values. The headings, lists, and numbers included herein are for ease of explanation only and are not meant to be limiting.
[0145] The foregoing detailed description of the technology has been presented for purposes of illustration and description. It is not intended to be exhaustive or to limit the technology to the precise form disclosed. Many modifications and variations may be made in light of the above teachings. The embodiments described were selected in order to best illustrate the principles of the technology, its practical application, and to enable others skilled in the art to utilize the technology in various embodiments and with various modifications as are suitable for the particular use contemplated. It is intended that the scope of the technology be defined by the claims.
Claims
1. A method for performing monocular depth estimation, comprising: receiving a plurality of images; selecting one or more images from the plurality of images; Processing the one or more images using a first machine learning model to generate a first prediction result; Processing the one or more images using a plurality of machine learning models to generate at least a second prediction result and a third prediction result, wherein the first machine learning model is larger than the plurality of machine learning models; selecting a second machine learning model from the plurality of machine learning models based on a comparison of the first prediction result with at least the second prediction result and the third prediction result; as well as The plurality of images are processed using the second machine learning model.
2. The computer-implemented method of claim 1 , wherein selecting the second machine learning model from the plurality of machine learning models comprises: generating a first accuracy value of the first prediction result, a second accuracy value of the second prediction result, and a third accuracy value of the third prediction result; as well as The first accuracy value, the second accuracy value, and the third accuracy value are compared, wherein the second machine learning model is selected based on the second accuracy value being higher than the first accuracy value and the third accuracy value.
3. The computer-implemented method of claim 1 , wherein selecting the second machine learning model comprises: Using a loss function to generate a first value of the first prediction result, a second value of the second prediction result, and a third value of the third prediction result; as well as The first value, the second value, and the third value are compared, wherein the second machine learning model is selected based on the second value being higher than the first value and the third value.
4. The computer-implemented method of claim 1 , wherein the second machine learning model is configured to generate a depth estimate map for an image in the plurality of images.
5. The computer-implemented method of claim 1, wherein the second machine learning model is configured to perform semantic segmentation on an image in the plurality of images.
6. The computer-implemented method of claim 1, wherein the second machine learning model is configured to perform instance segmentation using an image from the plurality of images.
7. The computer-implemented method of claim 1, wherein the plurality of machine learning models are deep neural networks.
8. The computer-implemented method of claim 1 , wherein the first machine learning model comprises more layers than each of the plurality of machine learning models.
9. A system for performing monocular depth estimation, comprising: one or more processors; as well as A non-transitory computer-readable medium storing instructions, which, when executed by the one or more processors, cause the one or more processors to perform the steps of any one of claims 1 to 8.
10. A non-transitory computer-readable medium storing instructions which, when executed by one or more processors, cause the one or more processors to perform the steps of any one of claims 1 to 8.
