Secure-based Prediction Devices, Systems, and Methods

Abstract

The present invention provides a security-based prediction device, system, and method. A machine learning hardware accelerator (MLHA) includes a main classifier (MC) module, at least one monitoring classifier (GC) module, and a final prediction class decision module. The MC module predicts an MC prediction class based on input data and includes a pre-trained machine learning main classifier (MLMC) having at least one security-critical (SC) class and a plurality of non-SC classes. Each monitoring classifier (GC) module is associated with an SC class and predicts a GC prediction class based on the input data. Each GC module includes a pre-trained machine learning monitoring classifier (MLGC) having two classes, the two classes including the associated SC class and a residual class, the residual class including any non-associated SC classes and the plurality of non-SC classes. The decision module determines and outputs a final prediction class based on the MC prediction class and each GC prediction class.

Description

Technical Field

[0001] The present disclosure relates to computer systems. More specifically, the present disclosure relates to a security-based prediction device, system, and method. Background Art

[0002] The present disclosure relates to computer systems. More specifically, the present disclosure relates to a security-based prediction device, system, and method.

[0003] Prediction is an essential element of many applications, including machine learning (ML) models, artificial neural networks (ANNs), convolutional neural networks (CNNs), recurrent neural networks (RNNs), binary neural networks (BNNs), support vector machines (SVMs), decision trees, naive Bayes, etc. ML models implemented in hardware can be used in safety-critical systems to perform critical classification tasks, especially in autonomous vehicles, robotic medical devices, etc. However, the inherent predictive nature of ML models can never guarantee 100% prediction accuracy, which occasionally leads to incorrect predictions. ML models implemented in hardware are also prone to hard or soft errors, which can exacerbate prediction accuracy or lead to fatal events. For example, a transient fault in a flip-flop in the output layer of a neural network can cause the ML inference hardware to predict an incorrect classification.

[0004] Safety-critical electronic systems must ensure functional safety, which is defined as the absence of unacceptable risk caused by hazards, where a hazard is caused by a fault that results in a failure of the electronic system. The ultimate goal of functional safety is to prevent death or injury to people due to failures of electronic systems used in intelligent autonomous devices (such as autonomous vehicles or robots). Currently, safety-critical systems include pattern recognition hardware that heavily relies on predictive ML algorithms to identify patterns of surrounding objects and people. Functional safety must also handle non-fault errors caused by incorrect predictions resulting from ML algorithms running on the hardware. Summary of the Invention

[0005] In one aspect, an embodiment of the present application provides a machine learning hardware accelerator MLHA. The MLHA includes: a main classifier MC module. The MC module includes a pre-trained machine learning main classifier MLMC having at least one safety-critical SC category and a plurality of non-SC categories, and is configured to predict an MC prediction category based on input data including sensor data and output the MC prediction category; at least one monitoring classifier GC module associated with each SC category. Each GC module includes a pre-trained machine learning monitoring classifier MLGC having two categories, the two categories including the associated SC category and a residual category, the residual category including any non-associated SC categories and a plurality of non-SC categories. Each GC module is configured to predict a GC prediction category based on the input data and output the GC prediction category; and a final prediction category decision module. The final prediction category decision module is coupled to the MC module and each GC module, and is configured to: receive the MC prediction category and each GC prediction category, determine a final prediction category based on the MC prediction category and each GC prediction category, and output the final prediction category, wherein an autonomous process or a semi-autonomous process is controlled based on the final prediction category.

[0006] In another aspect, an embodiment of the present application provides a hardware-based method. The hardware-based method includes: predicting, by a main classifier MC module, an MC prediction category based on input data including sensor data, the MC module including a pre-trained machine learning main classifier MLMC having at least one safety-critical SC category and a plurality of non-SC categories; predicting, by each monitoring classifier GC module associated with each SC category, a GC prediction category based on the input data, each GC module including a pre-trained machine learning monitoring classifier MLGC having two categories, the two categories including the associated SC category and a residual category, the residual category including any non-associated SC categories and a plurality of non-SC categories; determining, by a final prediction category decision module, a final prediction category based on the MC prediction category and each GC prediction category; outputting, by the final prediction category decision module, the final prediction category; and controlling an autonomous process or a semi-autonomous process based on the final prediction category.

[0007] In yet another aspect, an embodiment of the present application provides a system, including: a memory configured to store input data including sensor data; a machine learning hardware accelerator (MLHA) coupled to the memory and including: a main classifier (MC) module including a pre-trained machine learning main classifier (MLMC) having at least one safety-critical (SC) class and a plurality of non-SC classes, and configured to predict an MC prediction class based on the input data and output the MC prediction class; a monitoring classifier (GC) module associated with each SC class, each GC module including a pre-trained machine learning monitoring classifier (MLGC) having two classes, the two classes including the associated SC class and a residual class, the residual class including any non-associated SC classes and a plurality of non-SC classes, each GC module being configured to predict a GC prediction class based on the input data and output the GC prediction class; and a final prediction class decision module coupled to the MC module and each GC module and configured to: receive the MC prediction class and each GC prediction class, determine a final prediction class based on the MC prediction class and each GC prediction class, and output the final prediction class; and a processor coupled to the memory and the MLHA and configured to: transmit the input data to the MLHA, receive the final prediction class from the MLHA, and control an autonomous process or a semi-autonomous process based on the final prediction class. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] Figure 1 A block diagram of a system having a machine learning hardware accelerator (MLHA) according to one embodiment of the present disclosure is depicted.

[0009] Figure 2A 、 Figure 2B 、 Figure 2C and Figure 2D A block diagram of the MLHA according to an embodiment of the present disclosure is depicted.

[0010] Figure 3 A flowchart depicting functions for a safety-based prediction process according to one embodiment of the present disclosure is depicted.

[0011] Figure 4A and Figure 4B A decision flowchart for determining a final prediction class according to one embodiment of the present disclosure is depicted.

[0012] Figure 5A A block diagram of a training system for a machine learning main classifier (MLMC) according to one embodiment of the present disclosure is depicted.

[0013] Figure 5B A block diagram of a training system for a machine learning monitoring classifier (MLGC) according to one embodiment of the present disclosure is depicted. Detailed Implementation Manner

[0014] Embodiments of the present disclosure will now be described with reference to the accompanying drawings, in which like reference numerals always refer to like components.

[0015] Embodiments of the present disclosure provide a system, device, and method that advantageously include safety-critical attributes within a category predicted by a classification-based ML model to improve the resilience of implementing an ML model in a safety-critical system. More specifically, a safety-critical (SC) category is a category associated with an entity, such as a person, animal, object, etc., that will be identified by a predictive ML algorithm running on a safety-critical system, where an incorrect prediction of the category may result in death or injury.

[0016] In one embodiment, the MLHA includes a main classifier (MC) module, a monitoring classifier (GC) module associated with each SC category, and a final prediction category (FPC) decision module.

[0017] The MC module includes a pre-trained machine learning main classifier (MLMC) having at least one SC category and a plurality of non-SC categories. The MC module is configured to predict an MC prediction category based on input data and output the MC prediction category.

[0018] Each GC module includes a pre-trained machine learning monitoring classifier (MLGC) having two categories, which include the associated SC category and a residual category, and the residual category includes any category other than the associated SC category, that is, any non-associated SC category and a plurality of non-SC categories. Each GC module is configured to predict a GC prediction category based on input data and output the GC prediction category.

[0019] The FPC decision module is coupled to the MC module and each GC module. The FPC decision module is configured to: receive the MC prediction category and each GC prediction category, determine a final prediction category based on the MC prediction category and each GC prediction category, and output the final prediction category.

[0020] An ML model is a mathematical model trained by a learning process to generate an output (such as a monitoring signal) from an input (such as a feature vector). Neural networks (such as ANN, CNN, RNN, BNN, etc.), support vector machines, naive Bayes, and K-nearest neighbor classifiers are types of ML models. For example, a supervised learning process uses fully labeled training data including known input-output pairs to train an ML model. A semi-supervised or weakly supervised learning process uses incomplete training data to train an ML model, and the incomplete data is a small amount of labeled data (i.e., input-output pairs) and a large amount of unlabeled data (only inputs). An unsupervised learning process uses unlabeled data (i.e., only inputs) to train an ML model.

[0021] An ANN uses a network of interconnected nodes trained through the above learning process to model the relationship between input data or signals and output data or signals. The nodes are arranged into various layers (including, for example, an input layer, one or more hidden layers, and an output layer). The input layer receives input data (such as, for example, image data), and the output layer generates output data (such as, for example, the probability that the image data contains a known object). Each hidden layer provides at least a partial transformation of the input data to the output data. A deep neural network (DNN) has multiple hidden layers in order to model complex, non-linear relationships between input data and output data.

[0022] In a fully connected feedforward ANN, each node is connected to all nodes in the previous layer, as well as to all nodes in the next layer. For example, for a simple ANN having an input layer, a hidden layer, and an output layer, each input layer node is connected to each hidden layer node, each hidden layer node is connected to each input layer node and each output layer node, and each output layer node is connected to each hidden layer node. For a more complex ANN having multiple hidden layers, each additional hidden layer is similarly connected to all nodes in the previous hidden layer and all nodes in the next hidden layer. Each connection has a weight value, and each node has an activation function, such as a linear function, step function, sigmoid function, hyperbolic tangent function, rectified linear unit (ReLu) function, etc., that determines the output of the node based on the weighted sum of the inputs to the node. The input data propagates from the input layer nodes through the corresponding connection weights to the hidden layer nodes, and then propagates through the corresponding connection weights to the output layer nodes.

[0023] More specifically, at each input node, the input data is provided to the activation function of the node, and then the output of the activation function is provided as an input data value to each hidden layer node. At each hidden layer node, the input data values received from each input layer node are multiplied by the corresponding connection weights, and the resulting products are summed or accumulated into an activation signal value that is provided to the activation function of the node. Then the output of the activation function is provided as an input data value to each output layer node. At each output layer node, the output data values received from each hidden layer node are multiplied by the corresponding connection weights, and the resulting products are summed or accumulated into an activation signal value that is provided to the activation function of the node. Then the output of the activation function is provided as the output data. Additional hidden layers can be similarly configured to process the data.

[0024] Training an ANN involves optimizing the connection weights between the nodes by minimizing the prediction error of the output data until the ANN achieves a specific accuracy level. One method is error backpropagation or backward propagation, which iteratively and recursively determines the gradient descent with respect to the connection weights, and then adjusts the connection weights to improve the performance of the network.

[0025] A multi-layer perceptron (MLP) is a fully connected ANN with an input layer, an output layer, and one or more hidden layers, and uses a non-linear activation function to classify linearly inseparable data. The MLP can be used in natural language processing applications such as machine translation, speech recognition, etc. A CNN is a variant of the MLP with an input layer, an output layer, and multiple hidden layers, including a series of convolutional layers, followed by pooling layers, fully connected layers, and normalization layers. Each convolutional layer applies a sliding dot product or cross-correlation to the input data and then transfers the result to the next layer. To exploit the spatial local patterns in the input data, some hidden layers of the CNN may not be fully connected. Instead, the nodes in these hidden layers are connected to a small number of nodes in the previous hidden layer. The locally connected hidden layers are combined with the fully connected hidden layers to form the CNN architecture. The CNN can be used for classification, recognition, prediction, etc., such as in applications like machine translation, speech recognition, etc., such as image classification, speech recognition, etc. Other ANNs include long short-term memory (LSTM), sequence-to-sequence models including encoder RNN and decoder RNN, shallow neural networks, etc.

[0026] The implementation of many ML inference hardware uses classification-based ML models such as ANNs, CNNs, RNNs, etc. to perform pattern recognition (e.g., images, speech, activities, etc.). A typical classification-based ML model predicts a finite set of classes. For example, in the case of an autonomous vehicle, the set of image classes may include, for example, "pedestrian", "bicycle", "vehicle", "animal", "traffic sign", "traffic light", "intersection", "exit", "junk", etc. Some of these classes are extremely critical for real-time prediction; otherwise, incorrect predictions may lead to injury or death. For example, "pedestrian", "bicycle", "vehicle", etc. can be defined as safety-critical classes, while "animal", "traffic sign", "traffic light", "intersection", "exit", "junk", etc. are not defined as safety-critical classes.

[0027] The ML inference hardware can also implement other types of ML models, such as support vector machines, decision trees, Gaussian naive Bayes, etc.

[0028] Figure 1 A block diagram of a system with an MLHA according to one embodiment of the present disclosure is depicted.

[0029] The computer 100 includes a bus 110 coupled to one or more processors 120, a memory 130, an I / O interface 140, a display interface 150, one or more communication interfaces 160, and one or more MLHAs 200. Generally, the I / O interface 140 is coupled to I / O devices 142, sensors 144, and actuators 146 using a wired or wireless connection, the display interface 150 is coupled to a display 152, and the communication interface 160 is connected to a network 162 using a wired or wireless connection.

[0030] The bus 110 is a communication system that transfers data between the processor 120, the memory 130, the I / O interface 140, the display interface 150, the communication interface 160, the MLHA 200, and other components not depicted Figure 1 herein. A power connector 112 is coupled to the bus 110 and a power supply (not shown).

[0031] The processor 120 includes one or more general-purpose or special-purpose microprocessors that execute instructions to perform control, computing, input / output, and other functions of the computer 100. The processor 120 may include a single integrated circuit such as a microprocessing device, or multiple integrated circuit devices and / or circuit boards that work together to implement the functions of the processor 120. Additionally, the processor 120 may execute computer programs or modules stored in the memory 130, such as an operating system 132, software modules 134, etc. For example, the software module 134 may include autonomous vehicle applications, robotic applications (e.g., a robot performing a surgical procedure, working with humans in a collaborative environment), etc.

[0032] Generally, a storage element or the memory 130 stores instructions for execution by the processor 120, as well as data. The memory 130 may include a variety of non-transitory computer-readable media accessible by the processor 120. In various embodiments, the memory 130 may include volatile and non-volatile media, non-removable media, and / or removable media. For example, the memory 130 may include any combination of random access memory (RAM), dynamic RAM (DRAM), static RAM (SRAM), read-only memory (ROM), flash memory, cache memory, and / or any other type of non-transitory computer-readable media.

[0033] Memory 130 includes various components for retrieving, presenting, modifying, and storing data. For example, memory 130 stores software modules that provide functionality when executed by processor 120. The software modules include operating system 132, which provides operating system functionality for computer 100. Software module 134 provides various functions as described above. Data 136 may include data associated with operating system 132, software module 134, and so on.

[0034] I / O interface 140 is configured to transfer and / or receive data from I / O device 142, sensor 144, and actuator 146. I / O interface 140 enables the connection between processor 120 and I / O device 142 by encoding data to be sent from processor 120 to I / O device 142 and decoding data received from I / O device 142 for processor 120. Generally, data can be sent via wired and / or wireless connections. For example, I / O interface 140 may include one or more wired communication interfaces (such as USB, Ethernet, etc.) and / or one or more wireless communication interfaces coupled to one or more antennas (such as WiFi, Bluetooth, cellular, etc.).

[0035] Generally speaking, I / O device 142 provides input to and / or output from computer 100. As discussed above, I / O device 142 is operably connected to computer 100 using wired and / or wireless connections. I / O device 142 may include a local processor coupled to a communication interface configured to communicate with computer 100 using wired and / or wireless connections. For example, I / O device 142 may include a keyboard, mouse, touchpad, joystick, and so on.

[0036] Display interface 150 is configured to transfer image data from computer 100 to monitor or display 152.

[0037] Communication interface 160 is configured to transfer data to and from network 162 using one or more wired and / or wireless connections. Network 162 may include one or more local area networks, wide area networks, the Internet, etc., and may execute various network protocols, such as, for example, wired and / or wireless Ethernet, Bluetooth, etc. Network 162 may also include various combinations of wired and / or wireless physical layers, such as copper wire or coaxial cable networks, fiber optic networks, Bluetooth wireless networks, WiFi wireless networks, CDMA cellular wireless networks, FDMA cellular wireless networks, and TDMA cellular wireless networks, etc.

[0038] The MLHA 200 is configured to execute classification-based ML models, such as ANN, CNN, RNN, SVM, Naive Bayes, etc., to support software module 134. Generally, the MLHA 200 can be a graphics processing unit (GPU), a digital signal processor (DSP), a neural processing unit (NPU), such as an ARM machine learning processor or a custom-designed processing engine that executes underlying ML inference algorithms. The MLHA 200 can also be an application-specific integrated circuit (ASIC) or a custom processor implemented in a field-programmable gate array (FPGA), etc. Thus, the MLHA 200 can include one or more processors, microprocessors, microcontrollers, etc., one or more memory blocks including RAM, ROM, EEPROM, flash memory, etc., integrated circuits, programmable circuits, matrix multiplication accelerators, etc. For example, the matrix multiplication accelerator can include an N×N systolic array of multiply-accumulate (MAC) cells, as well as support memory registers and control circuits that support fast, dense matrix-matrix and matrix-vector operations.

[0039] Embodiments of the present disclosure advantageously include SC attributes for each class, each class being predicted by a classification-based ML model hosted by the MLHA 200. In other words, each class is classified as an SC class or a non-SC class. This safety-critical class architecture improves the resilience of the MLHA 200 in safety-critical systems, increases the prediction accuracy of classes with safety-critical attributes set, and detects potential hardware-related errors. Classification-based ML models known, such as CNN, DNN, etc., are trained in such a way that all classes are treated the same. When a classification-based ML model is implemented in a hardware accelerator such as the MLHA 200, the overall class prediction accuracy is the main performance metric. However, when the MLHA 200 is used in a safety-critical system, the prediction of some classes (i.e., SC classes) is more critical than that of other classes (i.e., non-SC classes). Therefore, a classification-based ML model for a safety-critical system is trained in such a way that all classes are treated differently. Additionally, SC classes can have different levels of criticality of priority, for example, the "pedestrian" class can have a higher level of priority than the "traffic sign" class.

[0040] The classification-based ML model hosted by MLHA 200 includes an MC module and a GC module associated with each SC category. The MC module includes a pre-trained MLMC having at least one SC category and multiple non-SC categories. The MC module is configured to predict an MC prediction category based on input data and output the MC prediction category. Each GC module includes a pre-trained MLGC having two categories, the two categories including an associated SC category and a residual category. The residual category includes any non-associated SC category and multiple non-SC classes. Each GC module is configured to predict a GC prediction category based on input data and output the GC prediction category. In other words, each GC is a binary classifier that predicts or does not predict an associated SC category.

[0041] Importantly, each MLGC can be implemented using a simpler ML model than the MLMC. For example, the MLHA 200 can implement the MLMC as a CNN and each MLGC as a Naive Bayesian, etc. Each MLGC is trained using the same dataset as the MLMC, but the labels within the training data are collapsed to the associated SC class label (e.g., 1) or residual class label (e.g., 0). Because the MLGC solves a binary classification problem, the prediction accuracy can be as high as that of a CNN, while the cost of implementing the additional MLGC is minimal.

[0042] Figure 2A 、 Figure 2B 、 Figure 2C and Figure 2D Depicted is a block diagram of an MLHA according to an embodiment of the present disclosure.

[0043] Figure 2A An embodiment of the MLHA 200 is depicted including the MC module 210 , the GC module 220 , and the FPC decision module 230 .

[0044] The MC module 210 includes a pre-trained MLMC with one SC category (e.g., “pedestrian”) and several non-SC categories (e.g., “bicycle,” “vehicle,” “animal,” “traffic sign,” “traffic light,” “intersection,” “exit,” “garbage,” etc.). The MC module 210 is configured to predict an MC-predicted category based on an input and output the MC-predicted category.

[0045] The GC module 220 includes a pre-trained MLGC having two categories, the two categories including an associated SC category (e.g., “pedestrian”) and a residual category, the residual category including a non-SC category (e.g., “bicycle,” “vehicle,” “animal,” “traffic sign,” “traffic light,” “intersection,” “exit,” “trash,” etc.). The GC module 220 is configured to predict a GC predicted category based on input data and output the GC predicted category to the FPC decision module 230.

[0046] At each prediction, the same input is sent to the MC module 210 and the GC module 220, and the FPC decision module 230 determines the final prediction class, which will be described in more detail below.

[0047] Figure 2B An embodiment of the MLHA 200 including the MC module 210, the GC modules 220-1 to 220-N, and the FPC decision module 230 is depicted. Each of the GC modules 220-1 to 220-N is associated with a different SC class.

[0048] The MC module 210 includes a pre-trained MLMC having N numbers of SC classes and a number of non-SC classes. In one example, N is equal to 2, the SC classes are "pedestrian" and "bicycle", and the non-SC classes are "vehicle", "animal", "traffic sign", "traffic light", "intersection", "exit", "garbage", etc. The MC module 210 is configured to predict an MC prediction class based on the input and output the MC prediction class.

[0049] Each of the GC modules 220-1 to 220-N includes a pre-trained MLGC having two classes, the two classes including the associated SC class and the residual class, and the residual class including any non-associated SC classes and non-SC classes. In the above example, for the GC module 220-1, the associated SC class includes "pedestrian", and the residual class includes "bicycle", "vehicle", "animal", "traffic sign", "traffic light", "intersection", "exit", "garbage", etc. Similarly, for the GC module 220-N, the associated SC class includes "bicycle", and the residual class includes "pedestrian", "vehicle", "animal", "traffic sign", "traffic light", "intersection", "exit", "garbage", etc. Each of the GC modules 220-1 to 220-N is configured to predict a GC prediction class based on the input data and output the GC prediction class to the FPC decision module 230.

[0050] At each prediction, the same input is sent to the MC module 210 and the GC modules 220-1 to 220-N, and the FPC decision module 230 determines the final prediction class, which will be described in more detail below.

[0051] Figure 2C An embodiment of the MLHA 200 including the MC module 210, the GC modules 220-11 to 220-1 j 、the GC prediction class decision module 222-1 and the FPC decision module 230 is depicted. Each of the GC modules 220-11 to 220-1 j is associated with the same SC class.

[0052] The MC module 210 includes a pre-trained MLMC having one SC category (e.g., "pedestrian") and several non-SC categories (e.g., "bicycle", "vehicle", "animal", "traffic sign", "traffic light", "intersection", "exit", "junk", etc.). The MC module 210 is configured to predict an MC prediction category based on the input and output the MC prediction category.

[0053] Each GC module 220-11 to 220-1 j includes the same pre-trained MLGC having two categories, the two categories including the same associated SC category (e.g., "pedestrian") and a residual category, the residual category including non-SC categories (e.g., "bicycle", "vehicle", "animal", "traffic sign", "traffic light", "intersection", "exit", "junk", etc.). Each GC module 220-11 to 220-1 j is configured to predict a GC prediction category based on the input data and output the GC prediction category to the GC prediction category decision module 222-1.

[0054] At each prediction, the same input is sent to the MC module 210 and the GC modules 220-11 to 220-1 j , and the GC prediction category decision module 222-1 determines and outputs the final GC prediction category to the FPC decision module 230 that determines the final prediction category.

[0055] In one embodiment, when at least half of the categories in the GC prediction category are SC categories, the GC prediction category decision module 222-1 selects the SC category as the final GC prediction category; and when less than half of the categories in the GC prediction category are SC categories, the GC prediction category decision module 222-1 selects the residual category as the final GC prediction category.

[0056] Figure 2D An embodiment of the MLHA 200 including the MC module 210, several GC module groups, and the FPC decision module 230 is depicted. Due to the low area and low power requirements of each GC module 220, multiple voting redundant GC modules 220 for each SC category can be trained and implemented by the MLHA 200 to increase redundancy and fault tolerance.

[0057] The MC module 210 includes a pre-trained MLMC having N numbers of SC categories and several non-SC categories. In one example, N is equal to 2, the SC categories are "pedestrian" and "bicycle", and the non-SC categories are "vehicle", "animal", "traffic sign", "traffic light", "intersection", "exit", "junk", etc. The MC module 210 is configured to predict an MC prediction category based on the input and output the MC prediction category.

[0058] Each GC module group includes a number of GC modules and a GC prediction category decision module. For example, the first group of GC modules includes GC modules 220-11 to 220-1 j and GC prediction category decision module 222-1, while the last group of GC modules includes GC modules 220-N1 to 220-N k and GC prediction category decision module 222-N. Each group of GC modules is associated with a different SC category. For example, GC modules 220-11…220-1 j are associated with the "A" category, and GC modules 220-N1…220-N k are associated with the "N" category, etc. In the above example, the "A" category is "pedestrian", and the "N" category is "bicycle".

[0059] Each of GC modules 220-11 to 220-1 j includes the same pre-trained MLGC with two categories. The two categories include the same associated SC category ("A" category, such as "pedestrian") and a residual category. The residual category includes any non-associated SC category ("N" category, such as "bicycle") and non-SC categories (such as "vehicle", "animal", "traffic sign", "traffic light", "intersection", "exit", "garbage", etc.). Each of GC modules 220-11 to 220-1 j is configured to predict the GC prediction category based on the input data and output the GC prediction category.

[0060] Similarly, each of GC modules 220-N1 to 220-N k includes the same pre-trained MLGC with two categories. The two categories include the same associated SC category ("N" category, such as "bicycle") and a residual category. The residual category includes any non-associated SC category ("A" category, such as "pedestrian") and non-SC categories (such as "vehicle", "animal", "traffic sign", "traffic light", "intersection", "exit", "garbage", etc.). Each of GC modules 220-N1 to 220-N j is configured to predict the GC prediction category based on the input data and output the GC prediction category.

[0061] At each prediction, the same input is sent to the MC module 210 and all GC module groups, that is, GC modules 220-11 to 220-1 j …GC modules 220-N1 to 220-N k; all GC prediction category decision modules determine corresponding GC prediction categories, that is, GC prediction category decision module 222-1 determines and outputs the final GC prediction category ("A" category) to the FPC decision module 230, GC prediction category decision module 222-N determines and outputs the final GC prediction category ("N" category) to the FPC decision module 230, etc.; and the FPC decision module 230 determines the final prediction category, which will be described in more detail below.

[0062] In one embodiment, when at least half of the GC prediction categories are SC categories, the GC prediction category decision module 222-1 selects the SC category as the final GC prediction category; and when less than half of the GC prediction categories are SC categories, the GC prediction category decision module 222-1 selects the residual category as the final GC prediction category. Similarly, when at least half of the GC prediction categories are SC categories, the GC prediction category decision module 222-N selects the SC category as the final GC prediction category; and when less than half of the GC prediction categories are SC categories, the GC prediction category decision module 222-N selects the residual category as the final GC prediction category.

[0063] Figure 3 Depicts a flowchart presenting the functions for a security-based prediction process according to an embodiment of the present disclosure.

[0064] At 310, the MC prediction category is predicted by the MC module 210 based on the input data. As described above, the MC module 210 includes a pre-trained MLMC having at least one SC category and multiple non-SC categories.

[0065] At 320, the GC prediction category is predicted by each GC module 210-1 to 210-N associated with each SC category based on the input data. As described above, each GC module 210-1 to 210-N includes a pre-trained MLGC having two categories, the two categories including the associated SC category and the residual category, and the residual category including any non-associated SC categories and multiple non-SC categories.

[0066] At 330, the final prediction category is determined by the FPC decision module 230 based on the MC prediction category and each GC prediction category. In certain embodiments described above, each GC prediction category is the final GC prediction category output by the GC prediction category decision modules 222-1 to 222-N.

[0067] At 340, the final prediction category is output by the FPC decision module 230.

[0068] Figure 4A and Figure 4BDepicts a decision flow chart for determining a final predicted category according to an embodiment of the present disclosure.

[0069] At 410, the MC predicted category and each GC predicted category are received by the FPC decision module 230. In certain embodiments above, each final GC predicted category is received by the FPC decision module 230.

[0070] At 420, the FPC decision module 230 determines whether the MC predicted category is an SC category. If so, the process proceeds to 422; if not, the process proceeds to 450.

[0071] At 422, the FPC decision module 230 selects the GC module 220 associated with the SC category identified by the MC predicted category. In the example above, if the MC predicted category is "pedestrian", then the GC module 220-1 is selected. Similarly, if the MC predicted category is "bicycle", then the GC module 220-N is selected. And so on.

[0072] At 430, the FPC decision module 230 determines whether the GC predicted category from the selected GC module 220 is an SC category. If so, the process proceeds to 440; if not, the process proceeds to 470( Figure 4B ). In the example above, if the MC predicted category is "pedestrian" and the GC predicted category from the GC module 220-1 is "pedestrian", then the process proceeds to 440. Similarly, if the MC predicted category is "bicycle" and the GC predicted category from the GC module 220-1 is "bicycle", then the process proceeds to 440. And so on.

[0073] At 440, the FPC decision module 230 selects the MC predicted category (i.e., the SC category) for output.

[0074] In other words, when the MLMC has several SC categories, the MC predicted category is the first SC category, and the GC predicted category from the GC module associated with the first SC category is the first SC category, the FPC decision module 230 selects the first SC category as the final predicted category.

[0075] At 450, the MC predicted category is a non-SC category, and the FPC decision module 230 determines whether at least one GC predicted category includes an SC category. If so, the process proceeds to 460; if not, the process proceeds to 440, where the FPC decision module 230 selects the MC predicted category (i.e., the SC category) for output.

[0076] At 460, the FPC decision module 230 determines whether multiple GC prediction categories include the SC category. If so, the process proceeds to 462; if not, the process proceeds to 442, where the FPC decision module 230 selects the GC prediction category that includes the SC category for output.

[0077] At 462, the FPC decision module 230 sorts the multiple GC prediction categories based on the SC priority level, and the process proceeds to 444, where the FPC decision module 230 selects the GC prediction category with the highest SC priority level for output.

[0078] In other words, when the MC prediction category is one of the non - SC categories: when all GC prediction categories are residual categories, the FPC decision module 230 selects the MC prediction category as the final prediction category; when one of the GC prediction categories is the SC category and any remaining GC prediction categories are residual categories, the FPC decision module 230 selects the GC prediction category as the final prediction category; and when two or more of the GC prediction categories are SC categories, the FPC decision module 230 selects the GC prediction category with the highest SC priority level as the final prediction category.

[0079] At 470, the MC prediction category is the SC category, the GC prediction category from the GC module associated with the SC category is a residual category, and the FPC decision module 230 determines whether other GC prediction categories are available. If so, the process proceeds to 480; if not, the process proceeds to 440, where the FPC decision module 230 selects the MC prediction category (i.e., the SC category) for output.

[0080] At 480, the FPC decision module 230 determines whether at least one other GC prediction category includes the SC category. If so, the process proceeds to 490; if not, the process proceeds to 440, where the FPC decision module 230 selects the MC prediction category (i.e., the SC category) for output.

[0081] At 490, the FPC decision module 230 determines whether multiple GC prediction categories include the SC category. If so, the process proceeds to 492; if not, the process proceeds to 442, where the FPC decision module 230 selects the GC prediction category that includes the SC category for output.

[0082] At 492, the FPC decision module 230 sorts the multiple GC prediction categories based on the SC priority level, and the process proceeds to 444, where the FPC decision module 230 selects the GC prediction category with the highest SC priority level for output.

[0083] In other words, when the MC prediction category is the first SC category and the GC prediction category from the GC module associated with the first SC category is the residual category: when the GC prediction category from the GC module not associated with the first SC category is the residual category, the FPC decision module 230 selects the MC prediction category as the final prediction category; when the GC prediction category from one of the GC modules not associated with the first SC category is the second SC category and any remaining GC prediction categories are the residual category, the FPC decision module 230 selects the second SC category as the final prediction category; and when the GC prediction categories from two or more of the GC modules not associated with the first SC category are SC categories, the FPC decision module 230 selects the GC prediction category with the highest SC priority level as the final prediction category.

[0084] Generally speaking, after the architectures of the MLMC and each MLGC have been designed, the MLMC and each MLGC are presented in software to train the weights / parameters within various classification layers. The architectures include, for example, the input layer, hidden layer, and output layer of an ANN, the convolutional layer, pooling layer, fully connected layer, and normalization layer of a CNN, the fully connected layer and binary activation layer of a BNN, an SVM classifier, and so on. The resulting pre-trained MLMC and each pre-trained MLGC can be implemented by the MLHA 200 in several ways. For the MLHA 200 including one or more processors, microprocessors, microcontrollers, etc. (such as GPUs, DSPs, NPUs, etc.), the implementation of the pre-trained MLMC software and the implementation of each pre-trained MLGC software are adjusted and optimized to run on the local processor. In these examples, the MC module, GC module, and FPC decision module are software modules. For the MLHA 200 including programmable circuits (such as ASICs, FPGAs, etc.), the programmable circuit system is programmed to implement the implementation of the pre-trained MLMC software and the implementation of each pre-trained MLGC software. In these examples, the MC module, GC module, and FPC decision module are hardware modules. Regardless of the specific implementation, the MLHA 200 provides hardware-based acceleration for the MLMC and each MLGC.

[0085] Figure 5A A block diagram of a training system for a machine learning master classifier (MLMC) according to an embodiment of the present disclosure is depicted.

[0086] The training system 500 is a computer system that includes one or more processors, memory, etc., and executes one or more software modules that train the MLMC included in the MC module 210. The software modules include an MLMC module 510, a comparison module 512, and a learning module 514. Initially, the MLMC module 510 includes an untrained form of the MLMC included in the MC module 210. Generally, the MLMC includes one or more SC categories and a number of non-SC categories.

[0087] During each training cycle, the MLMC module 510 receives training data (input) and determines an MC prediction category based on the input; the comparison module 512 receives and compares the training data (expected category) and the MC prediction category, and outputs error data; and the learning module 514 receives the error data and the learning rate for all categories (i.e., categories A through N), and determines and sends a weight adjustment to the MLMC module 510.

[0088] In some embodiments, a single learning rate for all categories (i.e., categories A through N) can be used to train the MLMC. When the learning rate is high, the training time is long. Conversely, when the learning rate is low, the training time may be short, but the MLMC may be less accurate. In one embodiment, the learning module 514 provides a supervised learning process to train the MLMC using fully labeled training data that includes known input-output pairs. In another embodiment, the learning module 514 provides a semi-supervised or weakly supervised learning process to train the MLMC using incomplete training data, which is a small amount of labeled data (i.e., input-output pairs) and a large amount of unlabeled data (input only). In another embodiment, the learning module 514 provides an unsupervised learning process to train the MLMC using unlabeled data (i.e., input only).

[0089] Figure 5B A block diagram depicting a training process for a machine learning monitoring classifier (MLGC) according to one embodiment of the present disclosure is shown.

[0090] The training system 502 is a computer system that includes one or more processors, memory, etc., and executes one or more software modules that train each MLGC included in each GC module 220. The software modules include an MLGC module 520, a comparison module 522, and a learning module 524. Initially, the MLGC module 520 includes an untrained form of each MLGC included in each GC module 220. Each MLGC includes an associated SC category and a residual category, and the residual category includes any non-associated SC categories and non-SC categories.

[0091] During each training loop for each MLGC, the MLGC module 520 receives training data (input) and determines a GC prediction class based on the input; the comparison module 522 receives and compares the training data (expected class) and the GC prediction class, and outputs error data; and the learning module 524 receives the error data and a learning rate for an associated SC class (i.e., class A to class N) and determines and sends a weight adjustment to the MLGC module 520.

[0092] As described above, a single learning rate for all classes (i.e., class A to class N) can be used to train the MLMC. Advantageously, due to the simplicity of the MLGC, a learning rate higher than the MLMC learning rate can be used to train the MLGC. In some embodiments, a learning rate based on the priority level of the associated SC class can be used to train each MLGC associated with a particular SC class.

[0093] In one embodiment, the learning module 524 provides a supervised learning process to train the MLGC using fully labeled training data including known input-output pairs. In another embodiment, the learning module 524 provides a semi-supervised or weakly supervised learning process to train the MLGC using incomplete training data, which is a small amount of labeled data (i.e., input-output pairs) and a large amount of unlabeled data (only input). In another embodiment, the learning module 524 provides an unsupervised learning process to train the MLGC using unlabeled data (i.e., only input).

[0094] In one embodiment, the MLHA 200 is embedded in a "PlasticArmpit" sensor module. The "PlasticArmpit" sensor module includes 8 sensors, and each sensor provides 5-bit integer sensor data indicating odor intensity. The MLHA 200 includes an MC module 210 with an MLMC, a GC module 220 with an MLGC, and an FPC decision module 230.

[0095] The MLMC is a binary neural network (BNN) with binary (i.e., 1-bit) weights and activations and 8 features. The output of the MLMC is a class prediction from one of 5 different classes indicating odor intensity. The untrained MLMC is trained using PlasticArmpit sensor module training data; the average prediction accuracy of at least one set of training data is 91%. The pre-trained MLMC has an 8-node input stage, 20 hidden nodes, and 5 output nodes. The MLHA 200 implements the pre-trained MLMC in 65 nm.

[0096] The MLGC is a Feature Voting Classifier (FVC). In one embodiment, the MLGC is a Lookup Table (LUT)-based Naive Bayes classifier; in other embodiments, the MLGC can be a Decision Tree, Gaussian Naive Bayes, etc. The MLGC has 2 classes, including a single SC class (i.e., the "A" class, one of the 5 odor intensities) and a residual class that includes the remaining 4 classes (i.e., the "B", "C", "D", and "E" classes). The untrained MLGC is trained using PlasticArmpit sensor module training data that is converted into 2-class representations; the average prediction accuracy of at least one set of training data is 92%. The MLHA 200 also implements the pre-trained MLGC in 65 nm.

[0097] Table 1 presents the performance and area comparison results of MLMC and MLGC 。

[0098] Type Classifier Accuracy Gate Count Main Classifier 5-class BNN (A, B, C, D, E) 91％ 1315 Monitoring Classifier 2-class FVC (A, non-A) 92％ 70

[0099] Advantageously, a relatively small MLGC can be used to protect the SC class predicted by the MLMC without sacrificing prediction accuracy. Additionally, there is no need to add resilience with redundant MLMCs. Instead, a miniature and dedicated MLGC can be trained and implemented to provide hardware redundancy and diversity.

[0100] Embodiments of the present disclosure advantageously provide security-based prediction systems, devices, and hardware-based methods. The embodiments described above and outlined below are combinable.

[0101] In one embodiment, a Machine Learning Hardware Accelerator (MLHA) includes a Main Classifier (MC) module, a Monitoring Classifier (GC) module associated with each SC class, and a Final Prediction Class Decision module coupled to the MC module and each GC module. The MC module includes a pre-trained Machine Learning Main Classifier (MLMC) having at least one Security Critical (SC) class and a plurality of non-SC classes, and is configured to predict an MC prediction class based on input data and output the MC prediction class. Each GC module includes a pre-trained Machine Learning Monitoring Classifier (MLGC) having two classes, including the associated SC class and a residual class that includes any non-associated SC classes and the plurality of non-SC classes. Each GC module is configured to predict a GC prediction class based on input data and output the GC prediction class. The Final Prediction Class Decision module is configured to: receive the MC prediction class and each GC prediction class, determine a final prediction class based on the MC prediction class and each GC prediction class, and output the final prediction class.

[0102] In another embodiment, the hardware-based method includes: predicting, by a main classifier (MC) module, an MC predicted class based on input data, the MC module including a pre-trained machine learning main classifier (MLMC) having at least one safety-critical (SC) class and a plurality of non-SC classes; predicting, by each monitoring classifier (GC) module associated with each SC class, a GC predicted class based on the input data, each GC module including a pre-trained machine learning monitoring classifier (MLGC) having two classes, the two classes including the associated SC class and a residual class, the residual class including any non-associated SC classes and a plurality of non-SC classes; determining, by a final predicted class decision module, a final predicted class based on the MC predicted class and each GC predicted class; and outputting, by the final predicted class decision module, the final predicted class.

[0103] In another embodiment, the system includes a memory configured to store input data including sensor data; a machine learning hardware accelerator (MLHA) coupled to the memory and including: a main classifier (MC) module; a monitoring classifier (GC) module associated with each SC class; a final predicted class decision module coupled to the MC module and each GC module; and a processor coupled to the memory and the MLHA, the processor being configured to: transmit the input data to the MLHA, receive the final predicted class from the MLHA, and control an autonomous process or a semi-autonomous process based on the final predicted class. In one embodiment, the autonomous process or the semi-autonomous process includes an autonomous driving process or a robotic surgery process.

[0104] In another embodiment, the MLMC has a plurality of SC classes, and the determining the final predicted class includes selecting the first SC class as the final predicted class when the MC predicted class is the first SC class and the GC predicted class from the GC module associated with the first SC class is the first SC class.

[0105] In another embodiment, each SC category has an SC priority level, and determining the final prediction category includes when the MC prediction category is a first SC category and the GC prediction category from the GC module associated with the first SC category is a residual category: when the GC prediction category from a GC module not associated with the first SC category is a residual category, selecting the MC prediction category as the final prediction category; when the GC prediction category from one of the GC modules not associated with the first SC category is a second SC category and any remaining GC prediction categories are residual categories, selecting the SC category as the final prediction category; and when the GC prediction categories from two or more of the GC modules not associated with the first SC category are SC categories, ranking the GC prediction categories based on the SC priority level and selecting the GC prediction category with the highest SC priority level as the final prediction category.

[0106] In another embodiment, determining the final prediction category includes when the MC prediction category is one of the non - SC categories: when all GC prediction categories are residual categories, selecting the MC prediction category as the final prediction category; when one of the GC prediction categories is a second SC category and any remaining GC prediction categories are residual categories, selecting the second SC category as the final prediction category; and when two or more of the GC prediction categories are SC categories, ranking the GC prediction categories based on the SC priority level and selecting the GC prediction category with the highest SC priority level as the final prediction category.

[0107] In another embodiment, the MLMC has one SC category, and determining the final prediction category includes when the MC prediction category is the SC category, selecting the SC category as the final prediction category; and when the MC prediction category is one of the non - SC categories: when the GC prediction category is the SC category, selecting the SC category as the final prediction category; and when the GC prediction category is a residual category, selecting the MC prediction category as the final prediction category.

[0108] In another embodiment, the MLHA further includes at least one additional GC module associated with the SC category; and a GC prediction category decision module that is coupled to the GC module and the final prediction category decision module and is configured to: receive each GC prediction category, determine a final GC prediction category based on each GC prediction category, and output the final GC prediction category to the final prediction category decision module, wherein determining the final prediction category is based on the MC prediction category and the final GC prediction category.

[0109] In another embodiment, determining the final GC prediction class includes selecting the SC class as the final GC prediction class when at least half of the classes in the GC prediction class are SC classes; and selecting the residual class as the final GC prediction class when less than half of the classes in the GC prediction class are SC classes.

[0110] While specific implementations of the present disclosure can have many different forms of embodiments, specific embodiments are shown in the drawings and will be described in detail herein. It should be understood that the present disclosure should be regarded as an example of the principles of the present disclosure and is not intended to limit the present disclosure to the specific embodiments shown and described. In the above description, like reference numerals may be used to describe the same, similar, or corresponding components in several views of the drawings.

[0111] In this document, relational terms such as first and second, top and bottom, etc. may be used only to distinguish one entity or action from another entity or action, and do not necessarily require or imply any actual such relationship or order between such entities or actions. The terms "comprises", "comprising", "contains", "having" or any other variation thereof are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a series of elements includes not only those elements but also other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element preceded by "comprises" does not exclude the presence of additional identical elements in the process, method, article, or apparatus that comprises the element.

[0112] References throughout this document to "one embodiment", "certain embodiments", "embodiment", "specific implementation", "aspect" or similar terms mean that a particular feature, structure, or characteristic described in connection with the embodiment is included in at least one embodiment of the present disclosure. Thus, the appearances of such phrases in various places in this specification are not necessarily all referring to the same embodiment. Additionally, the specific features, structures, or characteristics may be combined in any suitable manner without limitation in one or more embodiments.

[0113] As used herein, the term "or" will be interpreted as inclusive and mean any one or any combination. Thus, "A, B, or C" means "any of the following: A; B; C; A and B; A and C; B and C; A, B, and C." The exception to this definition occurs only when the combination of elements, functions, steps, or acts is inherently mutually exclusive in some way. Additionally, grammatical connections are intended to express any and all separate and joined combinations of the joined sub-items, sentences, words, etc., unless otherwise stated or clear from the context. Thus, the term "or" should generally be understood to mean "and / or" and the like. References to an item in the singular form should be understood to include the plural form of the item and vice versa, unless otherwise expressly stated or clear from the context.

[0114] Unless otherwise indicated, the recitation of a range of values herein is not intended to be limiting, but rather is meant to refer individually to any and all values falling within the range, and each separate value within such range is incorporated herein as if it were individually recited herein. When accompanying a numerical value, words such as "about," "approximately," etc. are to be understood as indicating a deviation as would be appreciated by one of ordinary skill in the art, which deviation operates satisfactorily for the intended purpose. Values and / or ranges of numerical values are provided herein only as examples and do not constitute a limitation on the scope of the described embodiments. The use of any and all examples, or exemplary language ("e.g.," "such as," etc.) provided herein is only intended to better illustrate the embodiments and is not a limitation on the scope of the embodiments. Any language in this specification should not be construed as indicating any non-claimed element as essential to the practice of the embodiments.

[0115] For simplicity and clarity of illustration, reference numerals may be repeated in the figures to indicate corresponding or analogous elements. Numerous details are set forth to provide an understanding of the embodiments described herein. The embodiments may be practiced without these details. In other instances, well-known methods, procedures, and components have not been described in detail to avoid obscuring the described embodiments. This specification should not be regarded as limited to the scope of the embodiments described herein.

[0116] In the following description, it should be understood that terms such as "first," "second," "top," "bottom," "upper," "lower," "above," "below," etc. are for convenience only and should not be construed as limiting terms. Also, the terms "device," "apparatus," "system," etc. may be used interchangeably herein.

[0117] From the detailed description, many features and advantages of the present disclosure are apparent, and thus, the appended claims are intended to cover all such features and advantages of the present disclosure that fall within the scope of the present disclosure. Additionally, since many modifications and variations will be readily apparent to those skilled in the art, it is not desired to limit the present disclosure to the exact construction and operation shown and described, and accordingly, all suitable modifications and equivalents falling within the scope of the present disclosure may be adopted.

Claims

1. A machine learning hardware accelerator MLHA, the MLHA comprising: A main classifier MC module, the MC module including a pre-trained machine learning main classifier MLMC having at least one safety-critical SC category and a plurality of non-SC categories, and configured to predict an MC prediction category based on input data including sensor data and output the MC prediction category, wherein the sensor data indicates an image or speech; At least one monitoring classifier GC module associated with each SC category, each GC module including a pre-trained machine learning monitoring classifier MLGC having two categories, the two categories including the associated SC category and a residual category, the residual category including any non-associated SC categories and the plurality of non-SC categories, each GC module configured to predict a GC prediction category based on the input data and output the GC prediction category; And A final prediction category decision module, the final prediction category decision module coupled to the MC module and each GC module and configured to: receive the MC prediction category and each GC prediction category, determine a final prediction category based on the MC prediction category and each GC prediction category, and output the final prediction category, Wherein an autonomous process or a semi-autonomous process is controlled based on the final prediction category.

2. The MLHA according to claim 1, wherein the MLMC has a plurality of SC categories, and the determining the final prediction category includes: When the MC prediction category is a first SC category and the GC prediction category from the GC module associated with the first SC category is the first SC category, selecting the first SC category as the final prediction category.

3. The MLHA according to claim 2, wherein each SC category has an SC priority level, and the determining the final prediction category includes: When the MC prediction category is the first SC category and the GC prediction category from the GC module associated with the first SC category is the residual category: When the GC prediction category from the GC module not associated with the first SC category is the residual category, selecting the MC prediction category as the final prediction category, When the GC prediction category from one of the GC modules not associated with the first SC category is a second SC category and any remaining GC prediction categories are the residual category, selecting the second SC category as the final prediction category, and When the GC prediction categories from two or more of the GC modules not associated with the first SC category are SC categories, sorting the GC prediction categories based on the SC priority level and selecting the GC prediction category having the highest SC priority level as the final prediction category.

4. The MLHA according to claim 2, wherein each SC category has an SC priority level, and the determining the final prediction category includes: When the MC prediction category is one of the non-SC categories: When all of the GC prediction classes are residual classes, select the MC prediction class as the final prediction class. When one of the GC prediction classes is a second SC class and any remaining GC prediction classes are residual classes, select the second SC class as the final prediction class, and When two or more of the GC prediction classes are SC classes, rank the GC prediction classes based on the SC priority level and select the GC prediction class with the highest SC priority level as the final prediction class.

5. The MLHA according to claim 1, wherein the MLMC has one SC class, and the determining the final prediction class includes: When the MC prediction class is the SC class, select the SC class as the final prediction class; And When the MC prediction class is one of the non-SC classes: When the GC prediction class is the SC class, select the SC class as the final prediction class, and When the GC prediction class is the residual class, select the MC prediction class as the final prediction class.

6. The MLHA according to claim 5, the MLHA further comprising: At least one additional GC module associated with the SC class; And A GC prediction class decision module, the GC prediction class decision module being coupled to the GC module and the final prediction class decision module and configured to: receive each GC prediction class, determine a final GC prediction class based on each GC prediction class, and output the final GC prediction class to the final prediction class decision module, Wherein the determining the final prediction class is based on the MC prediction class and the final GC prediction class.

7. The MLHA according to claim 6, wherein the determining the final GC prediction class includes: When at least half of the GC prediction classes are the SC class, select the SC class as the final GC prediction class; And When less than half of the GC prediction classes are the SC class, select the residual class as the final GC prediction class.

8. A hardware-based method, the hardware-based method comprising: Predicting an MC prediction class by a main classifier MC module based on input data including sensor data, the MC module including a pre-trained machine learning main classifier MLMC having at least one safety-critical SC class and a plurality of non-SC classes, wherein the sensor data indicates an image or speech; Predicting a GC prediction class by each monitoring classifier GC module associated with each SC class based on the input data, each GC module including a pre-trained machine learning monitoring classifier MLGC having two classes, the two classes including the associated SC class and a residual class, the residual class including any non-associated SC class and the plurality of non-SC classes; Determining a final prediction class by a final prediction class decision module based on the MC prediction class and each GC prediction class; Output the final predicted category by the final predicted category decision module; and Control an autonomous process or a semi-autonomous process based on the final predicted category.

9. The hardware-based method according to claim 8, wherein the MLMC has a plurality of SC categories, and the determining the final predicted category includes: When the MC predicted category is the first SC category and the GC predicted category from the GC module associated with the first SC category is the first SC category, select the first SC category as the final predicted category.

10. The hardware-based method according to claim 9, wherein each SC category has an SC priority level, and the determining the final predicted category includes: When the MC predicted category is the first SC category and the GC predicted category from the GC module associated with the first SC category is the residual category: When the GC predicted category from the GC module not associated with the first SC category is the residual category, select the MC predicted category as the final predicted category, When the GC predicted category from one of the GC modules not associated with the first SC category is the second SC category and any remaining GC predicted categories are the residual category, select the second SC category as the final predicted category, and When the GC predicted categories from two or more of the GC modules not associated with the first SC category are SC categories, sort the GC predicted categories based on the SC priority level, and select the GC predicted category with the highest SC priority level as the final predicted category.

11. The hardware-based method according to claim 9, wherein each SC category has an SC priority level, and the determining the final predicted category includes: When the MC predicted category is one of the non-SC categories: When all the GC predicted categories are the residual category, select the MC predicted category as the final predicted category, When one of the GC predicted categories is the second SC category and any remaining GC predicted categories are the residual category, select the second SC category as the final predicted category, and When two or more of the GC predicted categories are SC categories, sort the GC predicted categories based on the SC priority level, and select the GC predicted category with the highest SC priority level as the final predicted category.

12. The hardware-based method according to claim 8, wherein the MLMC has one SC category, and the determining the final predicted category includes: When the MC predicted category is the SC category, select the SC category as the final predicted category; and When the MC predicted category is one of the non-SC categories: When the GC predicted category is the SC category, select the SC category as the final predicted category, and When the GC predicted class is the residual class, select the MC predicted class as the final predicted class.

13. The hardware-based method according to claim 12, wherein at least one additional GC module is associated with the SC class; The predicting GC predicted class includes: At a GC predicted class decision module coupled to the GC module and the final predicted class decision module: Receiving each GC predicted class, Determining a final GC predicted class based on each GC predicted class, and Outputting the final GC predicted class to the final predicted class decision module, and The determining the final predicted class is based on the MC predicted class and the final GC predicted class.

14. The hardware-based method according to claim 13, wherein the determining the final GC predicted class includes: When at least half of the GC predicted classes are the SC class, select the SC class as the final GC predicted class; And When less than half of the GC predicted classes are the SC class, select the residual class as the final GC predicted class.

15. A system, comprising: A memory configured to store input data including sensor data, wherein the sensor data indicates an image or speech; A machine learning hardware accelerator MLHA coupled to the memory and including: A main classifier MC module, the MC module including a pre-trained machine learning main classifier MLMC having at least one safety-critical SC class and a plurality of non-SC classes, and configured to predict an MC predicted class based on the input data and output the MC predicted class; Monitoring classifier GC modules associated with each SC class, each GC module including a pre-trained machine learning monitoring classifier MLGC having two classes, the two classes including the associated SC class and a residual class, the residual class including any non-associated SC classes and the plurality of non-SC classes, each GC module being configured to predict a GC predicted class based on the input data and output the GC predicted class; and A final predicted class decision module coupled to the MC module and each GC module and configured to: receive the MC predicted class and each GC predicted class, determine a final predicted class based on the MC predicted class and each GC predicted class, and output the final predicted class; and A processor coupled to the memory and the MLHA and configured to: Transmit the input data to the MLHA, Receive the final predicted class from the MLHA, and Control an autonomous process or a semi-autonomous process based on the final predicted class.

16. The system according to claim 15, wherein the MLMC has a plurality of SC classes, each SC class having an SC priority level, and the determining the final predicted class includes: When the MC predicted class is the first SC class and the GC predicted class from the GC module associated with the first SC class is the first SC class, select the first SC class as the final predicted class; and When the MC predicted class is the first SC class and the GC predicted class from the GC module associated with the first SC class is the residual class: When the GC predicted class from the GC module not associated with the first SC class is the residual class, select the MC predicted class as the final predicted class, When the GC predicted class from one of the GC modules not associated with the first SC class is the second SC class and any remaining GC predicted classes are residual classes, select the second SC class as the final predicted class, and When the GC predicted classes from two or more of the GC modules not associated with the first SC class are SC classes, rank the GC predicted classes based on the SC priority level and select the GC predicted class with the highest SC priority level as the final predicted class.

17. The system according to claim 15, wherein the MLMC has a plurality of SC classes, each SC class having an SC priority level, and the determining the final predicted class includes: When the MC predicted class is the first SC class and the GC predicted class from the GC module associated with the first SC class is the first SC class, select the first SC class as the final predicted class; and When the MC predicted class is one of the non - SC classes: When all the GC predicted classes are residual classes, select the MC predicted class as the final predicted class, When one of the GC predicted classes is the second SC class and any remaining GC predicted classes are residual classes, select the second SC class as the final predicted class, and When two or more of the GC predicted classes are SC classes, rank the GC predicted classes based on the SC priority level and select the GC predicted class with the highest SC priority level as the final predicted class.

18. The system according to claim 15, wherein the MLMC has one SC class, and the determining the final predicted class includes: When the MC predicted class is the SC class, select the SC class as the final predicted class; And When the MC predicted class is one of the non - SC classes: When the GC predicted class is the SC class, select the SC class as the final predicted class, and When the GC predicted class is the residual class, select the MC predicted class as the final predicted class.

19. The system according to claim 18, the system further includes: At least one additional GC module associated with the SC class; And GC prediction category decision module, the GC prediction category decision module is coupled to the GC module and the final prediction category decision module, and is configured to: receive each GC prediction category, determine a final GC prediction category based on each GC prediction category, and output the final GC prediction category to the final prediction category decision module, wherein the determination of the final prediction category is based on the MC prediction category and the final GC prediction category, and includes: when at least half of the GC prediction categories are the SC category, selecting the SC category as the final GC prediction category, and when less than half of the GC prediction categories are the SC category, selecting the residual category as the final GC prediction category.

20. The system according to claim 15, wherein the autonomous process or the semi-autonomous process includes an autonomous driving process or a robotic surgery process.

Images