Validating Deep Neural Network (DNN) Predictions Based on Pre-Trained Classifiers

Abstract

To provide a method and system for validating deep neural network (DNN) prediction.SOLUTION: A method includes: receiving a first data point associated with a real-time application; predicting a first class for the received first data point, by a Deep Neural Network (DNN) pre-trained for a classification task of the real-time application; extracting, from the DNN, a first set of features and a corresponding first set of weights, for the predicted first class; determining, by a pre-trained classifier associated with the predicted first class, a confidence score for the predicted first class; and generating output information to indicate correctness of the predicted first class based on the determined confidence score.SELECTED DRAWING: Figure 3

Description

[Technical field]

[0001] The embodiments discussed in this disclosure relate to validating deep neural network (DNN) predictions based on pre-trained classifiers. [Background technology]

[0002] Recent advances in the field of neural networks have led to the development of various techniques for classification of data that may be relevant for various real-time applications. For example, trained deep neural networks may be utilized in various applications for different classification tasks, e.g., classification or detection of various data points (i.e., images). Although DNNs have achieved good classification accuracy in various classification tasks, in certain applications, such as autonomous driving and medical diagnosis, incorrect detection or misclassification (even in a portion of the cases) may lead to financial or physical losses.

[0003] Typically, a native confidence score associated with a DNN can predict or classify a data point. However, in certain circumstances, the native confidence score associated with a DNN may not be an accurate predictor, which may further lead to a misclassification or prediction of a data point. There are certain conventional systems that can verify the reliability of the predictions or outputs of various classical machine learning models. However, such conventional systems may not be well suited to effectively determine the reliability of a DNN due to the high dimensionality of the data used in DNNs.

[0004] The subject matter claimed in this disclosure is not limited to embodiments that solve any shortcomings or that operate only in such environments. Rather, this background is provided only to illustrate one example technology area where some embodiments described in this disclosure may be practiced. Summary of the Invention [Means for solving the problem]

[0005] According to an aspect of an embodiment, the operations may include receiving a first data point associated with a real-time application. The operations may further include predicting a first class associated with the received first data point by a deep neural network (DNN) pre-trained for a classification task of the real-time application. The operations may further include extracting, from the DNN, a first set of features and a first set of weights corresponding to the first set of features for the predicted first class of the first data point. The extracted first set of features may be associated with a convolutional layer of the DNN. The operations may further include determining, by a pre-trained classifier associated with the predicted first class, a confidence score for the predicted first class for the first data point based on the extracted first set of features and the corresponding first set of weights. The operations may further include generating output information indicative of a correctness of the prediction of the first class for the first data point based on the determined confidence score.

[0006] The object and advantages of the embodiments will be realized and attained at least by the elements, features, and combinations particularly pointed out in the claims.

[0007] Both the foregoing general description and the following detailed description are provided by way of example and explanation and are not restrictive of the invention, as claimed. [Brief description of the drawings]

[0008] The exemplary embodiments will be described and explained with additional specificity and detail through the use of the accompanying drawings.

[0009] [Figure 1] FIG. 1 illustrates an example environment relevant to validating prediction results of a deep neural network (DNN) based on a pre-trained classifier.

[0010] [Diagram 2] FIG. 1 is a block diagram of an exemplary system for validation of prediction results of a deep neural network (DNN) based on a pre-trained classifier.

[0011] [Diagram 3] 1 is a flowchart of an exemplary method for validation of prediction results of a deep neural network (DNN) based on a pre-trained classifier.

[0012] [Figure 4] 1 is a flowchart of an exemplary method for selection of a first set of features associated with a convolutional layer of a deep neural network (DNN).

[0013] [Diagram 5] An exemplary scenario illustrating the extraction of a set of features and a set of weights corresponding to the set of features from a deep neural network (DNN) is presented.

[0014] [Figure 6] 1 is a flowchart of an exemplary method for training a classifier for validation of prediction results of a deep neural network (DNN).

[0015] [Figure 7] An exemplary decision tree classifier for validation of prediction results of deep neural networks (DNN) is presented.

[0016] All based on at least one embodiment described in this disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0017] Some embodiments described in this disclosure relate to methods and systems for effective validation of prediction results of a deep neural network (DNN). In this disclosure, a set of pre-trained classifiers corresponding to a set of classes associated with the DNN may be used to validate the prediction results of the DNN. The set of pre-trained classifiers may be orthogonal or independent to the DNN that may be trained to predict or classify data points of the classes of the set. The validation of the prediction results of the DNN may be based on an analysis of a set of relevant features of the DNN (which may be correct or incorrect) that may provide confidence or reliability to the prediction of the DNN and may be responsible for or used for a prediction of a particular class (i.e., correct or incorrect).

[0018] According to one or more embodiments of the present disclosure, the technical field of classification by DNNs may be improved by configuring a computing system so that the computing system can effectively determine the accuracy or reliability of the predicted result of the DNN. The computing system may include at least one classifier that can verify the predicted result of the DNN (i.e., pre-trained on a class related to the DNN) to further improve the accuracy of the prediction compared to other conventional systems that can verify the predicted result of a classical machine learning model.

[0019] The system may be configured to receive data points associated with a real-time application. Examples of data points may include, but are not limited to, image data, speech data, audio data, text data, or other forms of digital signals. The system may be further configured to predict a class associated with the received data points by a DNN pre-trained for a classification task associated with the real-time application. For example, the classification task may be an image classification task and the data points may include input images of animals (e.g., cats). The system may be configured to control the DNN to predict a class (e.g., a label of cat) based on the received data input (e.g., an input image of a cat).

[0020] The system may further be configured to extract a set of features and a set of weights corresponding to the set of features from the DNN for a predicted class of the input data point. The set of features may be associated with a convolutional layer of the DNN. For example, the convolutional layer may precede an output layer of the DNN. The set of extracted features may be extracted and / or selected to determine the correctness of the DNN's prediction. For example, the system may select a top N of the features and may further use the selected features and corresponding weights to determine the correctness or confidence score of the DNN's prediction.

[0021] The system may be further configured to control the pre-trained classifier for the predicted class to determine a confidence score (or trust score) for the class predicted by the DNN for the data point. The confidence score determination may be based on the set of extracted features and the corresponding set of weights. The confidence score may indicate the correctness or reliability of the prediction by the DNN of the class for the data point. The indicative correctness or trust score provided by the pre-trained classifier may further validate the prediction result of the DNN. The system may be further configured to generate output information based on the confidence score.

[0022] The set of selected features extracted from the convolutional layer of the DNN may represent, for example, high-level features that may be correlated with the predicted class of the data point. For example, in the case of utilizing the DNN for an image classification task, the data point may be an image classified into a class such as a type of animal (e.g., cat), and the set of extracted features may correspond to high-level features of the animal, such as the animal's head, eyes, nose, tail, legs, and other relevant body parts. The system may control a pre-trained classifier for the predicted class (e.g., cat) to use (or be trained based on) such high-level or relevant features of the image and corresponding weights to provide an effective validation of the class predicted by the DNN for the input image. Furthermore, the utilization of a limited number of high-level features (e.g., top-N features) selected from the set of extracted features by the pre-trained classifier (i.e., independent of the DNN) may further lead to minimization of the dimensionality of the data associated with the DNN.

[0023] Embodiments of the present disclosure will now be described with reference to the accompanying drawings.

[0024] FIG. 1 is a diagram illustrating an example environment related to validation of a prediction result of a deep neural network (DNN) based on a pre-trained classifier, configured according to at least one embodiment described in the present disclosure. With reference to FIG. 1, an environment 100 is shown. The environment 100 may include an electronic device 102. The electronic device may further include a deep neural network (DNN) 104. The electronic device 102 may further include a collection of classifiers 106, which may include a first classifier 106A, a second classifier 106B, ..., and an Nth classifier 106N. The environment 100 may further include a database 108, a user end device 110, and a communication network 112. The electronic device 102, the database 108, and the user end device 110 may be communicatively coupled to each other via the communication network 112.

[0025] Examples of electronic devices 102 may include, but are not limited to, mobile devices, desktop computers, laptops, computer workstations, training devices, computing devices, mainframe machines, servers such as cloud servers, and server clusters. In one or more embodiments, electronic devices 102 may include a user end terminal device and a server communicatively coupled to the user end terminal device. Examples of user end terminal devices may include, but are not limited to, mobile devices, desktop computers, laptops, and computer workstations.

[0026] The electronic device 102 may include suitable logic, circuitry, interfaces, and / or code that may be configured to perform one or more operations for validation of a prediction result of a deep neural network (DNN) based on a pre-trained classifier. The electronic device 102 may be implemented using hardware including a processor, a microprocessor (for performing or controlling the execution of one or more operations), a field programmable gate array (FPGA), or an application specific integrated circuit (ASIC). In some other examples, the electronic device 102 may be implemented using a combination of hardware and software.

[0027] In one or more embodiments, the electronic device 102 may be configured to receive a first data point from a plurality of data points in a dataset (i.e., stored in the database 108) via the communication network 112. In some embodiments, the electronic device 102 may be configured to store the first data point in a memory (not shown in FIG. 1 ) of the electronic device 102. The first data point may correspond to, but is not limited to, image data, audio data, speech data, or text data. The dataset may correspond to a real-time application that may perform a specific classification task. Examples of real-time applications may include, but are not limited to, image recognition or classification applications, speech recognition applications, text recognition applications, malware detection applications, autonomous vehicle applications, anomaly detection applications, machine translation applications, pattern recognition from various digital signals, such as, but not limited to, electrical biosignals, motion data, and depth data.

[0028] The electronic device 102 may be configured to control the pre-trained deep neural network 104 to predict a first class (e.g., a label) associated with a received first data point 104. For example, if the DNN 104 is pre-trained for an image classification task and the first data point is an image, the DNN 104 may be controlled to predict an object in the image, such as an animal (e.g., a cat), as the first class. In some embodiments, the electronic device 102 may control the DNN 104 to generate a first confidence score (i.e., a native confidence score as a probability value) that may be indicative of a prediction about the first class associated with the received data point.

[0029] The electronic device 102 may be further configured to extract a first set of features for the predicted first class from the pre-trained DNN 104. The first set of features may be associated with a convolutional layer of the pre-trained DNN 104. The convolutional layer may correspond to, but is not limited to, a layer of the pre-trained DNN 104 preceding an output layer of the pre-trained DNN 104. The electronic device 102 may be further configured to extract a first set of weights corresponding to the first set of features from the pre-trained DNN 104. A gradient-based Class Activation Mapping (CAM) technique may be used to extract the first set of features and the corresponding first set of weights from the pre-trained DNN 104.

[0030] The electronic device 102 may further be configured to control a classifier (e.g., the first classifier 106A) associated with the first class to determine a second confidence score for the first class predicted by the DNN 104106 for the first data point. The second confidence score determined by the classifier may be different from the first confidence score generated by the pre-trained DNN 104. In some embodiments, the electronic device 102 may determine the second confidence score based on the first set of features and the first set of weights corresponding to the first set of features. The electronic device 102 may be configured to select a classifier (e.g., the first classifier 106A) from the set of classifiers 106 based on the prediction of the first class by the DNN 104 for the first data point. Details of the selection of the classifier are described, for example, in FIG. 3.

[0031] The electronic device 102 may be further configured to generate output information that may indicate the correctness of the first class prediction for the first data point based on the second confidence score determined by the classifier (such as the first classifier 106A). The indicated correctness of the prediction may further validate the first class prediction (i.e., the classification of the data point into the first class) performed by the pre-trained DNN 104. In some embodiments, the output information may provide a probability of correctness of the prediction performed by the DNN 104. Details of the first set of features, the classifier, and the validation of the prediction results of the DNN 104 are described in detail, for example, in FIG. 3, FIG. 4, and FIG. 5. Although in FIG. 1 the set of classifiers 106 and the pre-trained DNN 104 are shown as part of the electronic device 102, in some embodiments the set of classifiers 106 and the pre-trained DNN 104 may be integrated into a single device without departing from the scope of the present disclosure. Alternatively, each of the collection of classifiers 106 and the pre-trained DNN 104 may be implemented in a separate device without departing from the scope of this disclosure.

[0032] The deep neural network 104 may include suitable logic, circuitry, interfaces, and / or code that may be configured to classify or recognize input data points to generate an output result for a particular real-time application. For example, the pre-trained DNN 104 may recognize various objects in an input image and provide a unique label for each object in the input image. The unique labels may correspond to different animate objects (e.g., humans, animals, plants, etc.) or non-animate objects (e.g., vehicles, buildings, computers, books, etc., but are not limited to). In another example, the pre-trained DNN 104 associated with a speech recognition application may recognize various input audio samples to identify the source of the audio samples (e.g., human speakers). In an embodiment, the output unique label may correspond to a prediction result of the DNN 104 for the input data points. The DNN 104 may be configured to output a first confidence score (as a native confidence score) that may indicate a probability (between 0 and 1) of the output prediction result of the DNN 104. For example, for an input data point as an animal (such as a cat), the trained DNN 104 may generate a higher first confidence score (e.g., 0.95 closer to 1.0) for predicting the input data point with a unique label as an animal (e.g., cat). The DNN 104 may be implemented using hardware including a processor, a microprocessor (e.g., for performing or controlling the execution of one or more operations), a field programmable gate array (FPGA), or an application specific integrated circuit (ASIC). In some other examples, the DNN 104 may be code, a program, or a set of software instructions. The DNN 104 may be implemented using a combination of hardware and software.

[0033] In some embodiments, the DNN 104 may support multiple recognition layers (not shown) for recognition of input data points, where each successive layer may use the output of the previous layer as input. For example, the multiple recognition layers may include an input layer, one or more hidden layers, and an output layer. Each recognition layer may be associated with multiple neurons, and each neuron may be further associated with multiple weights. During training of the DNN 104, the multiple recognition layers and the multiple neurons in each layer may be determined from hyperparameters of the DNN 104. Such hyperparameters may be set before or during training of the DNN 104 on a training dataset (e.g., various images of a particular class). The DNN 104 may be trained to adjust the multiple weights in various layers based on the input data points and the output results (i.e., hard facts) of the DNN 104.

[0034] Each neuron or node of the DNN 104 may correspond to a mathematical function (e.g., a sigmoid function or a rectified linear unit) having a set of parameters that can be tuned to train the DNN 104 on the relationship between a first data point (e.g., image data) as an input of the DNN 104 and a predicted outcome or class as an output of the DNN 104. The set of parameters can include, for example, weight parameters, regularization parameters, etc. Each neuron can use the mathematical function to calculate an output based on one or more inputs from neurons in other layer(s) of the DNN 104 (e.g., previous layer(s)). All or some of the neurons of the DNN 104 may correspond to the same or different mathematical functions.

[0035] In training the DNN 104, one or more parameters (e.g., weights) of each neuron of the DNN 104 may be updated based on whether the output of the final layer for a given input (from the training dataset) matches the correct result based on a loss function for the DNN. This update process may be repeated for the same or different inputs until a minimum of the loss function is achieved and the training error is minimized. Several methods for training DNNs are known in the art, such as gradient descent, stochastic gradient descent, batch gradient descent, gradient boosting, metaheuristics, etc. The DNN 104 may include code and routines configured to enable a computing device, such as an electronic device, to perform one or more operations for classification of one or more data inputs (i.e., for example, image data) into one or more outputs (i.e., class labels).

[0036] Examples of DNN 104 may include, but are not limited to, a recurrent neural network (RNN), an artificial neural network (ANN), a convolutional neural network (CNN), a CNN recurrent neural network (CNN-RNN), R-CNN, Fast R-CNN, Faster R-CNN, a long short-term memory (LSTM) network based RNN, a CNN+ANN, a LSTM+ANN, a gated recurrent unit (GRU) based RNN, a fully connected neural network, a connectionist temporal classification (CTC) based RNN, a deep Bayesian neural network, a generative adversarial network (GAN), and / or a combination of such networks.

[0037] Each classifier in the set of classifiers 106 may include suitable logic, circuitry, interfaces, and / or code that may be configured to classify input data points into one of a set of classes associated with the DNN 104. Each classifier may be pre-trained for a class associated with the DNN 104 for classification of input data points. For example, the first classifier 106A may be pre-trained to classify an input image into a class, such as an animal, associated with the input image. Each classifier in the set of classifiers 106 may be trained based on a test data set stored in the database 108. A classifier associated with a class (e.g., animal) may be used to validate prediction results of the DNN 104, where the prediction results may correspond to the same class for which the classifier may be trained. For example, a classifier pre-trained for a particular class (e.g., animal) may be used to determine the accuracy or correctness of prediction or classification results of the DNN 104 for input images classified into the same class (i.e., animal). In an embodiment, the pre-trained classifier (e.g., the first classifier 106A) may be configured to determine the accuracy or correctness of the predicted result (i.e., the class predicted by the DNN 104 for the input data point) as a second confidence score associated with the predicted result (or the predicted class). The first classifier 106A may determine the second confidence score (as an orthogonal confidence score) for the predicted class based on a first set of features and a first set of corresponding weights extracted from the DNN 104. Using the set of classifiers 106 for validation of the prediction result of the pre-trained DNN 104 is described in detail, for example, in FIG. 3. Training of a classifier (e.g., the first classifier 106A) from the set of classifiers 106 is described in detail, for example, in FIG. 6. Examples of the collection of classifiers 106 may include, but are not limited to, a decision tree classifier, a support vector machine (SVM) classifier, a naive Bayes classifier, a logistic regression classifier, or a k-nearest neighbor classifier, or a combination thereof.

[0038] The database 108 may include suitable logic, interfaces, and / or code that may be configured to store a data set including a plurality of data points related to a real-time application. The electronic device 102 may receive a first data point of the plurality of data points from the database 108. Additionally, the plurality of data points may include a set of training data points (or a training data set) that may be used to train the DNN 104. The plurality of data points may further include a set of test data points (or a test data set) that may be used to test the DNN 104 or to train each classifier of the set of classifiers 106 for a class predicted by the DNN 104. The database 108 may be a relational or non-relational database that includes the training or test data set. Also, in some cases, the database 108 may be stored on a server, such as a cloud server, or may be cached and stored on the electronic device 102. The server of the database 108 may be configured to receive a request to provide a test data set from the electronic device 102 via the communication network 112. In response, the server of the database 108 may be configured to retrieve and provide the test data set or specific data points to the electronic device 102 based on the received request via the communication network 112. In some embodiments, the database 108 may be configured to store a set of trained classifiers 106 associated with corresponding classes from a set of classes of the pre-trained DNN 104. In some embodiments, the database 108 may be configured to store pre-trained DNNs 104 for certain real-time applications. In some embodiments, the database 108 may be configured to store output information generated by the set of classifiers 106 (i.e., indicative of the correctness of the predictions of the DNN 104).In some embodiments, the database 108 may store a set of extracted features based on which the set of classifiers 106 may be trained to generate output information. Additionally or alternatively, the database 108 may be implemented using hardware including a processor, a microprocessor (e.g., for performing or controlling the execution of one or more operations), a field programmable gate array (FPGA), or an application specific integrated circuit (ASIC). In some other examples, the database 108 may be implemented using a combination of hardware and software.

[0039] The user end device 110 may include suitable logic, circuitry, interfaces, and / or code that may be configured to store a real-time application for which a particular classification task (i.e., the task for which the DNN 104 and the set of classifiers 106 are trained) may be performed. In some embodiments, the user end device 110 may deploy a pre-trained DNN 104 and a set of classifiers 106 for validating prediction results of the deployed pre-trained DNN 104. The user end device 110 may utilize the deployed DNN 104 to perform classification or detection tasks of the real-time application and may utilize the set of deployed classifiers 106 to validate prediction or classification results generated by the deployed DNN 104. For example, the user end device 110 may be an electronic device that may receive input images from an embedded camera or a server and may perform image classification or recognition on the input images based on the trained DNN 104 deployed on the user end device 110. The user end device 110 may further use a deployed pre-trained classifier (e.g., one of the set of classifiers 106) associated with the predicted image class to determine the accuracy or reliability of the classification of the image into the predicted image class performed by the DNN 104 (i.e., deployed on the user end device 110). In another example, the user end device 110 may be an autonomous vehicle that can receive real-time images from its surroundings and detect various objects captured in those images through the embedded trained DNN 104. In such a scenario, the user end device 110 may use a pre-trained classifier from the set of classifiers 106 to validate the predicted output of the DNN 104 and indicate or warn a user associated with the autonomous vehicle of potential misjudgments or incorrect detections performed by the autonomous vehicle's DNN 104.In some embodiments, the user end device 110 may take appropriate action (e.g., applying brakes or controlling steering of the autonomous vehicle) based on an incorrect detection or misjudgment performed by the DNN 104 deployed in the autonomous vehicle.

[0040] In another example, the user end device 110 may be an audio security system that can perform user authentication based on speech recognition performed by the DNN 104 trained on various speech data samples. Similarly, the user end device 110 may verify the authentication of the user performed by the DNN 104 by using an appropriate pre-trained classifier from the collection of classifiers 106 to enhance the accuracy of the authentication. It should be noted that the above examples are not to be construed as limiting the present disclosure and the DNN 104 may be used in many possible applications that are not mentioned for brevity. Examples of the user end device 110 may include, but are not limited to, mobile devices, desktop computers, laptops, computer workstations, computing devices, servers such as mainframe machines, cloud servers, and server clusters.

[0041] The communication network 112 may include a communication medium through which the electronic device 102 may communicate with a server that may store the database 108 and the user end device 110. Examples of the communication network 112 may include, but are not limited to, the Internet, a cloud network, a Wireless Fidelity (Wi-Fi) network, a personal area network (PAN), a local area network (LAN), and / or a metropolitan area network (MAN). The various devices in the environment 100 may be configured to connect to the communication network 112 according to various wired and wireless communication protocols. Examples of such wired and wireless communication protocols may include, but are not limited to, at least one of Transmission Control Protocol and Internet Protocol (TCP / IP), User Datagram Protocol (UDP), Hypertext Transfer Protocol (HTTP), File Transfer Protocol (FTP), ZigBee, EDGE, IEEE 802.11, Light Fidelity (Li-Fi), 802.16, IEEE 802.11s, IEEE 802.11g, multi-hop communications, wireless access points (AP), device-to-device communications, cellular communications protocols, and / or Bluetooth (BT) communications protocols, or combinations thereof.

[0042] Modifications, additions, or omissions may be made to FIG. 1 without departing from the scope of the present disclosure. For example, environment 100 may include more or fewer elements than those shown and described in this disclosure. For example, in some embodiments, environment 100 may include electronic device 102 but not database 108 and user end device 110. Further, in some embodiments, the respective functionality of database 108 and user end device 110 may be incorporated into electronic device 102 without departing from the scope of the present disclosure.

[0043] FIG. 2 is a block diagram of an example system for validation of prediction results of a deep neural network (DNN) based on a pre-trained classifier, configured in accordance with at least one embodiment described in this disclosure. FIG. 2 is described in conjunction with elements from FIG. 1. Referring to FIG. 2, a block diagram 200 of an example system 202 is shown. The example system 202 may include a collection of electronic devices 102, DNNs 104, and classifiers 106. The electronic devices 102 may include a processor 204, a memory 206, persistent data storage 208, input / output (I / O) devices 210, and a network interface 212.

[0044] The processor 204 may include suitable logic, circuitry, and / or interfaces that may be configured to execute program instructions associated with various operations to be performed by the electronic device 102. For example, some of the operations may include receiving a first data point, controlling the DNN 104 for predicting a first class associated with the received first data point, extracting a first set of features and a first set of weights, selecting the first set of features, controlling one of the set of classifiers 106 for determining a second confidence score, and generating output information. The processor 204 may include any suitable special purpose or general purpose computer, computational entity, or processing device, including various computer hardware or software modules, and may be configured to execute instructions stored on any applicable computer readable storage medium. For example, the processor 204 may include a microprocessor, a microcontroller, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), or any other digital or analog circuitry configured to interpret and / or execute program instructions and / or process data. 2 as a single processor, processor 204 may include any number of processors configured to individually or collectively perform or direct the performance of any number of operations of electronic device 102, as described in this disclosure. Further, one or more of the processors may be present on one or more different electronic devices, e.g., different servers.

[0045] In some embodiments, the processor 204 may be configured to interpret and / or execute program instructions and / or process data stored in the memory 206 and / or persistent data storage 208. In some embodiments, the processor 204 may retrieve program instructions from the persistent data storage 208 and load the program instructions into the memory 206. After the program instructions are loaded into the memory 206, the processor 204 may execute the program instructions. Some examples of the processor 204 may be a GPU, a CPU, a RISC processor, an ASIC processor, a CISC processor, a co-processor, and / or combinations thereof.

[0046] Memory 206 may include suitable logic, circuits, interfaces, and / or code that may be configured to store program instructions executable by processor 204. In certain embodiments, memory 206 may be configured to store an operating system and associated application-specific information. Memory 206 may include a computer-readable storage medium for carrying or storing computer-executable instructions or data structures. Such computer-readable storage media may include any available medium that may be accessed by a general-purpose or special-purpose computer, such as processor 204. For example, without limitation, such computer-readable storage media may include tangible or non-transitory computer-readable storage media, including random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), compact disk read-only memory (CD-ROM), or other optical disk storage, magnetic disk storage, or other magnetic storage devices, flash memory devices (e.g., solid-state memory devices), or any other storage medium used to carry or store tangible program code in the form of computer-executable instructions or data structures and that may be accessed by a general-purpose or special-purpose computer. Combinations of the above may also be included within the scope of computer-readable storage media. Computer-executable instructions may include, for example, instructions and data configured to cause the processor 204 to perform a certain operation or group of operations in connection with the electronic device 102.

[0047] Persistent data store 208 may include suitable logic, circuitry, and / or interfaces that may be configured to store program instructions executable by processor 204, an operating system, and / or application-specific information, such as logs and application-specific databases. Persistent data store 208 may include computer-readable storage media for carrying or storing computer-executable instructions or data structures. Such computer-readable storage media may include any available media that can be accessed by a general-purpose or special-purpose computer, such as processor 204.

[0048] By way of example, and not limitation, such computer-readable storage media may include tangible or non-transitory computer-readable storage media, including compact disk read-only memory (CD-ROM) or other optical disk storage, magnetic disk storage or other magnetic storage devices (e.g., hard disk drives (HDD)), flash memory devices (e.g., solid state drives (SSD), secure digital (SD) cards, other solid state memory devices), or any other storage medium that may be used to carry or store tangible program code in the form of computer-executable instructions or data structures and that can be accessed by a general purpose or special purpose computer. Combinations of the above may also be included within the scope of computer-readable storage media. Computer-executable instructions may include, for example, instructions and data configured to cause the processor 204 to perform a certain operation or group of operations related to the electronic device 102.

[0049] In some embodiments, either the memory 206, the persistent data storage 208, or a combination may store the collection of classifiers 106 and the DNN 104 as software instructions. The processor 204 may retrieve software instructions associated with the collection of classifiers 106 and the DNN 104 to perform various operations of the disclosed electronic device 102. In some embodiments, either the memory 206, the persistent data storage 208, or a combination may store the first data points and / or training / testing data set, the first plurality of features, and the corresponding first plurality of weights.

[0050] The input / output device 210 may include suitable logic, circuitry, interfaces, and / or code that may be configured to receive user input. The input / output device 210 may further be configured to provide output in response to the user input. For example, the input / output device 210 may receive commands or handwritten text from a user as user input, which may be used to train the DNN 104, train a classifier from the set of classifiers 106, or validate the predicted results of the trained DNN 104. The input / output device 210 may include various input and output devices that may be configured to communicate with other components, such as the processor 204 and the network interface 212. The input / output device 210 may include an input device or an output device. Examples of input devices may include, but are not limited to, a touch screen, a keyboard, a mouse, a joystick, and / or a microphone. Examples of output devices may include, but are not limited to, a display and a speaker.

[0051] The network interface 212 may include suitable logic, circuitry, interfaces, and / or code that may be configured to establish communications between the electronic device 102, the database 108, and the user end device 110 over the communications network 112. The network interface 212 may be implemented by using various known technologies to support wired or wireless communications of the electronic device 102 over the communications network 112. The network interface 212 may include, but is not limited to, an antenna, a radio frequency (RF) transceiver, one or more amplifiers, a tuner, one or more oscillators, a digital signal processor, an encoder-decoder (codec) chipset, a subscriber identity module (SIM) card, and / or a local buffer.

[0052] The network interface 212 may communicate via wireless communication with the Internet, an intranet, and / or a wireless network, such as a cellular telephone network, a wireless local area network (LAN), and / or a metropolitan area network (MAN), etc. The wireless communication may use any of a number of communication standards, protocols, and technologies, such as Global System for Mobile Communications (GSM), Enhanced Data GSM Environment (EDGE), Wideband Code Division Multiple Access (W-CDMA), Long Term Evolution (LTE), Code Division Multiple Access (CDMA), Time Division Multiple Access (TDMA), Bluetooth, Wireless Fidelity (Wi-Fi) (e.g., IEEE802.11a, IEEEE802.11b, IEEEE802.11g, and / or IEEE802.11n, etc.), Voice over Internet Protocol (VoIP), Light Fidelity (Li-Fi), or Wi-MAX.

[0053] Modifications, additions, or omissions may be made to the example system 202 without departing from the scope of the present disclosure. For example, in some embodiments, the example system 202 may include any number of other components that may not be explicitly shown or described for the sake of brevity.

[0054] FIG. 3 is a flowchart of an example method for validation of prediction results of a deep neural network (DNN) based on a pre-trained classifier, according to at least one embodiment described in this disclosure. FIG. 3 is described in conjunction with elements of FIG. 1 and FIG. 2. Referring to FIG. 3, a flowchart 300 is shown. The method illustrated in the flowchart 300 may begin at 302 and may be performed by any suitable system, apparatus, or device, such as the example electronic device 102 of FIG. 1 or the example system 202 of FIG. 2. For example, one or more of the electronic device 102, the pre-trained DNN 104, or a classifier from the collection of classifiers 106 may perform one or more of the operations associated with the method 300. Although shown in discrete blocks, steps and operations associated with one or more blocks of the method 300 may be divided into additional blocks, combined into fewer blocks, or eliminated, depending on the specific implementation.

[0055] At block 302, a first data point may be received. The first data point may be associated with a real-time application. For example, the first data point may include an image, an audio / speech sample, a text character, software instructions, or other form of digital signal, such as, but not limited to, an electrical biosignal, motion data, or depth data. Examples of real-time applications may include, but are not limited to, an image recognition application, an image classification application, a speech recognition application, a text recognition application, a malware detection application, an autonomous vehicle application, an anomaly detection application, a machine translation application, or a pattern recognition application from a digital signal / data.

[0056] In some embodiments, the processor 204 may be configured to receive a first data point (e.g., an image) from a plurality of data points in a dataset, which may be stored in any or a combination of the memory 206, the persistent data storage 208, or the database 108. The first data point (e.g., an image) may be received for classification or prediction into a particular class label, where the classification or prediction may be performed by the pre-trained DNN 104.

[0057] At block 304, a first class associated with the received first data point may be predicted. The pre-trained DNN 104 may be controlled to predict the first class associated with the received first data point. In one or more embodiments, the processor 204 may be configured to control the DNN 104 to predict the first class for the received first data point. For example, if the DNN 104 is pre-trained for an image classification task and the first data point is an image, the pre-trained DNN 104 may predict the first class for the image as an animate object (e.g., an animal, a plant, or a human) or a non-animate object (e.g., a building, a vehicle, a road, a sign, or any other object). If the image input to the DNN 104 is of a dog animal, the DNN 104 may output a unique class label that may indicate a classification of the image into a dog label. The output class label may be considered as a predicted result of the DNN 104.

[0058] The DNN 104 may be configured to generate a first confidence score of the DNN 104 as a native confidence score. The first confidence score may indicate a probability value (e.g., between 0 and 1) indicative of a predicted result of the DNN 104. In other words, the first confidence score generated by the DNN 104 may indicate a prediction of a first class (i.e., a class label) for a received input image (or data point).

[0059] At block 306, a first set of features and a first set of weights corresponding to the first set of features may be extracted for a predicted first class of a first data point. In one or more embodiments, the processor 204 may be configured to extract the first set of features and the first set of weights corresponding to the first set of features from a pre-trained DNN 104. The processor 204 may extract the first set of features and the first set of corresponding weights from a convolutional layer of the DNN 104. In an embodiment, the convolutional layer from which the features are extracted may be a layer of the DNN 104 that may precede an output layer of the DNN 104. In such a case, the convolutional layer may be the last convolutional layer of the DNN 104. In some embodiments, the processor 204 may be configured to extract the first set of features and the first set of corresponding weights from a convolutional layer of the DNN 104 using a process of class activation mapping (CAM). The process of class activation mapping (CAM) and the convolutional layers of the DNN 104 from which features (or a set of feature values) and corresponding weights may be extracted are described in detail, for example, in FIG. 5.

[0060] For one example, the first data point may be an image to be classified by the DNN 104. The first set of features may include features or characteristics of an object in the image, where the object may be identified or classified in a convolutional layer of the DNN 104. The convolutional layer associated with the first set of features may be a layer preceding the output layer of the DNN 104 (or may be the last convolutional layer), such that the first set of features may be high-level features associated with the first data point. In one example, if the object is an animal in the image, the high-level features of the convolutional layer may correspond to various characteristics or body parts of the animal, for example, the animal's head, eyes, tail, and legs. Such high-level features may be relevant features upon which the DNN 104 may be trained to potentially generate a prediction result as a first class (e.g., an animal in the image) for the input data point. In other words, the trained DNN 104 may use such high-level features for prediction or classification (whether correct or incorrect) of a first class for a first data input. Furthermore, the first set of weights may correspond to a convolutional layer multiplier that may be associated with such high-level feature (or the first set of features). The feature multiplier may give the weight or importance of the feature relative to other features to determine the output of the convolutional layer in the output layer of the DNN 104. Extraction of the first set of features and the first set of weights from the DNN 104 is described in detail, for example, in FIG. 5.

[0061] In some embodiments, the processor 204 may be configured to extract a second set of features and a second set of weights corresponding to the second set of features from the convolutional layer of the DNN 104. The second set of features may include the first set of features. In other words, the number of features in the first set may be less than or equal to the number of features in the second set, and the first set of features may be a subset of the second set of features. Furthermore, the processor 204 may be configured to select the first set of features from the extracted second set of features based on the second set of weights. In such a scenario, the extraction of the first set of features may correspond to the selection of the first set of features from the second set of features. The extraction of the second set of features and the selection of the first set of features from the second set of features are described in detail, for example, in FIG. 4.

[0062] At block 308, a confidence score may be determined for the predicted first class of the first data point. In one or more embodiments, the processor 204 may be further configured to control a classifier (e.g., the first classifier 106A) to determine a confidence score (or a second confidence score) for the first class predicted by the DNN 104 for the first data point. The classifier (e.g., the first classifier 106A) may be associated with the predicted first class of the first data point. Because the classifier (e.g., the first classifier 106A) is independent of the DNN 104, the second confidence score generated by the first classifier 106A may be referred to as an orthogonal confidence score that may be different from the first confidence score (i.e., native confidence score) generated by the pre-trained DNN 104.

[0063] In some embodiments, the first classifier 106A may determine a second confidence score for the predicted class based on the extracted or selected first set of features and a first set of weights corresponding to the first set of features. Examples of the first classifier 106A (or others in the set of classifiers 106) may include, but are not limited to, a decision tree classifier, a support vector machine (SVM) classifier, a naive Bayes classifier, a logistic regression classifier, or a k-nearest neighbor classifier.

[0064] In an embodiment, the first set of features may include one or more first features of the convolutional layer that may be related to the predicted first class, and may further include one or more second features of the convolutional layer that may not be related (or may be less related) to the predicted first class. For example, for a predicted first class of animal “cat” associated with an image, the one or more first features related to or important for a correct prediction of the first class may include, but are not limited to, a distinctive head, eyes, or ears. Additionally, the one or more second features that may be less important or may lead to an incorrect prediction of the first class may include, but are not limited to, a body color, a tail, or fur, which may be common to other animals as well. In an embodiment, the processor 204 may control or use the first classifier 106A to determine a second confidence score based on a correlation of the one or more first features with a correct prediction of the first class, and based on another correlation of the one or more second features with an incorrect prediction of the first class. Each feature from the one or more first features and the one or more second features may include a set of feature values. The correlation of the one or more first features with correct predictions or the correlation of the one or more second features with incorrect predictions of the first class may be based on the set of feature values ​​and a weight corresponding to each feature. This is for example described in Figures 5 and 7.

[0065] In one example, the first classifier 106A for the predicted first class may be a pre-trained decision tree classifier that may include a set of internal nodes and a set of leaf nodes. Each internal node of the set of internal nodes of the decision tree classifier may represent a condition associated with a feature of the first set of features and a corresponding weight of the feature. The condition may be with respect to a set of feature values ​​for each feature of the first set of features and a weight corresponding to each feature. Furthermore, each leaf node of the set of leaf nodes may represent a predicted output of the decision tree classifier that may include a second confidence score associated with the predicted first class for the first data point. The determination of the condition set in the set of internal nodes of the decision tree classifier and the second confidence score for the predicted first class of the first data point based on the set of feature values ​​for each feature is described in detail, for example, in FIG. 7.

[0066] In some embodiments, the processor 204 may be configured to select a classifier (e.g., the first classifier 106A) from the set of classifiers 106 for the first class predicted by the DNN 104 (e.g., in step 304 of FIG. 3). For example, the set of classifiers 106 may correspond to various classes for which the DNN 104 may be pre-trained for a particular classification task (e.g., image classification). Examples of such classes include animate objects (such as animals, birds, plants, and humans) and non-animate objects (such as, but not limited to, buildings, vehicles, roads, electronic devices, or any other object). If the predicted first class is a cat (i.e., an animal), the processor 204 may select a classifier (e.g., the first classifier 106A) related to animals or specifically to cats from the set of classifiers 106 to further determine a second confidence score for the predicted first class (i.e., a cat). In an embodiment, the selected classifier (e.g., the first classifier 106A) may be pre-trained for the first class (i.e., cat) predicted by the DNN 104 (i.e., predicted in step 304). Training a classifier (e.g., the first classifier 106A) of the collection of classifiers 106 for a particular class is described in detail, for example, in FIG.

[0067] At block 310, output information may be generated. In one or more embodiments, the processor 204 may be further configured to generate the output information based on a second confidence score determined by the selected or pre-trained classifier (e.g., the first classifier 106A). The generated output information may indicate a correctness of the prediction of the first class for the first data point by the DNN 104. In other words, the generated output information may indicate a correctness of the prediction performed by the DNN 104 (in classifying the received first data point into the first class 304). In some embodiments, the second confidence score may correspond to a probability value between 0 and 1, such that a probability value closer to "1" may indicate a correct prediction of the first class for the first data point by the DNN 104, and a probability value closer to "0" may indicate an incorrect prediction made by the DNN 104 as the first class for the received first data point. In other words, the second confidence score determined by the classifier (e.g., the first classifier 106A orthogonal to the DNN 104) may indicate a trust or reliability score for the prediction performed by the DNN 104 or may verify the correctness of the prediction performed by the DNN 104 (e.g., performed in step 304). In other words, the second confidence score may indicate the probability of a correct prediction being performed by the DNN 104 during real-time operation.

[0068] In an embodiment, the processor 204 may be further configured to compare the determined second confidence score with a predefined score threshold. The processor 204 may generate output information indicative of the correctness of the prediction of the first class based on the comparison. For example, if the second confidence score corresponds to a probability value between 0 and 1, the predefined score threshold may be 0.5. In such a case, the output information generated by the processor 204 may indicate a correct prediction of the first class if the determined confidence score is greater than 0.5, and may indicate an incorrect prediction of the DNN 104 if the second confidence score is less than 0.5 (i.e., the predefined score threshold).

[0069] In an embodiment, the output information may correspond to at least one of, but not limited to, displaying the second confidence score on the input / output device 210, storing the second confidence score in a log file, or a notification / alert based on the second confidence score. For example, the user end device 110 (or the electronic device 102) may display the second confidence score to the user on a display device together with the predicted first class to indicate the degree of correctness of the first class predicted by the DNN 104. In another example, the user end device 110 (or the electronic device 102) may store the determined second confidence score in a log file in a memory (e.g., memory 206). For example, the log file may indicate how many times the DNN 104 predicted correctly or not to determine the accuracy of the DNN 104 in real-time operation (e.g., in an autonomous vehicle). In certain scenarios, the second confidence score may be stored in a log file along with the predicted first class based on a comparison of the second confidence score with a predefined score threshold. For example, the second confidence score may be stored in the log file if the second confidence score is greater than (or less than) a predefined score threshold. In another example, the output information may be indicated as a notification (e.g., an alert or warning) to a user of the user end device 110 (or electronic device 102) based on the second confidence score determined by the selected classifier (e.g., the first classifier 106A). For example, if the user end device 110 is an autonomous vehicle and the second confidence score is below a predefined score threshold (e.g., 0.5), the user end device 110 (i.e., the autonomous vehicle) may notify a user (e.g., a passenger of the autonomous vehicle). The notification may include a warning or alert for the user to control the autonomous vehicle due to a first class of potential erroneous predictions (e.g., misclassification or misidentification of a potential obstacle object) performed by the DNN 104 deployed in the user end device 110.In some embodiments, the output information generated by the electronic device 102 may correspond to some automatic action to be taken in case of an erroneous prediction of the DNN 104 detected by a classifier (e.g., the first classifier 106A), for example. For example, in case of detection of a misclassification or an erroneous prediction performed by the DNN 104 for a received first data point in an autonomous vehicle, the user end device 110 or the electronic device 102 may generate said output information to automatically apply brakes, control steering, or significantly reduce the speed of the autonomous vehicle.

[0070] Thus, the disclosed electronic device 102, which may include a classifier orthogonal to or independent of the DNN 104, can determine a second confidence score (or trust score) to determine the correctness of the prediction of the DNN 104. This can further improve the accuracy of the prediction or classification of the DNN 104 by appropriately detecting the misclassification or incorrect prediction and taking certain actions (as output information) to avoid economic or physical losses caused by the misprediction. Control may proceed to end.

[0071] Although flowchart 300 is shown as discrete operations, such as 302, 304, 306, 308, and 310, in certain embodiments, such discrete operations may be further divided into additional operations, combined into fewer operations, or eliminated, depending on the particular implementation, without departing from the essence of the disclosed embodiments.

[0072] FIG. 4 is a flowchart of an example method for selection of a first set of features associated with a convolutional layer of a deep neural network (DNN), according to at least one embodiment described in this disclosure. FIG. 4 is described in conjunction with elements of FIG. 1, FIG. 2, and FIG. 3. Referring to FIG. 4, a flowchart 400 is shown. The method illustrated in the flowchart 400 may begin at 402 and may be performed by any suitable system, apparatus, or device, such as the electronic device 102 of FIG. 1 or the example system 202 of FIG. 2. For example, one or more of the electronic devices 102 or the set of classifiers 106 may perform one or more of the operations associated with the method 400. Although shown in discrete blocks, steps and operations associated with one or more blocks of the method 400 may be divided into additional blocks, combined into fewer blocks, or eliminated, depending on the specific implementation.

[0073] At block 402, a second set of features and a second set of weights corresponding to the second set of features may be extracted from the pre-trained DNN 104. In one or more embodiments, the processor 204 may be configured to extract the second set of features and the second set of weights corresponding to the second set of features from a convolutional layer of the pre-trained DNN 104. For example, the convolutional layer may correspond to a layer that may precede the output layer of the pre-trained DNN 104, for example, as described in step 306 of FIG. 3. For example, a first data point input to the pre-trained DNN 104 may be an image to be classified by the DNN 104. The second set of features may include features or characteristics of an object in the image, where the object may be identified or classified in the convolutional layer of the DNN 104. The convolutional layer associated with the second set of features may be a layer preceding the output layer of the DNN 104, such that the second set of features may be high-level features (e.g., head, eyes, tail, or legs if the first class corresponds to an animal) associated with the first data point (e.g., an image of an animal). The second set of weights may correspond to a relative importance or score assigned to each corresponding feature of the second set of features. In other words, the weight of a feature may be the relative contribution of that feature in determining a prediction outcome or classification performed by the DNN 104 in the output layer, e.g., as described in detail in FIG. 3 and FIG. 5. In some embodiments, the processor 204 may be configured to use a process of class activation mapping (CAM) to extract the second set of features and corresponding second set of weights from the convolutional layer of the DNN 104, e.g., as described in detail in FIG. 5.

[0074] At block 404, a first set of features may be selected from the second set of features. In one or more embodiments, the processor 204 may be configured to select the first set of features from the second set of features based on a second set of weights corresponding to the second set of features. The first set of features to be selected may be a subset of the second set of features extracted from the convolutional layer of the DNN 104. In some embodiments, the processor 204 may be configured to compare each weight of the second set of weights to a predefined weight threshold. The processor 204 may select the first set of features from the second set of features based on a comparison of the second set of weights to the predefined weight threshold. For example, the value of each weight of the second set of weights may be between 0 and 1, and the predefined weight threshold may be 0.6. In such a scenario, the processor 204 may compare each weight of the second set of weights to 0.6 and select (as the first set of features) from the second set of features a feature whose corresponding weight is greater than 0.6. Such selected features (having corresponding weights greater than the predefined weight threshold) may be important or more relevant to predicting the first class. In an embodiment, the predefined weight threshold may be stored in the memory 206 for a particular predicted class. In some embodiments, the electronic device 102 may define different weight thresholds (i.e., for selection of the first set of features) for different classes (e.g., automobiles, birds, animals, flying objects) predicted or classified by the pre-trained DNN 104.

[0075] In other embodiments, the processor 204 may be configured to sort the second set of features based on application of a predefined sorting criterion to the second set of weights corresponding to the second set of features. For example, the processor 204 may sort the second set of features in descending order (predefined sorting criterion) of the second set of weights corresponding to the second set of features. In other words, features having higher weights (i.e., greater importance or relevance) may be sorted and placed before features having lower weights in the sorted second set of features. In one embodiment, the processor 204 may be configured to select the first set of features from the sorted second set of features. For example, the processor 204 may select features corresponding to the top N weights (e.g., features with the top 10 weights, i.e., the top 10 features by weight) from the extracted second set of features as the first set of features. The first set of selected features may be further trained by a classifier (e.g., a first classifier 106A selected from the set of classifiers 106) and further used to determine a second confidence score and output information indicative of the correctness of the prediction performed by the DNN, e.g., as described in steps 308 and 310 of FIG. 3.

[0076] The selection of the first set of features (e.g., the top 10 high-level features) from the second set of features may reduce the high dimensionality associated with a larger number of feature sets of the DNN 104. This reduction in the dimensionality of the large datasets and features in the DNN 104 may further improve the efficiency of the classifiers that may be used for validation of the predicted first class of the first data point. Furthermore, the selection of features (i.e., the first set of features) with weights above a certain weight threshold, or the selection of features with top weights from the second set of features, may ensure that the selected first set of features may be relevant and important for accurate validation of the first class predicted by the DNN 104 of the first data point. In some cases, a small number of features that may be the cause of an incorrect prediction or misclassification may also be important features for training the set of classifiers 106 and further validating the prediction results of the DNN 104 in real-time operation. Training of the classifiers is described in further detail, for example, in FIG. 6 and FIG. 7.

[0077] Although flowchart 400 is shown as discrete operations, such as 402 and 404, in certain embodiments, such discrete operations may be further divided into additional operations, combined into fewer operations, or eliminated, depending on the particular implementation, without departing from the essence of the disclosed embodiments.

[0078] Although Figure 4 is described with respect to extracting a second set of features and a second set of weights, and selecting a first set of features, the scope of the present disclosure is not limited thereto. The description of Figure 4 may be similarly applicable to a classifier training process that may include extracting a second plurality of features and a second plurality of weights, and selecting a first plurality of features, for example, as described in Figure 6, without departing from the scope of the present disclosure.

[0079] FIG. 5 illustrates an exemplary scenario illustrating the extraction of a set of features from a deep neural network (DNN) and a set of weights corresponding to the set of features, configured in accordance with at least one embodiment described in the present disclosure. FIG. 5 is described in conjunction with elements from FIG. 1, FIG. 2, FIG. 3, and FIG. 4. Referring to FIG. 5, a test data set 502 is shown. The test data set 502 may be stored in any one or combination of the memory 206, the persistent data storage 208, or the database 108. The test data set 502 may include a first data point 504. In FIG. 5, the first data point 504 as image data (i.e., a cat image) is merely an example. However, without departing from the scope of the present disclosure, the test data set 502 may include multiple data points including the first data point 504, and may include other types of data points, such as audio data, text data, or other digital signals.

[0080] Also shown in FIG. 5 is the DNN 104. As described in FIG. 3 and FIG. 4, the processor 204 may be configured to control the DNN 104 to extract a first set of features and a first set of weights corresponding to the first set of features. In some embodiments, the processor 204 may extract a second set of features and a second set of corresponding weights from the DNN 104 and select the first set of features from the second set of features, for example as described in FIG. 4. For brevity, FIG. 5 is described with respect to the first set of features and the first set of weights, but the scope of the present disclosure is not limited thereto. The description of FIG. 5 may be similarly applicable to the extraction of the second set of features and the second set of weights without departing from the scope of the present disclosure.

[0081] 5, the DNN 104 may include multiple convolutional layers, including a first convolutional layer 506A, a second convolutional layer 506B, ..., and an Nth convolutional layer 506N. The DNN 104 may be configured to receive a first data point 504 as a data input to the first convolutional layer 504A for classification by the DNN 104. The DNN 104 may further include a flattening layer 508 and an output layer 510 (or fully connected layer or dense layer), which may each follow the Nth convolutional layer 506N (or activation layer) in the DNN 104.

[0082] In one embodiment, the Nth convolutional layer 506N may include multiple activation maps associated with different features of the data input (e.g., the first data point 504) input to the DNN 104. For example, the activation layer (i.e., the Nth convolutional layer 506N) may include 512 activation maps, each having dimensions 2×2, and each feature may be associated with a particular one of the 512 activation maps. Thus, each of the 512 activation maps having dimensions 2×2 may include four real numbers (i.e., the set of feature values ​​described in FIG. 3, FIG. 6, and FIG. 7). In that case, f k Let denote the k-th activation map, k ∈ {1, 2, …, 512}.

[0083] The DNN 104 may further include an average pooling layer (not shown in FIG. 5) between the Nth convolutional layer 506N and the flattening layer 508. The average pooling layer may be configured to perform a Global Average Pooling (GAP) operation on the output of the Nth convolutional layer 506N. Based on the GAP operation, the average pooling layer may be configured to reduce the size of the output from the Nth convolutional layer 506N by averaging the activation maps of each feature. For the above example, the average pooling layer may reduce the size of the activation maps of the features from dimensions "2,2,512" to "1,1,512". The flattening layer 508 may be configured to flatten the input to the flattening layer 508 without changing the information input to the flattening layer 508. 5, the flattening layer 508 may convert a reduced dimensional vector such as "1,1,512" (i.e., input to the flattening layer 508 from the average pooling layer) into a one-dimensional vector of features the same size as the number of nodes in the output layer 510. For example, the flattening layer 508 may provide 512 one-dimensional values ​​for the 512 number of nodes in the output layer 510. The flattening layer 508 may include a set of weights (e.g., W1, W2, ..., Wn) that may correspond to the features (e.g., f1, f2, ..., fn) in the Nth convolutional layer 506N.

[0084] The output layer 510 may include one node for each output class associated with the DNN 104. The output layer 510 may receive inputs from the flattening layer 508 of a one-dimensional vector of features and a set of weights corresponding to the features. In one example, the output layer 510 may apply a SoftMax function to each one-dimensional vector of features and its corresponding weights to determine the probability that the input first data point 504 is associated with a class of the DNN 104. The SoftMax function is represented by Equation 1 below.

number

[0085] For example, the DNN 104 may be pre-trained to classify various creatures from an input image (e.g., a first data point 504). The DNN 104 may include a set of nodes (e.g., C1, C2, ..., Cn) in the output layer 510 for each class of creatures (e.g., a total of n classes) that the DNN 104 may be configured to predict. The DNN 104 may predict a first class (e.g., cat) for the first data point 504 based on a probability for the first class at a corresponding node in the output layer 510. For example, the DNN 104 may obtain a probability value (i.e., a first confidence score) of 0.85 at node C2 of the output layer 510 that may correspond to a first class such as a cat label.

[0086] Also shown in FIG. 5 is a block 512 illustrating a process of class activation mapping (CAM) for a predicted first class (e.g., cat label) of the first data point 504 (i.e., cat image). In an embodiment, based on the predicted first class of the first data point 504, the processor 204 may be configured to extract activation maps (as a first set of features) from the Nth convolutional layer 506N that are associated with features of the first data point 504. In an embodiment, the extraction of a set of feature values ​​(i.e., 2×2 dimensions) for each of the activation maps may correspond to the extraction of a first set of features from the pre-trained DNN 104, for example, as described in step 306 of FIG. 3. Such extraction of a set of feature values ​​(i.e., 2×2 dimensions) for each of the activation maps may correspond to a class activation mapping (CAM) process. Because the CAM process may determine the causes (i.e., features that contributed to the DNN's prediction, whether correct or incorrect) of the classification of the first data point 504 into a first class (e.g., cat class), the CAM process may be referred to as an interpretability analysis technique for the DNN 104. The CAM process may indicate regions of pixels in the first data point 504 that may lead to the classification of the first data point 504 into the first class (e.g., cat class). The CAM process may also determine features and weights of these features that may contribute to the DNN 104's prediction of the first class for the first data point 504.

[0087] For example, the first set of features may include a first feature (f1), a second feature (f2), ..., and an Nth feature (fn), each of which may be associated with a predicted first class (e.g., cat) of the input cat image. Each feature may have an associated activation map (hereinafter referred to as a feature map having a 2x2 dimensional set of feature values). The feature map may be a heat map that may indicate regions or pixels in the input cat image that may be associated with a corresponding feature in the input cat image (i.e., the first data points 504). Block 512 illustrates a first feature map 514A related to a first feature (f1), which may be associated with, for example, a head portion 516A of the cat in the cat image. Block 512 further illustrates a second feature map 514B related to a second feature (f2), which may be associated with a tail portion 516B of the cat in the cat image. Similarly, block 512 shows an Nth feature map 514N related to an Nth feature (fn) that may be associated with, for example, the cat's eyes, nose, and / or mouth (collectively referred to as facial part 516N) in a cat image.

[0088] In FIG. 5, block 512 further illustrates a first set of weights, such as a first weight (W1), a second weight (W2), ..., and an Nth weight (Wn). The processor 204 may be configured to extract the first set of weights from the flattening layer 508. The first set of weights may correspond to a first set of extracted features (i.e., activation maps). In other words, each weight (e.g., the i-th weight w i ) is the feature (e.g., the i-th feature f i), as shown in block 512, the processor 204 may be configured to perform a weighted summation of the extracted set of feature values ​​of the first set of features and the first set of weights (corresponding to the features of the first set) to obtain a class activation map 518 for the predicted first class of the first data point 504. The class activation map 518 may be a heat map that may indicate regions in the cat image that may correspond to various high-level features (and associated portions) of the cat in the cat image. In other words, the class activation map 518 may identify and locate the position of the cat in the cat image. Thus, the processor 204 may be configured to extract a set (2×2 dimensions) of feature values ​​for each of the first feature (f1), the second feature (f2), ..., and the Nth feature (fn) as the first set of high-level features, and extract a corresponding first set of weight values ​​from the DNN 104. Thus, the disclosed electronic device 102 can perform a class activation mapping (CAM) process (also called interpretability analysis) on the DNN 104 to extract a set of feature values ​​and corresponding set of weights for each high-level feature (i.e., each feature in the first set of features) used for the prediction made by the DNN 104 for the first data point 504. The class activation mapping (CAM) process (also called interpretability analysis) can extract all features (i.e., top features) from the DNN 104 that were responsible for either a correct prediction (i.e., the first class) or an incorrect prediction (i.e., the other class) in the DNN 104. Training a classifier (i.e., orthogonal to the DNN 104) based on such extracted features (e.g., feature values ​​and corresponding weights) and their correlation with correct / incorrect predictions may allow the trained classifier to verify the correctness of the predictions of the DNN 104 or flag potential mispredictions of the DNN 104 during real-time operation (e.g., automatic object detection in an autonomous vehicle). Training a classifier for a particular class based on extracted features and weights is further described, for example, in FIGs. 6 and 7.

[0089] The processor 204 may be further configured to determine a second confidence score and determine output information using the first set of extracted features (or a set of feature values ​​for each feature) and the first set of weights to verify or indicate the correctness of the first class prediction of the DNN 104, e.g., as described in detail in Figures 3 and 7.

[0090] FIG. 6 is a flowchart of an example method for training a classifier for validation of prediction results of a deep neural network (DNN), according to at least one embodiment described in this disclosure. FIG. 6 is described in conjunction with elements of FIG. 1, FIG. 2, FIG. 3, FIG. 4, and FIG. 5. Referring to FIG. 6, a flowchart 600 is shown. The method illustrated in the flowchart 600 may begin at 602 and may be performed by any suitable system, apparatus, or device, such as the example electronic device 102 of FIG. 1 or the example system 202 of FIG. 2. For example, one or more of the classifiers (e.g., the first classifier 106A) from the electronic device 102 or the collection of classifiers 106 may perform one or more of the operations associated with the method 600. Although shown in discrete blocks, steps and operations associated with one or more blocks of the method 600 may be divided into additional blocks, combined into fewer blocks, or eliminated, depending on the specific implementation.

[0091] At block 602, a test data point may be selected. In one or more embodiments, the processor 204 may be configured to select the test data point from a test dataset that includes a plurality of test data points associated with a real-time application. The test data point (e.g., a cat image) may be associated with a first class (i.e., cat) on which the DNN 104 may be pre-trained. The test dataset may be stored in any one or combination of the memory 206, the persistent data store 208, or the database 108. The processor 204 may receive the test data point from one of the memory 206, the persistent data store 208, or the database 108.

[0092] For example, the test data points may include, but are not limited to, images, audio / speech samples, text characters, software instructions, or other forms of digital signals such as, but not limited to, electrical bio-signals, motion data, or depth data. Examples of real-time applications may include, but are not limited to, image recognition applications, image classification applications, speech recognition applications, text recognition applications, malware detection applications, autonomous vehicle applications, anomaly detection applications, machine translation applications, or pattern recognition applications from digital signals / data.

[0093] In one embodiment, the test data set may include a first set of test data points associated with a first class and a second set of test data points associated with one or more classes other than the first class. Each test data point in the test data set may have a known pre-assigned class associated therewith (a hard fact associated with the test data point). The processor 204 may select a test data point (e.g., a cat image) from the first set of test data points associated with the first class (e.g., cat) to train a classifier from the set of classifiers 106 (e.g., the first classifier 106A associated with the first class). In one embodiment, the processor 204 may control the set of classifiers 106 to be trained for each class associated with the DNN 104.

[0094] At block 604, a first plurality of features and a first plurality of weights corresponding to the first plurality of features may be extracted for a first class of test data points. In one or more embodiments, the processor 204 may be configured to control the DNN 104 to extract, from the pre-trained DNN 104, a first plurality of features and a first plurality of weights corresponding to the first plurality of features for a first class of test data points. The processor 204 may extract the first plurality of features and the first plurality of weights from a convolutional layer of the DNN 104, which may be, for example, before an output layer of the DNN 104, as described with respect to the first set of features and the first set of weights in, for example, FIG. 3 and FIG. 5. Further, based on the extraction of the first plurality of features, the processor 204 may obtain a set of feature values ​​for each feature of the first plurality of features and obtain a weight corresponding to the feature from the first plurality of weights, as described with respect to, for example, FIG. 5. For example, for a test data point as a cat image, the processor 204 may extract the plurality of high-level features as a cat head, a cat tail, or a cat face. For each feature, the processor 204 may obtain a 2×2 feature matrix (or activation map) of four real values ​​(as a set of feature values) and a corresponding weight as another real number. The extraction of the first plurality of features and the first plurality of weights from the DNN 104 is described in detail, for example, in FIG. 5 with respect to the first set of features (i.e., further used to determine the correctness of the prediction of the DNN 104, as described, for example, in FIG. 3).

[0095] In some embodiments, the processor 204 may be configured to extract a second plurality of features and a second plurality of weights corresponding to the second plurality of features from a convolutional layer of the DNN 104. Additionally, the processor 204 may be configured to select the first plurality of features from the second set of extracted features based on application of a predefined selection criterion to the second plurality of weights. In such a scenario, the extraction of the first plurality of features may correspond to the selection of the first plurality of features from the second plurality of features. The predefined selection criterion may be, for example, selecting the top N features or selecting all features having corresponding weights greater than a predefined weight threshold, as described in step 404 of FIG. 4.

[0096] At block 606, a classifier (e.g., the first classifier 106A) may be trained for the first class (e.g., cat). In one or more embodiments, the processor 204 may be configured to train the first classifier 106A for the first class (e.g., cat) based on a first plurality of features extracted from the DNN 104 and a first plurality of weights corresponding to the first plurality of features, for example, as described in step 306 of FIG. 3 and FIG. 5. In one example, the first classifier 106A may be one of a decision tree classifier, a support vector machine (SVM) classifier, a naive Bayes classifier, a logistic regression classifier, or a k-nearest neighbor classifier. In an embodiment, each classifier in the set of classifiers 106 may include one type of classifier, for example, a decision tree classifier. In another embodiment, each of the sets of classifiers 106 may include different types of classifiers.

[0097] In an embodiment, to train the first classifier 106A, the processor 204 may be configured to correlate one or more first features from the first plurality of features with the correct predictions for the selected test data points. Furthermore, the processor 204 may be further configured to correlate one or more second features from the first plurality of features with the incorrect predictions for the selected test data points. In other words, to train the first classifier 106A, the processor 204 may set conditions within the first classifier 106A such that the first classifier 106A may establish a correlation between the one or more first features and the correct predictions (e.g., cat class labels) for the selected test data points (e.g., cat images). Furthermore, the first classifier 106A may establish or learn another correlation between the one or more second features and the incorrect predictions (e.g., dog class labels) for the selected test data points (e.g., cat images). For training the first classifier 106A, an example set of feature values ​​and weights corresponding to two example features (i.e., one or more first features and one or more second features) for two example test data points of cat images is shown in Table 1 as follows: [Table 1]

[0098] It should be noted that the data provided in Table 1 can be viewed as merely experimental data and is not to be construed as limiting the present disclosure.

[0099] As shown in Table 1, the first test data point may correspond to image 1 of class “cat”, and the second test data point may correspond to image 2 of the same class “cat”. For example, for training the first classifier 106A for the first class (i.e., “cat” as in column 2 of Table 1), the DNN 104 may output or predict a correct classification result as “cat” (i.e., the predicted first class shown in column 5 of Table 1) and may output or predict an incorrect classification result for image 2, e.g., as dog (i.e., the predicted first class shown in column 5 of Table 1). The first set of extracted features may include a first feature, a second feature, …, and an Nth feature. As shown in columns 3-4 of Table 1, the set of feature values ​​for the first feature (e.g., the head part of the cat) may include real numbers such as 0.1, 0.03, 1.2, and 2.3, and corresponding weights of 0.6. Based on the relatively high value of the weight (i.e., 0.6), one of those feature values ​​for the first feature (e.g., 2.3), and the predicted class of the DNN 104 (i.e., column 5 of Table 1), the processor 204 may correlate the first feature (and corresponding feature value and weight) with a correct prediction result for training the first classifier 106A for that class (e.g., cat). In other words, for a test data point of "cat," the extracted first feature (e.g., head part) may be used by the DNN 104 to predict the correct class as "cat." Thus, to train the first classifier 106A, the processor 204 may correlate the first feature (and corresponding set of feature values ​​and weights) with a correct prediction. This may indicate that the first classifier 106A may know that for that test data point (as a "cat"), the extracted first feature (i.e., the head part) may help the DNN 104 to correctly predict the test data point as a "cat". Thus, the first feature (or one or more other first features) may be a correct feature pattern for the DNN 104 to be learned by the classifier because the DNN 104 correctly predicts using such a correct feature pattern.Thus, based on learning or training the correlation between the first features and the correct predictions for the test data points, the trained classifier (i.e., first classifier 106A) can determine the correctness of the predictions to validate the DNN 104, as described in steps 308 and 310 of FIG. 3.

[0100] Similarly, as shown in column 4 of Table 1, a set of feature values ​​for the second feature (e.g., cat tail) may include real numbers such as 0.3, 0.07, 1.5, and 3.6, and a weight of 0.7. Based on the relatively high value of the weight (i.e., 0.7), one of the feature values ​​of the second feature (e.g., 3.6), and the predicted class of the DNN 104 (i.e., column 5 of Table 1), the processor 204 may correlate the second feature (and corresponding feature value and weight) with an incorrect prediction result for the training of the first classifier 106A for that class (i.e., cat). In other words, for a test data point of "cat," the extracted second feature (e.g., tail portion) may predict an incorrect class as "dog" (as shown in column 5 of Table 1) when used by the DNN 104. Thus, to train the first classifier 106A, the processor 204 can correlate the second feature (and the corresponding set of feature values ​​and weights) with the incorrect predictions. This may indicate that the first classifier 106A can learn that for a test data point (as a “cat”), the extracted second feature (i.e., the tail portion) can be an incorrect or erroneous feature, such that the DNN 104 can erroneously predict the test data point (as a “cat”) to an incorrect class as a “dog”. Thus, based on learning or training the incorrect correlation between the second feature and the incorrect predictions for the test data point, the trained classifier (i.e., the first classifier 106A) can determine the correctness of the predictions and validate the DNN 104, as described in steps 308 and 310 of FIG. 3. Such learning or training of the incorrect predictions and the features that cause the incorrect predictions of the DNN 104 may correspond to the learning error patterns of the DNN 104 by a classifier (e.g., the first classifier 106A). Thus, the classifier may be trained based on the correlations of high-level features that lead to correct predictions (i.e., correct patterns) of the DNN 104 and the correlations of high-level features that lead to incorrect predictions of the DNN 104 to effectively determine the correctness of the predictions of the DNN 104 with higher accuracy.

[0101] In some embodiments, the first classifier 106A may correspond to a decision tree classifier that may include, for example, a set of internal nodes and a set of leaf nodes. Each node of the set of internal nodes may represent a condition associated with a feature of the first plurality of features and the corresponding weight of the feature. Each node of the set of leaf nodes may represent a predicted output of the decision tree classifier. After completion of training of the decision tree classifier, the predicted output of the decision tree classifier may correspond to a second confidence score associated with a predicted class of the input data point of the DNN 104. The processor 204 may determine, for the set of internal nodes of the decision tree classifier, a set of feature values ​​of each feature of the first plurality of features and a condition associated with the first plurality of weights to generate and train the decision tree classifier, for example, as described in FIG. 7. Examples of techniques that may be used by the processor 204 to train a decision tree classifier may include, but are not limited to, an Iterative Dichotomiser 3 (ID3) algorithm, a C4.5 algorithm, a classification and regression tree (CART) algorithm, a Chi-square automatic interaction detection (CHAID) algorithm, a Multi-variate adaptive regressive splines (MARS) algorithm, or a Conditional inference tree algorithm. Examples of decision tree classifiers and conditions set for training the decision tree classifier are described in detail, for example, in FIG. 7.

[0102] At block 608, a check is performed to determine whether a predefined number of epochs of training of the first classifier 106A has been completed. In one or more embodiments, the processor 204 may be configured to perform this check. In one example, the predefined number of epochs may be one of 30 or 50. If the predefined number of epochs of training has been completed, control may proceed to block 612. Otherwise, control may proceed to block 610.

[0103] At block 610, the next data point may be reselected from the testing data set as the selected testing data point. In one or more embodiments, the processor 204 may reselect the next data point from the testing data set as the newly selected testing data point to further train a classifier (e.g., the first classifier 106A) with all the testing data points in the testing data set for the first class. Control may proceed to block 604 to repeat steps 604 and 606 of FIG. 6 again based on the newly selected testing data point.

[0104] At block 612, a trained classifier (e.g., the first classifier 106A) may be obtained for a first class associated with the pre-trained DNN 104. In one or more embodiments, the processor 204 may be configured to obtain the trained first classifier 106A for the first class (e.g., cat) based on training the first classifier 106A for a predefined number of epochs. Control may proceed to an end.

[0105] Although flowchart 600 is depicted as discrete operations, such as 602, 604, 606, 608, 610, and 612, in certain embodiments, such discrete operations may be further divided into additional operations, combined into fewer operations, or eliminated, depending on the particular implementation, without departing from the essence of the disclosed embodiments.

[0106] In some embodiments, the disclosed electronic device 102 may use the test data set to train a set of classifiers 106 for corresponding classes associated with the DNN 104 based on the aforementioned operations described in the flowchart 600. Once trained, the set of classifiers 106 may be used to verify the predicted outcome or correctness of the predictions of the DNN 104. For example, the processor 204 may use a first classifier 106A trained for a first class (e.g., “cat”) to verify the predicted outcome of the first class (e.g., “cat”) of the DNN 104. Furthermore, the processor 204 may use a second classifier 106B trained for a second class (e.g., “airplane”) to verify the predicted outcome of the second class (e.g., “airplane”) of the DNN 104. Validation of the predicted results of the DNN 104 by a corresponding classifier (i.e., one selected from the set of classifiers 106 during real-time operation of the DNN 104) may be an orthogonal validation technique that may not depend on the DNN 104 and / or its parameters, and thus may be external to the DNN 104. Such an orthogonal validation technique may provide improved validation accuracy compared to certain conventional solutions.

[0107] FIG. 7 illustrates an example decision tree classifier for validation of prediction results of a deep neural network (DNN), configured in accordance with at least one embodiment described herein. FIG. 7 is described in conjunction with elements of FIG. 1, FIG. 2, FIG. 3, FIG. 4, FIG. 5, and FIG. 6. Referring to FIG. 7, a decision tree classifier 700 is illustrated. The decision tree classifier 700 may be stored in any or a combination of the memory 206, the persistent data storage 208, or the database 108. The decision tree classifier 700 may include a set of internal nodes including a root node 702, a first internal node 704A, a second internal node 704B, and a third internal node 704C. The decision tree classifier 700 may further include a set of leaf nodes including a first leaf node 706A, a second leaf node 706B, a third leaf node 706C, and a fourth leaf node 706D. Additionally, FIG. 7 illustrates an exemplary collection of images including a first image 708A, a second image 708B, a third image 708C, and a fourth image 708D.

[0108] In one example, the processor 204 may use the trained decision tree classifier 700 (e.g., as the first classifier 106A) to verify a class predicted by the pre-trained DNN 104 for an input image or input data point. For example, the decision tree classifier 700 may be pre-trained to verify a class “cat” associated with an input image to the DNN 104. In other words, the processor 204 may be configured to use the decision tree classifier 700 to verify or trust the predicted result of the DNN 104 if the predicted result of the DNN 104 is the class “cat”. As shown in FIG. 7, a set of images 708A-708D may be associated with a respective leaf node from the set of leaf nodes 706A-706D. The set of images 708A-708D may be examples of input images of the DNN 104 for the predicted class. For example, the first image 708A may be a cat image that may be indicated as a “correct” classification by the decision tree classifier 700.

[0109] In some embodiments, each node of the set of internal nodes, including the root node 702 and the remaining nodes (i.e., nodes 704A-704C), may represent a condition for determining a decision associated with a first data point (e.g., an input image of the DNN 104). In some embodiments, the condition may be associated with a feature value from a set of feature values ​​(e.g., four values) for each feature of the first plurality of features, or a weight from a first plurality of weights corresponding to the first plurality of features, e.g., as described in Figures 5 and 6. The first plurality of features (as high-level features) and the first set of corresponding features may be extracted from the DNN 104 based on a class activation mapping (CAM) process (also referred to as interpretability analysis), e.g., as described in Figures 5 and 6. In some embodiments, the features and corresponding weights may be extracted based on a gradient-based CAM process (i.e., grad-CAM).

[0110] For example, the root node 702 may represent a condition on the weight corresponding to the 208th feature, e.g., "W208>0.894" (i.e., the weight value of the 208th feature is greater than 0.894). The root node 702 may be the most important node on which a condition on the most important feature (e.g., the 208th feature from the top features extracted from the DNN 104) may be set. Furthermore, the first internal node 704A may represent a condition on the first value of the four values ​​(i.e., set of feature values) of the 305th feature, e.g., "V305_1<0.35", the second internal node 704B may represent a condition on the third value of the four values ​​(i.e., set of feature values) of the 409th feature, e.g., "V409_3≦-0.44", and so on. Similar to the root node 702, the first internal node 704A, and the second internal node 704B, there may be N internal nodes (e.g., as shown in FIG. 7 by at least the third internal node 704C) having multiple conditions that are checked by the trained decision tree classifier 700 to further verify the correctness of the DNN 104.

[0111] During training of the decision tree classifier 700 (see FIG. 6), the processor 204 may assign a probability value to each node of the set of leaf nodes 706A-706D. The probability value represented by a leaf node of the decision tree classifier 700 may be a second confidence score associated with a class predicted by the pre-trained DNN 104 for a first data point (e.g., an input image). In other words, the probability value or the second confidence score at a leaf node may be used to validate the predicted class of the DNN 104, for example, as described in step 308 of FIG. 3. In some embodiments, the probability value may be an orthogonal confidence score of the decision tree classifier 700. In some embodiments, the second confidence score may be a discrete value of either 0 (indicating an “incorrect” prediction) or 1 (indicating a “correct” prediction) based on a comparison of the probability value to a threshold (e.g., 0.5). For example, the second confidence score for probability values ​​less than 0.5 may be determined as "0," and the second confidence score for probabilities greater than 0.5 may be determined as "1." In other embodiments, the second confidence score may be the same as the probability value of the leaf node.

[0112] For example, as shown in FIG. 7, for the decision tree classifier 700 (trained for the class as cat), the first leaf node 706A may represent a "correct" prediction (e.g., a value of "1" for the second confidence score) associated with the image class of "cat" as evidenced by the first image 708A of the cat. Meanwhile, the second leaf node 706B may represent an "incorrect" prediction (e.g., a value of "0" for the second confidence score) for the image class of "cat" based on the set condition in the second internal node 704B for the 409th feature of the features extracted from the DNN 104 based on the CAM process (i.e., interpretability analysis). The incorrect prediction may be evident by the second image 708B of a dog. FIG. 7 further illustrates that for the image class of "cat", the third leaf node 706C may represent a "correct" prediction and the fourth leaf node 706D may represent an "incorrect" prediction based on the set condition in the third internal node 704B. This may be evident as shown in FIG. 7 by a third image 708C of a cat and a fourth image 708D of a bird.

[0113] The decision tree classifier 700 shown in FIG. 7 may be an exemplary pre-trained classifier of the set of classifiers 106, although the scope of the present disclosure may not be so limited. The set of classifiers 106 may include other classifiers, including but not limited to a support vector machine (SVM) classifier, a naive Bayes classifier, a logistic regression classifier, or a k-nearest neighbor classifier. Thus, the decision tree classifier 700 (i.e., one of the set of classifiers 106) may be trained by the disclosed electronic device 102 with respect to a decision condition (in each of the root node 702 and the set of internal nodes 704A-704C) based on the correlation established between the features (i.e., a set of feature values) and corresponding weights (i.e., extracted from the DNN 104 based on the CAM process) and the correct or incorrect predictions (i.e., as indicated by the set of leaf nodes 706A-706D). Such a trained classifier (e.g., a decision tree classifier 700 orthogonal to the DNN 104) can memorize or learn all correct predictions and associated correct features (i.e., success patterns) and / or incorrect predictions and associated erroneous features (i.e., error patterns) of the DNN 104 to further verify the correctness of the DNN 104 and further increase the predictive accuracy of the DNN 104 in various real-time applications.

[0114] An exemplary dataset and experimental setup for the present disclosure is presented in Table 2 as follows: [Table 2]

[0115] It should be noted that the data provided in Table 2 can be viewed merely as experimental data and is not to be construed as limiting the present disclosure.

[0116] Exemplary experimental data for validation of the prediction results of the DNN with the pre-trained classifier is shown in Table 3 as follows: [Table 3]

[0117] From Table 3, it can be seen that, of the entire first set of 55 test images of the "dog" class, 51 images can be correctly verified by the trained classifier (i.e., trained for the "dog" class) as belonging to the "dog" class, and the remaining 4 images can be incorrectly verified as not belonging to the "dog" class. Thus, the precision of the experimental data represented for the trained classifier in Tables 2-3 can be 93%. Furthermore, for each of the second set of 55 test images of other classes, the same classifier (i.e., trained on the dog class) can determine that the image can be incorrectly classified into the "dog" class. In other words, the classifier can determine that none of the test images of the second set belong to the "dog" class. Thus, the recall can be 100%.

[0118] Various embodiments of the present disclosure may provide one or more non-transitory computer-readable storage media configured to store instructions that, in response to being executed, cause a system (such as the exemplary system 202) to perform an operation. The operation may include receiving a first data point associated with a real-time application. The operation may further include predicting a first class associated with the received first data point by a deep neural network (DNN) pre-trained for a classification task of the real-time application. The operation may further include extracting, from the DNN, a first set of features and a first set of weights corresponding to the first set of features for the predicted first class of the first data point. The extracted first set of features may be associated with a convolutional layer of the DNN. The operation may further include determining a confidence score for the predicted first class for the first data point based on the first set of features extracted and the corresponding first set of weights by a pre-trained classifier associated with the predicted first class. The operations may further include generating output information indicative of a correctness of the first class prediction for the first data point based on the determined confidence score.

[0119] As used in this disclosure, the term "module" or "component" may refer to a specific hardware implementation configured to perform the actions of the module or component, and / or a software object or software routine that may be stored and / or executed on general-purpose hardware (e.g., computer-readable medium, processing unit, etc.) of a computing system. In some implementations, various components, modules, engines, and services described in this disclosure may be implemented as objects or processes that run on a computing system (e.g., as separate threads). Although some of the systems and methods described in this disclosure are generally described as being implemented in software (stored and / or executed on general-purpose hardware), specific hardware embodiments or combinations of software and specific hardware embodiments are also possible and contemplated. In this description, a "computing entity" may be any computing system previously defined in this disclosure, or a combination of any modules or sub-modules operating on a computing system.

[0120] The terms used in this disclosure, particularly in the appended claims (e.g., the body of the appended claims), are generally intended to be "open" terms (e.g., the term "including" should be interpreted as "including, but not limited to," the term "having" should be interpreted as "having at least," the term "including" should be interpreted as "including, but not limited to," etc.).

[0121] Moreover, where a particular number of claim recitations to be introduced is intended, such intent is expressly recited in the claim, and in the absence of such recitation, no such intent exists. For example, as an aid to understanding, the following appended claims may contain the use of the introductory phrases "at least one" and "one or more" to introduce claim recitations, but the use of such phrases should not be construed as implying that the introduction of a claim recitation with the indefinite article "a" or "an" limits a particular claim that includes a claim recitation so introduced to embodiments that include only one such recitation. This is true even if the same claim includes the introductory phrases "one or more" or "at least one" and an indefinite article such as "a" or "an" (e.g., "a" and / or "an" should be construed to mean "at least one" or "one or more"). The same is true for the use of definite articles used to introduce claim recitations.

[0122] Moreover, even when a particular number of introduced claim recitations is explicitly recited, one of ordinary skill in the art will recognize that such recitation should be interpreted to mean at least the recited number (e.g., the recitation "two recitations" without other modifiers means at least two recitations or more than two recitations). Furthermore, when similar idiomatic expressions such as "at least one of A, B, C, etc." or "one or more of A, B, C, etc." are used, such constructions are generally intended to include A only, B only, C only, A and B, A and C, B and C, or A, B and C, etc.

[0123] Furthermore, disjunctive phrases, whether in the specification, claims, or drawings, presenting two or more alternative terms, should be understood to contemplate the possibility of including one of those terms, either one of those terms, or both terms. For example, the phrase "A or B" should be understood to include the possibilities of "A" or "B" or "A and B."

[0124] All examples and conditional language described in this disclosure are intended for educational purposes to aid the reader in understanding the concepts contributed by the inventor to advance the present disclosure and the art, and should be construed as being without limitation to the examples and conditions specifically described as such. Although the embodiments of the present disclosure have been described in detail, various changes, substitutions, and alterations can be made thereto without departing from the spirit and scope of the present disclosure.

[0125] The following supplementary notes are further disclosed regarding the embodiments including the above examples. (Appendix 1) receiving first data points associated with a real-time application; predicting, by a deep neural network (DNN) pre-trained for a classification task of the real-time application, a first class associated with the received first data point; extracting from the DNN a first set of features and a first set of weights corresponding to the first set of features for a predicted first class of the first data point, the extracted first set of features being associated with a convolutional layer of the DNN; determining a confidence score for the predicted first class for the first data point based on the first set of features extracted by the pre-trained classifier associated with the predicted first class and the first set of corresponding weights; and generating output information indicative of a correctness of the first class prediction for the first data point based on the determined confidence score. method. (Appendix 2) The extraction of the first set of features further comprises: extracting from the DNN a second set of features and a second set of weights corresponding to the second set of features, the extracted second set of features being associated with the convolutional layer of the DNN; selecting the first set of features from the second set of features based on the second set of weights. The method described in Appendix 1. (Appendix 3) The selection of the first set of features from the second set of features further comprises: comparing each weight in the second set of weights corresponding to the second set of features to a predetermined weight threshold; selecting the first set of features from the second set of features based on the comparison. The method described in Appendix 2. (Appendix 4) The selection of the first set of features from the second set of features further comprises: sorting the second set of features based on application of a predetermined sorting criterion to the second set of weights corresponding to the second set of features; selecting the first set of features from the sorted second set of features. The method described in Appendix 2. (Appendix 5) 2. The method of claim 1, wherein the convolutional layer corresponds to a layer preceding an output layer of the DNN. (Appendix 6) comparing the determined confidence score to a predefined score threshold; and generating the output information indicative of a correctness of the first class prediction based on the comparison. The method described in Appendix 1. (Appendix 7) 2. The method of claim 1, wherein the output information corresponds to at least one of displaying the confidence score, storing the confidence score in a log file, or notifying based on the confidence score. (Appendix 8) selecting a pre-trained classifier for the predicted first class from a set of pre-trained classifiers each associated with a corresponding class of the pre-trained DNN; and determining the confidence score of a predicted first class for the first data point by the selected pre-trained classifier. The method described in Appendix 1. (Appendix 9) determining the confidence score for the predicted first class for the first data point based on a correlation between one or more first features of the convolutional layer of the DNN and a correct prediction and a correlation between one or more second features of the convolutional layer of the DNN and an incorrect prediction; the first set of features includes the one or more first features and the one or more second features; The method described in Appendix 1. (Appendix 10) 10. The method of claim 9, wherein each feature from the one or more first features and the one or more second features comprises a set of feature values, and the correlation with the correct prediction or the incorrect prediction of the first class is based on the set of feature values ​​and a weight corresponding to each feature. (Appendix 11) Training the classifier for the first class includes: selecting test data points of the first class from a test dataset of a plurality of test data points associated with the real-time application, the test dataset including a first set of test data points associated with the first class and a second set of test data points associated with one or more classes other than the first class; performing a first set of operations for training the classifier for the first class, the first set of operations comprising: extracting from the pre-trained DNN a first plurality of features of the first class of selected test data points and a first plurality of features and a first plurality of weights, the extracted first plurality of features being associated with the convolutional layer of the DNN; training the classifier for the first class based on the extracted first plurality of features and corresponding first plurality of weights; and reselecting a next test data point from the test data set as the selected test data point; and obtaining the pre-trained classifier for the first class by iteratively performing the first set of operations on the test data set based on the selected test data points. The method described in Appendix 1. (Appendix 12) The extraction of the first plurality of features further includes: extracting a second plurality of features and a second plurality of weights corresponding to the second plurality of features from the DNN, the extracted second plurality of features being associated with the convolutional layer of the DNN; selecting the first plurality of features from the second plurality of features based on the second plurality of weights. 12. The method according to claim 11. (Appendix 13) The training of the classifier for the first class further includes: correlating one or more first features from the first plurality of features with a correct prediction for the selected test data points; and correlating one or more second features from the first plurality of features with the incorrect predictions for the selected test data points. 12. The method according to claim 11. (Appendix 14) 2. The method of claim 1, wherein the pre-trained classifier corresponds to one of a decision tree classifier, a support vector machine (SVM) classifier, a naive Bayes classifier, a logistic regression classifier, or a k-nearest neighbor classifier. (Appendix 15) The pre-trained classifier: a set of interior nodes, each of which represents a condition associated with a feature of the first set of features and a corresponding weight of the feature; a set of leaf nodes representing prediction outputs, each of the leaf nodes including the confidence score associated with the predicted first class for the first data point; The method described in Appendix 1. (Appendix 16) 2. The method of claim 1, wherein the first data point corresponds to one of image data, audio data, or text data. (Appendix 17) 2. The method of claim 1, wherein the real-time application includes one of image classification, speech recognition, or text recognition. (Appendix 18) One or more non-transitory computer-readable storage media configured to store instructions that, in response to being executed, cause a system to perform operations, the operations including: receiving first data points associated with a real-time application; predicting, by a deep neural network (DNN) pre-trained for a classification task of the real-time application, a first class associated with the received first data point; extracting from the DNN a first set of features and a first set of weights corresponding to the first set of features for a predicted first class of the first data point, the extracted first set of features being associated with a convolutional layer of the DNN; determining a confidence score for the predicted first class for the first data point based on the first set of features extracted by the pre-trained classifier associated with the predicted first class and the first set of corresponding weights; and generating output information indicative of a correctness of the first class prediction for the first data point based on the determined confidence score. Medium. (Appendix 19) With a pre-trained deep neural network (DNN) for classification tasks in real-time applications; A system comprising a processor coupled to the DNN, the processor comprising: receiving first data points associated with the real-time application; predicting, by the pre-trained DNN, a first class associated with a first data point received; extracting from the pre-trained DNN a first set of features and a first set of weights corresponding to the first set of features for a predicted first class of the first data point, the extracted first set of features being associated with a convolutional layer of the DNN; determining a confidence score for the predicted first class for the first data point based on the first set of features extracted by the pre-trained classifier associated with the predicted first class and the first set of corresponding weights; and generating output information indicative of a correctness of the prediction of the first class for the first data point based on the determined confidence score. system. (Appendix 20) 20. The system of claim 19, wherein the output information corresponds to at least one of displaying the confidence score, storing the confidence score in a log file, or a notification based on the confidence score. [Explanation of symbols]

[0126] 102 Electronic equipment 104 Deep Neural Networks (DNN) 106 Ensemble of Classifiers 108 Database 110 User end device 112 Communication Networks 160A First Classifier 160B Second Classifier 160N Nth classifier 202 System 204 Processor 206 Memory 208 Persistent Data Storage 210 I / O Devices 212 Network Interface 302 First data point received 304 Predicting a first class associated with the first data point received by a pre-trained deep neural network (DNN). Extract from the 306 DNN a first set of features for a predicted first class of a first data point and a first set of weights corresponding to the first set of features. 308. Determine a confidence score for a predicted first class for a first data point by a pre-trained classifier associated with the predicted first class based on the first set of features and the corresponding first set of weights. 310 Generate output information indicating the correctness of the prediction of the first class for the first data point based on the determined confidence score. Extract from the 402 DNN a second set of features for the predicted first class of the first data point and a second set of weights corresponding to the second set of features. 404 Selecting a first set of features from the second set of extracted features based on a second set of weights corresponding to the second set of features. 602 Selecting test data points related to a first class from a test data set including a plurality of test data points related to a real-time application 604 Extracting from the pre-trained DNN a first plurality of features and a first plurality of weights corresponding to the first plurality of features for a first class for a selected test data point. 606 Training a classifier for a first class based on the first plurality of features and the corresponding first plurality of weights. 608 Predetermined number of epochs of training classifier completed? 610 Reselect the next data point from the test data set as the selected test data point 612 Get the trained classifier for the first class

Claims

1. receiving a first data point associated with a real-time application; predicting, by a deep neural network (DNN) pre-trained for a classification task of the real-time application, a first class associated with the received first data point; extracting from the DNN a first set of features and a first set of weights corresponding to the first set of features for a predicted first class of the first data point, the extracted first set of features being associated with a convolutional layer of the DNN; determining a confidence score for the predicted first class for the first data point based on a first set of features extracted by a pre-trained classifier associated with the predicted first class and a first set of corresponding weights; generating output information indicative of a correctness of the first class prediction for the first data point based on the determined confidence score; The extraction of the first set of features further includes: extracting from the DNN a second set of features and a second set of weights corresponding to the second set of features, the extracted second set of features being associated with the convolutional layer of the DNN; selecting the first set of features from the second set of features based on the second set of weights. method.

2. The selection of the first set of features from the second set of features further comprises: comparing each weight in the second set of weights corresponding to the second set of features to a predetermined weight threshold; selecting the first set of features from the second set of features based on the comparison. The method of claim 1.

3. The selection of the first set of features from the second set of features further comprises: sorting the second set of features based on application of a predetermined sorting criterion to the second set of weights corresponding to the second set of features; selecting the first set of features from the sorted second set of features. The method of claim 1.

4. The method of claim 1 , wherein the convolutional layer corresponds to a layer preceding an output layer of the DNN.

5. comparing the determined confidence score with a predefined score threshold; and generating the output information indicative of a correctness of the first class prediction based on the comparison. The method of claim 1.

6. The method of claim 1 , wherein the output information corresponds to at least one of: displaying the confidence score, storing the confidence score in a log file, or providing a notification based on the confidence score.

7. A method for determining whether a first data point is associated with a real-time application; and predicting, by a deep neural network (DNN) pre-trained for a classification task of the real-time application, a first class associated with the received first data point; selecting a pre-trained classifier for the predicted first class from a set of pre-trained classifiers each associated with a corresponding class of the pre-trained DNN; extracting from the DNN a first set of features and a first set of weights corresponding to the first set of features for a predicted first class of the first data point, the extracted first set of features being associated with a convolutional layer of the DNN; determining a confidence score for a predicted first class for a first data point based on a first set of extracted features and corresponding first set of weights by the selected pre-trained classifier; generating output information indicative of a correctness of the first class prediction for the first data point based on the determined confidence score; method.

8. The method of claim 1, further comprising: receiving a first data point associated with a real-time application; predicting, by a deep neural network (DNN) pre-trained for a classification task of the real-time application, a first class associated with the received first data point; extracting from the DNN a first set of features and a first set of weights corresponding to the first set of features for a predicted first class of the first data point, the extracted first set of features being associated with a convolutional layer of the DNN; determining a confidence score for the predicted first class for the first data point based on a first set of features extracted by a pre-trained classifier associated with the predicted first class and a first set of corresponding weights; generating output information indicative of a correctness of the first class prediction for the first data point based on the determined confidence score, The method further comprises: determining the confidence score for the predicted first class for the first data point based on a correlation between one or more first features of the convolutional layer of the DNN and correct predictions and a correlation between one or more second features of the convolutional layer of the DNN and incorrect predictions; the first set of features includes the one or more first features and the one or more second features; method.

9. 9. The method of claim 8, wherein each feature from the one or more first features and the one or more second features comprises a set of feature values, and the correlation with the correct prediction or the incorrect prediction of the first class is based on the set of feature values ​​and a weight corresponding to each feature.

10. The method of claim 1, further comprising: receiving a first data point associated with a real-time application; predicting, by a deep neural network (DNN) pre-trained for a classification task of the real-time application, a first class associated with the received first data point; extracting from the DNN a first set of features and a first set of weights corresponding to the first set of features for a predicted first class of the first data point, the extracted first set of features being associated with a convolutional layer of the DNN; determining a confidence score for the predicted first class for the first data point based on a first set of features extracted by a pre-trained classifier associated with the predicted first class and a first set of corresponding weights; generating output information indicative of a correctness of the first class prediction for the first data point based on the determined confidence score; Training the classifier for the first class includes: selecting test data points of the first class from a test dataset of a plurality of test data points associated with the real-time application, the test dataset including a first set of test data points associated with the first class and a second set of test data points associated with one or more classes other than the first class; performing a first set of operations for training the classifier for the first class, the first set of operations comprising: Extracting a first plurality of features of the first class of selected test data points and a first plurality of features and a first plurality of weights from the pre-trained DNN, the extracted first plurality of features being associated with the convolutional layer of the DNN; training the classifier for the first class based on the extracted first plurality of features and corresponding first plurality of weights; and reselecting a next test data point from the test data set as the selected test data point; and obtaining the pre-trained classifier for the first class by iteratively performing the first set of operations on the test data set based on the selected test data points. method.

11. The extraction of the first plurality of features further includes: extracting a second plurality of features and a second plurality of weights corresponding to the second plurality of features from the DNN, the extracted second plurality of features being associated with the convolutional layer of the DNN; selecting the first plurality of features from the second plurality of features based on the second plurality of weights. The method of claim 10.

12. The training of the classifier for the first class further includes: correlating one or more first features from the first plurality of features with a correct prediction for the selected test data points; and correlating one or more second features from the first plurality of features with the incorrect predictions for the selected test data points. The method of claim 10.

13. 2. The method of claim 1, wherein the pre-trained classifier corresponds to one of a decision tree classifier, a support vector machine (SVM) classifier, a naive Bayes classifier, a logistic regression classifier, or a k-nearest neighbor classifier.

14. The pre-trained classifier: a set of interior nodes, each of which represents a condition associated with a feature of the first set of features and a corresponding weight of the feature; a set of leaf nodes representing prediction outputs, each of the leaf nodes including the confidence score associated with the predicted first class for the first data point; The method of claim 1.

15. The method of claim 1 , wherein the first data point corresponds to one of image data, audio data, or text data.

16. The method of claim 1 , wherein the real-time application includes one of image classification, speech recognition, or text recognition.

17. One or more non-transitory computer-readable storage media configured to store instructions that, in response to being executed, cause a system to perform operations, the operations including: receiving a first data point associated with a real-time application; predicting, by a deep neural network (DNN) pre-trained for a classification task of the real-time application, a first class associated with the received first data point; selecting a pre-trained classifier for the predicted first class from a set of pre-trained classifiers each associated with a corresponding class of the pre-trained DNN; extracting from the DNN a first set of features and a first set of weights corresponding to the first set of features for a predicted first class of the first data point, the extracted first set of features being associated with a convolutional layer of the DNN; determining a confidence score for a predicted first class for a first data point based on a first set of extracted features and corresponding first set of weights by the selected pre-trained classifier; generating output information indicative of a correctness of the first class prediction for the first data point based on the determined confidence score; Medium.

18. A pre-trained deep neural network (DNN) for classification tasks in real-time applications; 1. A system comprising a processor coupled to the DNN, the processor comprising: receiving a first data point associated with the real-time application; predicting, by the pre-trained DNN, a first class associated with a received first data point; selecting a pre-trained classifier for the predicted first class from a set of pre-trained classifiers each associated with a corresponding class of the pre-trained DNN; extracting from the pre-trained DNN a first set of features and a first set of weights corresponding to the first set of features for a predicted first class of the first data point, the extracted first set of features being associated with a convolutional layer of the DNN; determining a confidence score for a predicted first class for a first data point based on a first set of extracted features and corresponding first set of weights by the selected pre-trained classifier; generating output information indicative of the correctness of the prediction of the first class for the first data point based on the determined confidence score. system.

19. 20. The system of claim 18, wherein the output information corresponds to at least one of: displaying the confidence score, storing the confidence score in a log file, or providing a notification based on the confidence score.

Images