Systems, methods and apparatus for implementing transparency models for computer vision and image recognition utilizing deep learning non-transparent black box models

Abstract

Transparent models are created for computer vision and image recognition utilizing deep learning non-transparent black box models. Explainable AI is created by training a convolutional neural network to classify objects and a multi-layer perceptron to recognize both objects and object parts. An image is received with an object embedded within it. The CNN and explainable AI model are run within the image recognition system to generate a prediction of the object in the image via the explainable AI model, recognize the object parts, provide the recognized parts in the object as evidence for the object prediction, and generate an explanation for why the image system predicted the object in the image based on the evidence including the recognized parts.

Description

[Technical field]

[0001] Claiming priority This patent application, filed under the Patent Cooperation Treaty (PCT), is related to and claims priority to U.S. Provisional Patent Application No. 63 / 236,393, entitled “SYSTEMS, METHODS, AND APPARATUSES FOR A TRANSPARENT MODEL FOR COMPUTER VISION / IMAGE RECOGNITION FROM A DEEP LEARNING NONTRANSPARENT BLACK BOX MODEL,” filed on August 24, 2021, and having Attorney Docket No. 37684.671P, the entire contents of which are incorporated by reference herein as if fully set forth.

[0002] Notice of Government Rights and Government Assistance Supporting grants include: 2021 Dean's Excellence in Research Summer Research Grant, W. P. Carey School of Business, ASU, and 2020 Dean's Excellence in Research Summer Research Grant, W. P. Carey School of Business, ASU.

[0003] Copyright Notice A portion of the disclosure of this patent document contains material that is subject to copyright protection. The copyright owner has no objection to anyone copying the patent document or the patent disclosure as it appears in the U.S. Patent and Trademark Office patent file or records, but otherwise reserves all copyright rights whatsoever.

[0004] Embodiments of the present invention relate generally to the field of computer vision / image recognition from deep learning non-transparent black box models for use in any application area of ​​deep learning for computer vision, including but not limited to military and medical applications that benefit from transparent and trustworthy models. [Background technology]

[0005] The subject matter discussed in this Background section should not be deemed to be prior art as a result of its mere mention in the Background section. Likewise, the problems described in the Background section or related to the subject matter of the Background section should not be deemed to have been previously recognized in the prior art. The subject matter in the Background section is merely representative of various approaches, which may themselves correspond to embodiments of the claimed invention.

[0006] Deep learning (also known as deep structured learning) is part of a broader family of machine learning methods based on artificial neural networks (ANNs) with representation learning. Learning can be supervised, semi-supervised, or unsupervised.

[0007] Deep learning architectures, such as deep neural networks, deep belief networks, deep reinforcement learning, recurrent neural networks, and convolutional neural networks, have been applied to areas including computer vision, speech recognition, natural language processing, machine translation, bioinformatics, drug design, medical image analysis, materials testing, and board game programming.

[0008] The adjective "deep" in deep learning refers to the use of multiple layers in the network. Early work showed that while linear perceptrons cannot be general-purpose classifiers, networks with non-polynomial activation functions and one hidden layer of infinite width can. Deep learning is a modern variant that concerns an infinite number of layers of finite size, preserving the theoretical generality under loose conditions while allowing practical applications and optimal implementations. In deep learning, the multiple layers are the "structured" part, since they can be heterogeneous and can deviate widely from biologically informed connectionist models for purposes of efficiency, trainability, and understandability.

[0009] Machine learning has been a tremendous success as a technology, along with the advent of deep learning. However, most deployments of this technology are in low-risk domains. Two potential application domains for deep learning-based image recognition systems - military and medical - are hesitant to use this technology because these deep learning models are non-transparent black box models that are almost impossible for anyone to understand. Summary of the Invention [Problem to be solved by the invention]

[0010] What is needed is a transparent and trustworthy model. [Means for solving the problem]

[0011] Therefore, the current state of the art can benefit from systems, methods and apparatus that implement transparency models for computer vision and image recognition that utilize deep learning non-transparent black box models as described herein.

[0012] The embodiments may be better understood with reference to the following detailed description, given by way of example and not by way of limitation, and considered in conjunction with the drawings in which: [Brief description of the drawings]

[0013] [Figure 1] FIG. 1 is a schematic diagram of an exemplary architecture of a DARPA-compliant Explainable AI (XAI) model having explanatory refinements and implemented for informed users, according to a described embodiment. [Diagram 2] FIG. 2 is an illustration of a technique according to an embodiment of the invention for classifying images into four distinct classes, according to a described embodiment. [Diagram 3] FIG. 2 is an illustration of a technique according to an embodiment of the invention for classifying images into two fine-grained classes, according to a described embodiment. [Figure 4] FIG. 1 illustrates transfer learning for a new classification task that involves only training the weights of an additional fully connected layer of a CNN, according to a described embodiment. [Diagram 5] FIG. 1 is a diagram of training a separate multi-target MLP according to a described embodiment, where the input comes from the activation functions of the fully connected layer of a CNN, and the output nodes of the MLP correspond to both objects and object parts. [Figure 6A] FIG. 1 is a diagram of training a separate multi-label MLP, where the input is the activation function of the fully connected layer of a CNN, according to a described embodiment. [Figure 6B] FIG. 6 is a diagram of training a multi-label CNN 601 to recognize objects and parts, as well as learn about configurations and connectivity, according to a described embodiment. [Figure 6C] FIG. 13 is a diagram of training a single-label CNN to recognize both objects and parts from parts and part connectivity, but not object configuration, according to a described embodiment. [Figure 7] 1A-1C are sample images of various parts of a cat, according to a described embodiment. [Figure 8] FIG. 13 is a diagram of sample images of various parts of a bird, according to a described embodiment. [Figure 9] 1A-1C are sample images of various parts of a car, according to a described embodiment. [Figure 10] 1A-1C are sample images of various parts of a motorcycle, according to a described embodiment; [Figure 11]1A-1C are sample images of a husky's eyes and a husky's ears, according to a described embodiment; [Figure 12] 1A-1C are diagrams of sample images of wolf eyes and wolf ears, according to a described embodiment; [Figure 13] FIG. 1 is a diagram of Table 1 according to a described embodiment, showing which learning takes place within a CNN+MLP architecture. [Figure 14] FIG. 11 is a diagram of Table 2 according to a described embodiment showing the number of images used for training and testing the CNN and MLP. [Figure 15] FIG. 1 is a diagram of Table 3 according to a described embodiment, showing results for the "car, motorcycle, cat, bird" classification problem. [Figure 16] FIG. 1 is a diagram of Table 4 according to a described embodiment, showing the results of the "cats vs dogs" classification problem. [Figure 17] FIG. 13 is a diagram of Table 5 according to a described embodiment, showing the results of the "huskies and wolves" classification problem. [Figure 18] FIG. 11 is a diagram of Table 6 according to the described embodiment, showing the results of comparing the best prediction accuracy of the CNN and XAI-MLP models. [Figure 19] 1 is a diagram of an image of the number "5" and a wolf, where the number "5" has been modified by a fast gradient method at various epsilon values, and the wolf image has also been modified by a fast gradient method at various epsilon values, according to a described embodiment. [Figure 20] FIG. 1 is a diagram of an example base CNN model utilizing a custom convolutional neural network architecture for MNIST, according to a described embodiment. [Figure 21] FIG. 1 is a diagram of an example base XAI-CNN model utilizing a custom convolutional neural network architecture for a MNIST explainable AI model, according to a described embodiment. [Figure 22] FIG. 11 is a diagram of Table 7 according to the described embodiment, showing the average test accuracy of the MNIST base CNN model on adversarial images generated by different epsilon values ​​over 10 different runs. [Diagram 23] FIG. 11 is a diagram of Table 8 according to the described embodiment, showing the average test accuracy of the XAI-CNN model on adversarial images generated by different epsilon values ​​across 10 different runs. [Figure 24] FIG. 11 is a plot of Table 9 according to the described embodiment, showing the average test accuracy of the base CNN model of huskies and wolves on adversarial images generated with different epsilon values ​​across 10 different runs. [Diagram 25] FIG. 10 is a diagram of Table 10 according to a described embodiment, showing the average test accuracy of the XAI-CNN models for huskies and wolves on adversarial images generated with different epsilon values ​​across 10 different runs. [Figure 26] 1 is a flow diagram illustrating a method for implementing a transparency model for computer vision and image recognition utilizing a deep learning non-transparent black box model, according to disclosed embodiments. [Figure 27] 1 is a schematic diagram of a system in which an embodiment may be run, installed, integrated or configured. [Figure 28] 1 is a schematic diagram of an exemplary embodiment of a computer system according to one embodiment of the present invention; DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0014] Described herein are systems, methods, and apparatus for implementing transparency models for computer vision and image recognition that utilize deep learning opacity black box models.

[0015] The Defense Advanced Research Projects Agency (DARPA) recognized the problems with deep learning for computer vision and has launched a program called Explainable AI ("XAI") which adopts the following goals:

[0016] According to DARPA, the Explainable AI (XAI) program aims to create a suite of machine learning techniques that produce more explainable models while maintaining high levels of learning performance (predictive accuracy) and enable human users to understand, appropriately trust, and effectively manage emerging generations of artificial intelligence partners.

[0017] DARPA further explains that XAI has led to dramatic successes in machine learning, leading to a wave of artificial intelligence (AI) applications. DARPA claims that continued evolution promises to produce autonomous systems that perceive, learn, reason, and act on their own. However, the effectiveness of these systems is limited by current machines' inability to explain their decisions and actions to human users. According to DARPA, the Department of Defense ("DoD") faces challenges that require more intelligent, autonomous, and symbiotic systems. Explainable AI - and especially explainable machine learning - is essential if future soldiers are to understand, properly trust, and effectively manage their emerging generation of artificially intelligent machine partners.

[0018] Thus, DARPA explains that the Explainable AI (XAI) program aims to generate a suite of machine learning techniques that will generate more explainable models while maintaining a high level of learning performance (predictive accuracy) and allow human users to understand, appropriately trust, and effectively manage the emerging generation of artificial intelligence partners. DARPA further explains that new machine learning systems will in the future have the ability to explain the rationale of new machine learning systems, characterize their strengths and weaknesses, and convey an understanding of how they behave. The path to achieving this goal is to develop new or modified machine learning techniques that generate more explainable models. According to DARPA, such models will be combined with state-of-the-art human-computer interface techniques that can translate the models into explanatory dialogue that is understandable and useful to end users. DARPA claims that its path is to pursue a variety of techniques to generate a portfolio of models that will provide future developers with a set of design options that cover the performance-versus-explainability trade space.

[0019] DARPA provides further context by explaining that XAI is one of a handful of current DARPA programs expected to enable "Tertiary AI systems," in which machines understand situations and environments that allow them to run and build underlying explanatory models over time, and characterize real-world phenomena using these explanatory models. According to DARPA, the XAI program will focus on the development of multiple systems by addressing challenging problems in two domains: (1) the machine learning problem of classifying events of interest in heterogeneous multimedia data, and (2) the machine learning problem of autonomous systems building decision policies to perform a variety of simulated missions. These two challenging problem domains were chosen because they represent the intersection of two important machine learning techniques (classification and reinforcement learning) and two important operational problem domains for the DoD (intelligence analysis and autonomous systems).

[0020] DARPA further states that researchers are investigating the psychology of explanations and, more specifically, that XAI research prototypes are being tested and continuously evaluated throughout the life of the program. In May 2018, XAI researchers demonstrated an early implementation of an explainable learning system and presented the results of an initial pilot study for a first phase evaluation of the explainable learning system. Evaluation of the full first phase system is scheduled for November 2018. The final deliverable at the end of the program will be a toolkit library consisting of machine learning and human-computer interface software modules that can be used for the development of future explainable AI systems. Upon completion of the program, these toolkits will be available for further refinement and transition to defense or commercial applications.

[0021] Exemplary embodiments

[0022] Certain embodiments of the present invention generate transparency models for computer vision and image recognition from deep learning non-transparent black box models, where the generated transparency models are consistent with DARPA's stated objectives through its Explainable AI (XAI) program. For example, if a disclosed image recognition system predicts that an image is an image of a cat, in addition to a representation that would otherwise be a non-transparent "black box" prediction, the disclosed system further provides an explanation for why the system "thinks" the image is an image of a cat or expresses a prediction that the image is an image of a cat. For example, such an exemplary system may output an explanation that supports the prediction that the transparency model running on the computer vision and image recognition believes the image is an image of a cat because the entity in the image appears to include whiskers, fur, and claws.

[0023] With such supporting explanations as to “why” the system made a particular prediction, it can no longer be described as a non-transparent or black box predictive model.

[0024] In one sense, DARPA's desired XAI system is based on recognizing parts of an object and presenting those parts as evidence to predict the object. Embodiments of the present invention, described in more detail below, implement this desired functionality.

[0025] An embodiment of the present invention further includes a computer-implemented method that is specifically configured to decode a Convolutional Neural Network (CNN), a type of deep learning model, to recognize object parts. A separate model (a multi-layer perceptron) actually learns to decode the CNN, which provides information about the object's configuration from its parts and part connectivity. This second model also embodies the symbolic information for explainable AI. It has been experimentally demonstrated that encoding of object parts exists at multiple levels of the CNN, and part information can be easily extracted to explain the reasoning behind the classification decision. The overall approach to an embodiment of the present invention is similar to teaching a human about object parts.

[0026] According to an exemplary embodiment, the following information is provided to the second model: information about the configuration of the object from its parts, including information about subassemblies and connectivity between parts. The configuration information is provided by a list of parts. For example, for a cat's head, the list may include eyes, nose, ears, and mouth. The embodiment may implement the overall approach in various ways. Conventional wisdom is to sacrifice accuracy for explainability. However, experimental results with this method show that explainability can substantially improve the accuracy of many CNN models. Furthermore, since the parts of the object are predicted by the second model and not just the object, there is a good chance that adversarial training will not be necessary.

[0027] The impact of the current state of the art, and in particular the commercial viability of such disclosed embodiments, may impact numerous application areas. For example, currently the U.S. military does not deploy existing deep learning based image recognition systems without explanation capabilities. Thus, the disclosed embodiments of the invention described herein may serve to open up that market and improve the capabilities and readiness of the U.S. military. Furthermore, in addition to national defense and military readiness, numerous other application areas would benefit from such explanation capabilities, such as medical diagnostic applications, human-computer interfaces, more efficient communication protocols, and even improvements in entertainment content delivery and game engines.

[0028] Several novel aspects associated with the described embodiments of the invention are detailed further below, including the following:

[0029] An embodiment that has a means for generating exactly the kinds of explainable AI (XAI) models envisioned by DARPA, while acknowledging that there is currently no known prior art capable of meeting the stated objectives.

[0030] An embodiment having a means for expressing predictions that conform to the DARPA XAI model of an object (e.g., a cat) based on examination of unique parts of the object (e.g., whiskers, fur, and claws).

[0031] An embodiment having a means for generating new predictive models trained to recognize unique parts of an object.

[0032] Recognizing that there is currently no known prior art that follows a procedure for teaching a model to recognize parts of various objects by showing them images of those parts, an embodiment has a means for teaching a model to recognize those parts by showing them images of those parts (e.g., an elephant's trunk).

[0033] An embodiment having a means for teaching constructivities of new models of objects (and subassemblies) from basic parts and part connectivity. For example, such an embodiment "teaches" or "learns" a model that an object defined as a "cat" is composed of limbs, body, face, tail, whiskers, fur, claws, eyes, nose, ears, mouth, etc. Again, acknowledging that there are currently no known prior systems that teach constructivities of models of objects (and subassemblies) from basic parts, such an embodiment teaches or learns a model that a subassembly, such as the face, of an object defined as a cat is composed of parts including eyes, ears, nose, mouth, whiskers, etc.

[0034] The DARPA XAI model operates at a symbolic level insofar as objects and object parts are all represented by symbols. Referring to the cat example, such a system would have symbols corresponding to the cat object and all the cat parts. The disclosed embodiments described herein extend and expand such capabilities by allowing the user to control the symbolic model in the sense that the user can define the list of parts of a given object. For example, the system allows the user to select to only recognize the cat's legs, face, body and tail, and nothing else. As mentioned above, there are no known prior systems that allow the user to flexibly define the symbolic model when configuring a specifically desired implementation as required for the user's particular purpose.

[0035] The DARPA XAI model provides protection from adversarial attacks by making predictions of objects conditional on independent verification of parts. The disclosed embodiments described herein extend and expand such capabilities by allowing users to define the parts to be verified. In general, enhanced verification of additional parts provides more protection from adversarial attacks. As previously mentioned, no known prior system allows end users to define protection levels in the manner enabled by the described embodiments.

[0036] According to an exemplary embodiment, symbolic AI models are integrated into a generative system that rapidly classifies objects in images.

[0037] Many existing systems rely on visualization, require human validation, and are not easily integrated into no-human-in-the-loop generative systems. For these reasons, there are several advantages to embodiments of the present invention compared to the known state of the art, including:

[0038] There are no other currently available systems on the market that can build symbolic AI models of the kind specified by DARPA. Embodiments of the present invention can build such models.

[0039] Currently, to protect against adversarial attacks, deep learning models must be specifically trained to recognize adversarial attacks. However, even this does not guarantee protection against such attacks. Embodiments of the present invention may provide a significantly higher level of protection from adversarial attacks than existing systems for computer vision, without the need for adversarial training.

[0040] Experiments show that higher prediction accuracy is achieved compared to existing methods involving symbolic AI systems where prediction is based on site recognition.

[0041] Symbolic AI models can be easily integrated into generative systems to quickly classify objects in images. Many existing systems rely on visualization, require human validation, and cannot be easily integrated into no-human-in-the-loop generative systems.

[0042] Embodiments of the present invention that allow for the creation of user-defined symbolic models provide transparency and trust in the model from the user's perspective. This transparency and trust in black box models is highly desirable in the field of computer vision.

[0043] An embodiment of the present invention includes a method for decoding a custom convolutional neural network (CNN) to recognize parts of an object. A separate multi-target model (e.g., MLP or equivalent), which provides information about the configuration of the object from the parts and part connectivity, actually learns to decode the CNN activation function. This second model also embodies symbolic information for explainable AI. Experiments have demonstrated that encoding of the parts of an object exists at multiple levels of the CNN, and the part information can be easily extracted to explain the reasoning behind the classification decision. The approach of the embodiment of the present invention is similar to the approach of teaching a human about the parts of an object. The embodiment provides the second model with information about the configuration of the object by its parts, including information of the subassembly and the connectivity between the parts. The embodiment provides the configuration information by listing the parts, but does not provide any location information. For example, for a cat's head, the list may include the eyes, nose, ears, and mouth. The embodiment lists only the parts of interest. The embodiment may implement the overall approach in various ways. The following description presents a specific embodiment and illustrates the technique using several ImageNet-trained CNN models, including models including Xception, Visual Geometry Group ("VGG"), and ResNet. Conventional wisdom holds that accuracy must be sacrificed for explainability. However, experimental results show that explainability can substantially improve the accuracy of a number of CNN models. Furthermore, since object parts, not just objects, are predicted in the second model, it is highly likely that adversarial training will not be necessary. The second model is formulated as a multi-target classification problem.

[0044] An embodiment of the invention uses a multi-target model. In one embodiment, the multi-target model is a multi-layer perceptron (MLP), which is a class of feed-forward artificial neural network (ANN). Other embodiments may use an equivalent multi-target model. The term MLP is used vaguely, sometimes loosely, to mean any feed-forward ANN, and sometimes strictly refers to a network composed of multiple layers of perceptrons (with threshold activation functions). Multi-layer perceptrons are sometimes colloquially referred to as "vanilla" neural networks, especially when they have a single hidden layer.

[0045] MLPs consist of at least three layers of nodes: an input layer, a hidden layer, and an output layer. Except for the input nodes, each node is a neuron that uses a nonlinear activation function. MLPs use a supervised learning technique called backpropagation for training. MLPs' multiple layers and nonlinear activation functions distinguish them from linear perceptrons. MLPs can distinguish data that is not linearly separable.

[0046] Any number of layers can be reduced to a two-layer input-output model if the multi-layer perceptron has linear activation functions in every neuron, such as a linear function that maps weighted inputs to the output of each neuron. In MLP, some neurons use non-linear activation functions developed to model the action potential or firing frequency of biological neurons. Learning occurs in the perceptron by modifying the connection weights after processing each data item based on the amount of error in the output compared to the expected result. This is an example of supervised learning, performed through backpropagation, a generalization of the least mean squares algorithm in linear perceptrons.

[0047] FIG. 1 shows a schematic diagram of an example architecture of a DARPA-compliant Explainable AI (XAI) model, which has explanatory improvements implemented for informed users.

[0048] As shown, two approaches are presented. First, the exemplary architecture 100 shows a model being trained on training data 105 that has been processed through a black box learning process 110, which results in a learning function in block 120. This trained model can then receive input images 115 for processing, in response to which a predicted output 125 is represented from the system to a user 130 who has a particular task to solve. Because this approach is non-transparent, no explanation is provided, resulting in frustration for the user. The user may ask questions such as "Why did they do this?" or "Why not something else?" or "When does it succeed?" or "When does it fail?" or "When can it be trusted?" or "How do I fix the error?".

[0049] Conversely, the improved model described herein is presented below with the same training data 105 provided to a transparent learning process 160, resulting in an explainable model 165 that is capable of receiving the same input image 115 from the previous example. However, unlike the previous model, there is now an explanation interface 170, providing transparent predictions and explanations to an informed user 175 as they attempt to solve a particular task. As shown, the explanation interface 170 provides the user with information such as "this is a cat" and "it has fur, whiskers and claws" and "it has these characteristics" along with a graphical depiction of a cat's ears.

[0050] The hierarchical organization of images allows concept generation and extraction from CNNs. Understanding image content has always been a target of computer vision. In an image parsing graph, a tree-like organization is used to decompose a scene from scene labels, showing object parts and primitive shapes, as well as their functional and spatial relationships. The GLOM model seeks an answer to the question: "How can a neural network with a fixed architecture parsing decompose an image into a part-whole hierarchy with different configurations for each image?" The term "GLOM" is derived from the slang term "glom together" as a representation method, which improves image processing through the use of transformers, neural fields, contrastive representation learning, distillation, and capsules that allow static neural nets to represent dynamic parsing trees.

[0051] The GLOM model generalizes the concept of a capsule and dedicates a group of neurons for a particular type of feature within a particular region of the image to the concept of stacked autoencoders for each small patch of the image. These autoencoders then address multiple levels of representation - from a person's nostrils to their nose to their face - either in their entirety or completely through the "whole" person.

[0052] Introduction to the illustrative embodiments

[0053] Some exemplary embodiments provide a specially configured computer-implemented method for identifying regions of an object from activation functions of a fully connected layer of a convolutional neural network (CNN). However, identification of regions is also possible from activation functions of other layers of the CNN. The embodiments involve teaching a separate model (a multi-target model, e.g., MLP) how to decode the activation functions by providing the separate model with information about the configuration of the object from its regions and region connectivity.

[0054] Identification of object parts generates information at the symbolic level of the kind that DARPA envisions for Explainable AI (XAI), as shown in Figure 1. This particular form conditions object recognition on identification of object parts. For example, this form requires that to predict that an object is a cat, the system must also recognize some of the specific features of a cat, such as its fur, whiskers, and claws. Making object predictions conditional on recognition of object parts or features provides additional validation to the object, making the predictions robust and reliable. For example, for such an image recognition system, a school bus with a small perturbation of a few pixels would not be predicted as an ostrich because the parts of an ostrich (e.g., long legs, long neck, small head) are not present in the image. Thus, requiring identification of several parts of an object provides a fairly high level of protection in an adversarial environment. Such a system cannot be easily fooled. Also, such a system may further eliminate the need for adversarial training with GANs and other mechanisms due to its inherent robustness.

[0055] There are several different approaches to the part-whole identification problem. For example, the GLOM approach builds parse trees in the network to show the part-whole hierarchical structure. Conversely, the described embodiment does not build or require such parse trees.

[0056] Fine-grained object recognition attempts to distinguish between subclasses of objects of a general class, such as different species, such as birds or dogs. Many fine-grained object recognition methods identify distinctive parts of a subclass of objects in various ways. Some of these methods are described below as related concepts. However, the method of identifying parts of an object according to embodiments of the present invention is different from all these methods. In particular, the described embodiments provide a learning system with information about the composition of the object from its parts, and about its parts from its component parts. For example, for an image of a cat, the embodiments list the visible parts of the cat, such as the face, legs, tail, etc. The embodiments do not show the system as such where these parts are bounding boxes or similar mechanisms. The described embodiments list the visible parts of an object in an image. For example, the described embodiments may show the system an image of a cat's face and list the visible parts--eyes, ears, nose, and mouth. Thus, the described embodiments need only list the parts of interest. Thus, if the nose and mouth are not of interest in a particular problem or task, the nose and mouth are not listed. Certain described embodiments also annotate the parts.

[0057] To reiterate, the embodiments of the present invention do not provide any indication of where the parts are in the image. Thus, the described embodiments provide configuration information, but not location information. Of course, the embodiments of the present invention show all separate images of the parts of interest - eyes, ears, nose, mouth, limbs, tail, etc. - so that the recognition system knows what these parts look like. However, the system learns the spatial relationships (also known as "connectivity") between these parts from the configuration information provided. Thus, it is this ability to provide configuration information that is significantly different from known prior techniques for recognizing parts of objects. The described embodiments teach a model (e.g., MLP), configuration, and spatial relationships of parts. Thus, the method of teaching the system about the parts of an object is different from any known prior method or system that solves the same or similar problem.

[0058] For the problem of providing names or labels (annotations) of parts, embodiments of the invention rely on an understanding of human learning. It is perhaps reasonable to claim that both dogs and humans recognize various features of the human body, such as legs, hands, and faces. The only difference is that humans have names for these parts, and dogs do not. Of course, humans do not inherit the names of parts from their human parents. In other words, humans are not born with the names of objects and parts, but must be taught them. And this teaching occurs only after the visual system has learned to recognize these parts. Embodiments of the invention follow the same two-stage approach to teaching part names: first, the system learns to visually recognize parts without being given the names of the parts, and then the names of the parts are taught. Embodiments of the invention provide a set of images with the names of the parts.

[0059] Higher levels of abstraction and single-cell encoding in the brain are often found outside the visual cortex. Neurophysiological experiments have demonstrated that the brain makes extensive use of localized single-cell representations, especially for invariant recognition of highly abstract concepts and multimodal objects. Prior techniques have used single-cell recordings from the visual system to find simple and complex cells, line orientation and motion detection cells, essentially confirming single-cell abstraction at the lowest levels of visual organization. However, other researchers have reported finding more complex single-cell abstractions at higher levels of processing that encode modality-invariant recognition of people (e.g., Jennifer Aniston) and objects (e.g., the Sydney Opera House). One estimate is that 40% of medial temporal lobe (MTL) cells are tuned to such explicit representations. Neuroscience experts claim that experimental evidence indicates that the PFC plays a crucial role in category formation and generalization. Experts claim that prefrontal neurons abstract commonalities across a variety of stimuli. Second, they categorize prefrontal neurons based on their common meanings by ignoring the physical properties of prefrontal neurons.

[0060] These neurophysiological discoveries mean that the brain generates many models outside the visual cortex to generate various kinds of abstractions. An embodiment of the present invention exploits these biological cues by (1) generating a single neuron (node) abstraction for object parts, since object parts are themselves abstractions, and (2) generating a separate model (MLP) outside the CNN to recognize object parts. Of course, this is nothing new for CNNs, since CNN models actually use a single output node for the object class. An embodiment of the present invention just extends this single-node representation scheme to object parts and adds these nodes to the output layer of the MLP.

[0061] The embodiments of the present invention train a CNN model to recognize various objects. The CNN model so trained does not provide any information about the configuration of the object from its parts. The embodiments of the present invention provide only information about the configuration of the object from its parts and the configuration of the parts (subassemblies) from other component parts to a subsequent MLP model, which receives its input from the fully connected layer of the CNN. The separate MLP model simply decodes the CNN activation function to recognize the object and parts and understand the spatial relationship between the parts. However, the described embodiments do not provide location information of any parts, as is the case for bounding boxes common to known prior techniques. Instead, the described embodiments simply provide a list of parts that make up an assembly in an image, such as a face.

[0062] However, it should be noted that embodiments do not need to build a separate model (MLP or any other classification model) to recognize parts: the MLP model is also tightly coupled with the CNN model, and the integrated model can recognize both objects and parts.

[0063] The following sections provide further context on explainable AI in general, followed by explainable AI for computer vision and fine-grained object recognition. The sections that follow provide an intuitive understanding of embodiments of the invention. The sections that follow provide further details on the algorithms used to implement specific embodiments of the invention, followed by a discussion of the experimental results and concluding remarks.

[0064] Explainable AI (XAI)

[0065] Explainability of an AI system takes many different forms depending on the use of the AI ​​system. In one such form, an object or concept is explained by its properties, which may be other abstractions (or sub-concepts). For example, a cat (a high level abstraction) may be described using some of its main features (which are abstract sub-concepts), such as its legs, tail, head, eyes, ears, nose, mouth and whiskers. This form of explainable AI relates directly to symbolic AI, where symbols represent abstractions and sub-concepts. An embodiment of the present invention presents a way in which convolutional neural networks may be decoded to extract this kind of abstract symbolic information.

[0066] From another perspective, explainable AI methods for machine learning can be categorized as (1) transparent by design and (2) post-mortem explanation. Transparency by design uses model construction that starts with an interpretable model such as a decision tree. Post-mortem explanation methods extract information from an already trained black-box model and make the performance of the black-box model closely approximate that of the new interpretable model. The benefit of this approach is that it does not affect the performance of the black-box model. Post-mortem methods are model-agnostic since they mainly address the input and output of the black-box model. From this perspective, an embodiment of the present invention adopts the post-mortem method.

[0067] The "COGLE" (Common Ground Learning and Explanation) system describes the learning capabilities of an XAI system that controls a simulation of an unmanned aerial system. COGLE uses a cognitive layer that bridges human-usable symbolic representations to the abstract concepts, configurations, and generalization patterns of the underlying model. The idea of ​​"common ground" here means establishing a common terminology for use in explanations and understanding the meaning of the common terms. The explanation of embodiments of the present invention also uses this idea of ​​common terminology.

[0068] A range of approaches to explainable AI for deep learning

[0069] Known prior methods are available for visualizing and understanding the representation (encoding) inside a CNN. For example, there is a class of methods that primarily synthesize images that maximally activate units or filters. Ascending convolution methods are also known that provide another kind of visualization by inverting CNN feature maps into images. There are also methods that go beyond visualization and try to understand the semantic meaning of the features encoded by the filters.

[0070] Furthermore, there are methods that perform image-level analysis for explanation. For example, the LIME method extracts image regions that are highly sensitive to the network's predictions and provides explanations of individual predictions by showing the associated image patches. General confidence in the model is based on testing a large number of such individual predictions. There is also a class of methods that identify pixels in the input image that are important for the prediction - e.g., sensitivity analysis and layer-wise relevance propagation (LRP).

[0071] Posterior methods include methods that learn a semantic graph that represents the CNN model. These methods generate an interpretable CNN by making each convolutional filter a node in the graph, and then have each node represent an object part. A related method learns a new interpretable model from the CNN through an active question-answering (ActiveQA) mechanism. There are also methods that generate textual explanations of predictions. For example, such a method might say "This is a Laysan albatross because it has a large wingspan, a hooked yellow beak, and a white underside." These methods use stacked LSTMs on top of the CNN model to generate textual explanations of predictions.

[0072] Another approach is to generate visual and textual information together using attention masks that localize salient regions in providing text alignment. Such approaches use visual question answering datasets to train such models. A caption-guided visual saliency map method has also been proposed, which uses an LSTM-based encoder-decoder that learns the relationship between pixels and caption words to generate a spatio-temporal heatmap for predicted captions. One model provides explanations by generating some high-level concepts from a deep network, and attaches a separate explanation network to a specific layer (which may be any layer) in the deep network, reducing the network to a few concepts. These concepts (features) may not be human understandable initially, but a domain expert may attach interpretable explanations to these features. Research has found that object detectors have emerged from training CNNs that perform scene classification, thus indicating that the same network can perform scene recognition and object localization, even though the object detectors are not explicitly taught object concepts.

[0073] Part Identification for Fine-Grained Object Recognition

[0074] There is a survey of deep learning-based methods for fine-grained object recognition. Most part-based methods focus on identifying subtle differences in similar object parts, such as the color or shape of a beak, a subcategory of birds. For example, one proposal learns a set of special features of parts that distinguish between fine-grained classes. Another proposal trains a part-based RCNN to detect both objects and distinctive parts. Part-based RCNN uses bounding boxes on images to localize both objects and distinctive parts. During testing, all object and part proposals (bounding boxes) are scored and the highest score is selected. Part-based RCNN trains a separate classifier for pose-normalized categorization based on features extracted from the localized parts. One part-stacked CNN approach uses one CNN to localize multiple object parts and a two-stream classification network to encode both object-level and part-level cues. They annotate the center of each object feature as a keypoint and train a fully convolutional network, called a localization network, on these keypoints to locate the object features. These feature locations are then fed into a final classification network. One proposal, deep LAC, involves localization, alignment, and classification of features in a single deep network. They train a localization network that recognizes the features and generates bounding boxes for the features in test images.

[0075] Embodiments of the present invention do not use bounding boxes or keypoints to localize objects or parts. In fact, embodiments of the present invention do not provide any location information to any of the models they train. Embodiments of the present invention show images of parts, albeit as separate images, as described in the next section. Embodiments of the present invention also provide a part (or part-subpart) configuration list of an object, but do not provide location information. Furthermore, embodiments of the present invention generally identify all parts of an object, not just distinct parts. Identification of all parts of an object provides additional protection against adversarial attacks.

[0076] In common with part-based RCNN, embodiments of the present invention identify parts as separate object categories in a second MLP model.

[0077] Algorithm Overview

[0078] We provide a general overview of embodiments of the present invention and how such embodiments may be algorithmically implemented. We illustrate our approach using two problems: (1) classifying images of four distinct classes - cars, motorcycles, cats, and birds (an easy problem), and (2) classifying images of two fine-grained classes - huskies and wolves (a harder problem).

[0079] FIG. 2 illustrates a technique 200 according to an embodiment of the invention for classifying images into four distinct classes.

[0080] In particular, from the top, row 1 shows an image of a cat 205 , row 2 shows an image of a bird 206 , row 3 shows an image of a car 207 , and row 4 shows an image of a motorcycle 208 .

[0081] FIG. 3 illustrates a technique 300 for classifying images into two fine-grained classes according to an embodiment of the invention.

[0082] In particular, from the top, row 1 shows an image of a husky 305 and row 2 shows an image of a wolf 306 .

[0083] As shown in FIG. 2 and FIG. 3, there is a sample image of the first problem shown in FIG. 2, and a sample image of the second problem shown in FIG.

[0084] Using CNNs for Object Classification

[0085] In the first step, the embodiment of the present invention trains a CNN to classify target objects. Here, the embodiment of the present invention may train the CNN from scratch or use transfer learning. In experiments, the embodiment of the present invention used transfer learning using some of the CNNs trained on ImageNet, such as ResNet, Xception and VGG models. For transfer learning, the embodiment of the present invention freezes the weights of the convolutional layer of the CNN trained on ImageNet, then adds one flattened fully connected (FC) layer, and then adds an output layer such as the output layer of FIG. 4, but with only one FC layer. The embodiment of the present invention then trains the weights of the fully connected layer for a new classification task.

[0086] FIG. 4 illustrates transfer learning 400 for a new classification task that involves only training the weights of an additional fully connected layer of a CNN, in accordance with an embodiment of the present invention.

[0087] In particular, a CNN network architecture 405 is shown that includes a frozen feature learning layer. Within the CNN network architecture 405, there is both a feature learner 435 and a classifier 440. Within the feature learner 435, there is shown an input image 410, a convolution+RELU 415, a max pooling 420, a convolution+RELU 425, and a max pooling 430. Within the classifier 440, there is shown a fully connected layer 445 that completes the processing for the CNN network architecture 405.

[0088] As presented herein, for a new classification task, the method trains only the weights of the additional fully connected layers of the CNN.

[0089] More specifically, in the illustrated architecture, a CNN is first trained to classify objects. Here, the CNN is trained from scratch or via transfer learning. In some experiments, several CNN models trained on ImageNet, such as the Xception and VGG models, were utilized for transfer learning. For transfer learning, the weights of the convolutional layer are generally frozen, then a flattening layer is added, then a fully connected (FC) layer is added, and finally an output layer is added, such as the example shown in FIG. 5, except that only one FC layer is generally added. Then, the weights of the fully connected layer are trained for a new classification task.

[0090] Using MLP for multi-target classification problems

[0091] An embodiment of the present invention does not train a CNN to explicitly recognize object parts. An embodiment of the present invention trains a CNN with a different model, and an embodiment of the present invention trains a multi-layer perceptron (MLP) to recognize both the object and the object parts, as shown in FIG. 5. For example, if the object is a cat, an embodiment of the present invention may recognize parts of the part such as the legs, tail, face or head and torso. For a car, an embodiment of the present invention may recognize such parts as the door, tires, radiator grill and roof. Note that all object parts may not be present for every object in a class (e.g., roof is a part of most cars, but some Jeeps do not have a roof) or may not be visible in the image. In general, an embodiment of the present invention wants to verify all visible parts as part of the object verification process. For example, an embodiment of the present invention would not verify that an object is a cat if it could not verify some of the visible parts of the cat.

[0092] FIG. 5 illustrates the training of a separate multi-target MLP 500 according to an embodiment of the present invention, where the input comes from the activation functions of the fully connected layer of a CNN, and the output nodes of the MLP correspond to both objects and object parts.

[0093] As shown herein, the process of MLP 500 involves training a separate multi-target MLP, from which MLP input 505 originates, using the activation function of a fully connected layer of a CNN. The output nodes 550 of MLP 500 correspond to both objects (e.g., whole cat or whole dog) and respective parts of the object (e.g., body, legs, head or tail of a cat or dog). More specifically, the output nodes 550 of multi-label MLP 500 correspond to objects and object parts and are shown in the form of emitting symbols. The input to this MLP (e.g., MLP input 505) comes from the activation function of a fully connected layer of a CNN model trained to recognize objects, but not parts.

[0094] Some a posteriori methods learn a semantic graph that represents the CNN model. Such methods generate an interpretable CNN by making each convolutional filter a node in a graph, and then have each node represent an object part. Other methods learn a new interpretable model from the CNN through an active question-answering (ActiveQA) mechanism. For example, some models provide explanations by generating some high-level concepts from a deep network, as described above, and then add a separate explanation network to a specific layer.

[0095] The described embodiment recognizes body parts by setting up an MLP for a multi-target classification problem, as shown in Figure 5. In the output layer of the MLP, each object class and object part has a separate output node. Thus, parts are also object classes in their own right. In this multi-target framework, if the input is, for example, an image of a whole cat, then all output nodes of the MLP that correspond to the cat object, including the cat parts (head, legs, body, and tail), should be activated.

[0096] FIG. 6A shows the training of a separate multi-label MLP 600 according to the described embodiment, where the input is the activation function of the fully connected layer of the CNN.

[0097] Specifically shown in Figure 6A is a multi-target MLP 600 architecture with an input image 605 leading to convolutional and pooling layers 610, then proceeding to a fully connected (FC) layer of either 256 or 512 nodes as shown in element 615, and finally to an MLP 620 with both an MLP input layer 555 and an MLP output layer 560. The multi-target MLP 600 trains a separate multi-target MLP, where the input is the activation function of the fully connected layer of the CNN. The output nodes of the MLP correspond to both the object and the object's parts.

[0098] As shown in FIG. 6A, the output nodes of the MLP correspond to both objects and parts of the objects.

[0099] FIG. 6B shows training a multi-label CNN 601 to recognize objects and parts 625 as well as learn about configurations and connectivity 630 according to a described embodiment.

[0100] 6C shows training of a single-label CNN 698 that recognizes both objects and parts 645 from parts and part connectivity, but not object configuration, according to a described embodiment. Also shown is training of a separate multi-label MLP, where the input is the activation function of the fully connected layer of the CNN. As shown in FIG. 6C, the MLP learns object configuration from parts and part connectivity.

[0101] In experiments, embodiments of the present invention generally added only one fully connected layer of size 512 or 256 to the CNN, as shown in Figure 6. The Experimental Results section below shows the results of using the activation functions from these fully connected (FC) layers as inputs to the MLP. Figure 6 also shows the general flow of the process of training an MLP: (1) present training images to the training CNN, (2) read the activation functions of the fully connected (FC) layer, (3) use these activation functions as inputs to the MLP, (4) set appropriate multi-target outputs for this training image, and (5) adjust the weights of the MLP using one of the weight adjustment methods.

[0102] For example, assume that an embodiment of the present invention uses the activation function of a 512-node fully connected (FC) layer as input to the MLP. Further assume that the training image is a cat's face and that we are interested in identifying the following features: eyes, ears, and mouth. In this case, the target values ​​of the MLP output nodes corresponding to the cat's face, eyes, ears, and mouth are set to 1. The overall training process for this image is as follows: (1) input the cat's face image into the CNN, (2) load the activation function of the 512-node fully connected (FC) layer, (3) use these activation functions as input to the MLP, (4) set the target outputs for the face, eyes, ears, and mouth nodes to 1, and (5) adjust the weights of the MLP according to the weight adjustment method.

[0103] 7 shows sample images of different parts of a cat, according to the described embodiment, specifically showing the cat's face 705 in the first row, the cat's legs 710 in the second row, the cat's body 715 in the third row, and the cat's tail 720 in the fourth row.

[0104] 8 shows sample images of various parts of a bird, according to the described embodiment, specifically, in the first row, the bird's body 805, in the second row, the bird's head 810, in the third row, the bird's tail 815, and in the fourth row, the bird's feathers 820.

[0105] 9 shows sample images of various parts of a car according to the described embodiment, in particular the rear of the car (e.g., the rear part of the car) 905 in the first row, the car doors 910 in the second row, the car radiator (e.g., the grill) 915 in the third row, the rear wheels of the car 920 in the fourth row, and the front of the car (e.g., the front part of the car) 925 in the fifth row.

[0106] 10 shows sample images of various parts of a motorcycle according to the described embodiment, in particular the rear wheel of the motorcycle 1005 in the first row, the front wheel of the motorcycle 1010 in the second row, the handlebars of the motorcycle 1015 in the third row, the seat of the motorcycle 1020 in the fourth row, the front of the motorcycle (e.g., the front part of the motorcycle) 1025 in the fifth row, and the rear of the motorcycle (e.g., the rear part of the motorcycle) 1030 in the sixth row.

[0107] Thus, Figures 7, 8, 9 and 10 provide illustrative sample images of various parts of a cat (head, legs, body and tail), a bird (body, head, tail and wings), a car (rear of the car, doors, radiator grill, rear wheel, front of the car) and a motorcycle (rear wheel, front wheel, handlebars, seat, front of the bike, and rear of the bike) that embodiments of the present invention use in training the MLP for the first problem.

[0108] For the second problem, which was the problem of recognizing huskies and wolves, an embodiment of the present invention added two more body parts - eyes and ears - to the list of body parts of the similar animal, cat, so that huskies and wolves have six body parts: face or head, legs, body, tail, eyes and ears.

[0109] FIG. 11 shows a sample image of a husky's eyes 1105 and a husky's ears 1110 according to the described embodiment.

[0110] FIG. 12 shows sample images of a wolf's eyes 1205 and wolf's ears 1210 according to the described embodiment.

[0111] Note that embodiments of the present invention annotate parts by tagging them with the name of the corresponding object. Thus, "cat head" and "dog head" as well as "husky ears" and "wolf ears". In general, embodiments of the present invention have MLP find the differences between similar parts across an object. Embodiments of the present invention used Adobe Photoshop to generate images of a number of parts. Some, such as "bike front" and "car back", were simply sliced ​​out of the overall image using Python code. Embodiments of the present invention are currently investigating ways to automate this task.

[0112] Teaching the structure of an object from its parts and part connectivity, and recognizing the parts of an object

[0113] To verify the existence of component parts, embodiments of the invention teach the MLP what these parts are and how they are connected to each other. In other words, embodiments of the invention teach the configuration of the object from its component parts and their connectivity. This teaching is at two levels. At the lowest level, to recognize individual elementary parts, embodiments of the invention simply show the MLP separate images of these parts, such as an image of a car door or a cat's eyes. At the next level, to teach the MLP how to assemble the elementary parts to generate a sub-assembly (e.g., just the cat's face) or an entire object (e.g., the entire cat), embodiments of the invention simply show the MLP images of the sub-assembly and the entire object, and list the parts contained within the images. Given the parts list and corresponding images for an assembly or sub-assembly, the MLP learns the configuration of the object and the sub-assembly, as well as the connectivity of the parts. Embodiments of the invention provide this parts list to the MLP in the form of a multi-target output for the image, as previously described. For example, for a cat face image, if the parts of interest are eyes, ears, nose and mouth, the embodiment of the present invention sets the target values ​​of the output nodes of these parts to 1 and the rest to 0. For a whole cat image, the embodiment of the present invention lists all parts such as face, legs, tail, body, ears, nose and mouth by setting the target values ​​of the corresponding output nodes to 1 and the rest to 0. Therefore, appropriately setting the target output values ​​of the output nodes in the multi-target MLP model is one way to list the parts of an assembly or subassembly. Of course, it is only necessary to list the parts of interest. If the verification that there is a tail is not of interest, the tail part does not need to be listed. However, the longer the list of parts, the better the verification for the object.

[0114] Explainable AI through configuration

[0115] According to the embodiment, the user is both the architect and builder of the explainable AI (XAI) model, which depends on the focal parts of the object, and validation is important. For example, in experiments with cat and dog images (results in section 5), the embodiment of the invention used only four features: body, face or head, tail and legs. In the case of huskies and wolves (results in section 5), the embodiment of the invention used six features: body, face or head, tail, legs, eyes and ears. Validation of more object features or parts can result in higher accuracy.

[0116] The output layer of the MLP essentially contains the base of a symbolic model. An activation function of an output node that exceeds a certain threshold indicates the presence of a corresponding part (or object). This activation function sets the value of the corresponding part symbol (e.g., the symbol representing the cat's eyes) to TRUE, indicating recognition of this part. Various symbolic models can be constructed for object recognition based on the symbolic output of the MLP output layer. At one extreme, to recognize an object, one can insist on the presence of all parts of the object in the image. Or, this condition can be relaxed to deal with situations where the object is only partially visible in the image. In the case of partially visible objects, a decision must be made based on the evidence at hand. In another variant, more emphasis can be placed on the verification of certain parts. For example, to predict that an object is a cat, one can insist on the visibility of the head or face to verify that the object is a cat. In this case, a prediction based on the recognition of other parts of the cat may not be acceptable.

[0117] An embodiment of the present invention presents here one symbolic model based on the counting of validated sites. i,k Assume that k=1...NPi,i=1...NOB denotes the kth part of the i-th object class, NPi denotes the total number of parts in the i-th object class, and NOB denotes the total number of object classes. If the existence of an object part is verified, P i,k Assume k=1, otherwise 0. PV i denotes the total number of verified sites of the i-th object class, and PVi min Let denote the minimum number of verifications of parts required to classify an object as an object of the i-th class. The general form of this symbolic model based on the number of verifications (recognitions) of object parts according to equations (1) and (2) is as follows: Formula (1) PV i ≧PV i min the i-th object class is a candidate class for recognition if Formula (2) PV i =Σ_ (k=1) ^NPi (P i,k is seen and recognized).

[0118] The predicted class is the maximum PV according to the following formula (3): i is the class provided, which satisfies the condition shown in formula (1): Formula (3) Predicted object class PO=argmax i (PV i ).

[0119] If validation of certain sites is important for the prediction, then equation (2) will only count these sites. Again, note that site counting is at the symbolic level.

[0120] Algorithm: For simplicity of notation, an embodiment of the present invention i,k Suppose that M denotes both elementary parts of an object (e.g., eyes or ears) and more complex parts of an object that are assemblies of elementary parts (e.g., the face of a husky, consisting of eyes, ears, nose, mouth, etc.). i Let,denote the set of original training images of the,i,th class, and,M,the total set of training images.

[0121] Thus, M consists of images of objects of the type shown in Figs. i,k ,k=1...NP iLet C denote the set of object part images available for the kth part of the ith object class, and let MP denote the total set of object part images. Thus, MP is composed of object part images of the types shown in Figures 7 to 12. An embodiment of the invention generates these MP object part images from M original images. Let MT = {M ∪ MP} be the total set of images. An embodiment of the invention uses the original M images to train and test the CNN and MT images, and to train and test the MLP.

[0122] FC j Assume that j denotes the jth fully connected (FC) layer in the CNN, and J denotes the total number of FC layers. Currently, embodiments of the present invention use the activation function of one of the FC layers as the input to the MLP, but multiple FC layers may be used. Assume that embodiments of the present invention select the jth FC layer to provide the input to the MLP. In this version of the algorithm, embodiments of the present invention train the MLP to decode the activation function of the jth FC layer to find the object part.

[0123] T i Let T denote the target output vector of the i-th object class for the multi-target MLP. iis a 0-1 vector indicating the presence or absence of an object and object parts in the image. For example, for a cat defined by parts limbs, body, tail, and head, this vector is of size 5. And the output vector for a cat may be defined as [cat object, limbs, head, tail, body] as shown in FIG. 5. For an image of a whole cat with all parts visible, this target output vector is [1,1,1,1,1]. If the cat's tail is not visible, this vector is [1,1,1,0,1]. An embodiment of the present invention used the following parts for a husky: husky_head, husky_tail, husky_body, husky_limbs, husky_eyes, husky_ears. Thus, the output vector is of size 7 for a husky and may be defined as [husky_object, husky_head, husky_tail, husky_body, husky_limbs, husky_eyes, husky_ears]. For an image of a husky's head, this vector is [0,1,0,0,0,1,1]. Note that embodiments of the present invention list only visible parts. Also, because the only visible part is the husky's head, embodiments of the present invention set the husky's object target value at the first position to 0. In general, T i The vector is the object at the first location, followed by the list of parts. As shown in FIG. 5, these object class output vectors T i are combined to generate a multi-target output vector for the MLP. For the cat and dog problems in Figure 5, the multi-target output vector is of size 10. For a whole cat image, it is [1,1,1,1,1,0,0,0,0,0]. For a whole dog image, for example, it is [0,0,0,0,0,1,1,1,1,1].

[0124] IM k Let TR be the k-th image in the total image set MT consisting of M object images and MP part images. k Assume that k is the corresponding multi-target output vector for the kth image.

[0125] To train the MLP on both the original M image and the MP region image, for each image IMk is first input to the training CNN, and the activation function of the jth FC layer specified is recorded. Then, the activation function of the jth Fc layer becomes the input to the MLP, and the TR k is the corresponding multi-target output variable.

[0126] The general form of the algorithm is as follows:

[0127] Step 1:

[0128] M images of C object classes are used to train and test a convolutional neural network (CNN) with a set of fully connected (FC) layers, where the CNN can be trained from scratch or using transfer learning to add an FC layer.

[0129] Step 2:

[0130] Train a multi-target MLP using a subset of MT images. For each training image IM k In contrast, Input image IMk to the training CNN, Record the activation function at the jth FC layer specified, The activation function of the jth FC layer is input to the MLP, Image IM k TR as a multi-target output vector for k Set The weights of the MLP are adjusted using a suitable weight adjustment method.

[0131] Testing Mechanism and Results:

[0132] Experimental setup: The present invention tested an embodiment of the present approach to XAI on three problems involving images from the following classes of objects: (1) cars, motorcycles, cats and birds, (2) huskies and wolves, and (3) cats and dogs. The first problem has images from four separate classes and is a little on the easier side. The other two problems are somewhat similar to fine-grained image classification problems, with closer objects. Table 1 shows the number of images used to train and test the CNN and MLP. The present invention used several augmented images to train both the CNN and the MLP. The present invention trained and tested the multi-target (multi-label) MLP using only images of object parts.

[0133] 13 shows Table 1 according to the described embodiment, which shows what is learned in the CNN+MLP architecture in element 1300. Multi-label MLP learns the configuration and connectivity between objects and parts.

[0134] 14 shows Table 2, indicating in element 1400 the number of images (original images plus enhanced images) used to train and test the CNN and MLP. An embodiment of the present invention trained and tested the multi-target MLP using only images of object parts.

[0135] An embodiment of the present invention used the Keras software library for transfer learning with an ImageNet-trained CNN and building a separate MLP model, and used Google Colab to build and run the models.

[0136] For transfer learning, embodiments of the present invention used ResNet, Xception, and VGG models. For transfer learning, as shown in Figure 4 above, embodiments of the present invention froze the weights of the convolutional layer, then added a fully connected layer after the flattening layer, and then added an output layer. Then, embodiments of the present invention trained the weights of the fully connected layer for a new classification task.

[0137] The present embodiment added only one fully connected (FC) layer of size either 512 or 256 between the flattening layer and the output layer, along with dropout and batch normalization. The output layer had a softmax activation function along with a ReLu activation function for the FC layer. The present embodiment tested an approach with two different fully connected (FC) layers (512 and 256) and showed that the encoding of the object's parts is present in the FC layers of different sizes and the part-based MLP can decode them properly. The present embodiment trained the CNN for 250 epochs using the RMSprop optimizer with "categorical cross entropy" as the loss function. The present embodiment also generated a separate test set and used this test set as the validation set. The present embodiment used 20% of the total dataset for both CNN and MLP testing.

[0138] The MLP had no hidden layers. The MLP had inputs that were directly connected to a multi-label (multi-target) output layer. For MLP training, any image, including images of object parts, was first passed through the training CNN and the output of the 512 or 256FC layer was recorded. This recorded output of the 512 or 256FC layer then became the input to the MLP. An embodiment of the present invention used a sigmoid activation function for the MLP output layer. An embodiment of the present invention also trained the MLP for 250 epochs using the "adam" optimizer with "binary cross-entropy" as the loss function since this is a multi-label classification problem.

[0139] An embodiment of the present invention used a slight variation of equation (2) to classify objects with MLP. An embodiment of the present invention simply sums the sigmoid activation functions of the nodes of each object class and the corresponding nodes of the object parts, and then compares the summed outputs of all object classes to classify the image. The object class with the highest sum activation function becomes the predicted object class. In this variation, an embodiment of the present invention uses P i,k = sigmoid activation function value between 0 and 1, P i,k ,k=1...NP i, i=1...NOB denotes the kth part of the ith object class, and NP i Let i denote the total number of sites in the i-th object class and NOB denote the total number of object classes. Here, embodiments of the present invention use the interpretation that the sigmoid output value represents the probability of the presence of a site in that object, according to the following equations (4) and (5):

[0140] Formula (4): PV i =Σ_ (k=1) ^NPi (P i,k = sigmoid output value of the corresponding output node)

[0141] Formula (5): Predicted object class PO=argmax i (PV i ), where PO is the predicted object class.

[0142] Experimental Results for Naming Object Parts: Embodiments of the present invention present here results for three problem embodiments of the present invention solved to test our approach to XAI. Embodiments of the present invention named similar object parts (e.g., cat and dog paws) with different names and let the MLP try to find the distinguishing features that make the object parts different. For example, embodiments of the present invention named husky dog ​​parts as "husky paws", "husky body", "husky head", "husky eyes", etc. Similarly, embodiments of the present invention named wolf dog parts as "wolf paws", "wolf body", "wolf head", "wolf eyes", etc. Presumably, huskies are well groomed by their owners, so husky dog ​​parts should look different from wolf dog parts.

[0143] The embodiment of the present invention used the following object part names for the three problems:

[0144] a) Object classes - cars, motorcycles, cats and birds:

[0145] car part names- rear_car, door_car, radiator_grill_car, roof_car, tire_car, front_car;

[0146] Cat body parts names - cat_head, cat_tail, cat_body, cat_limbs;

[0147] Names of poultry parts - poultry head, poultry tail, poultry body, poultry wings; and

[0148] Motorcycle part names - front_bike, rear_bike, seat_bike, rear_wheel_bike, front_wheel_bike, handlebars_bike.

[0149] b) Object class – cat, dog

[0150] Cat body parts names - cat_head, cat_tail, cat_body, cat_legs, and

[0151] Dog body parts names - dog_head, dog_tail, dog_body, dog_legs.

[0152] c) Object Class – Husky, Wolf

[0153] Husky body parts - husky head, husky tail, husky body, husky legs, husky eyes, husky ears; and

[0154] Wolf body part names - wolf_head, wolf_tail, wolf_body, wolf_legs, wolf_eyes, wolf_ears.

[0155] Classification results using the XAI-MLP model

[0156] FIG. 15 shows Table 3 in element 1500, according to the described embodiment, showing the results of the "cars, motorcycles, cats, birds" classification problem.

[0157] FIG. 16 shows Table 4 in element 1600, according to the described embodiment, showing the results of a "cats vs dogs" classification problem.

[0158] FIG. 17 shows Table 5 in element 1700, according to the described embodiment, showing the results of the "huskies and wolves" classification problem.

[0159] FIG. 18 shows Table 6 in element 1800, which shows the results of comparing the best prediction accuracy of the CNN and XAI-MLP models, according to the described embodiment.

[0160] Tables 2, 3, and 4 show the classification results, respectively. In these tables, columns A and B have the training and testing accuracy of ResNet50, VGG19, and Xception models with two different FC layers, one with 512 nodes and the other with 256 nodes. One model with FC-512 layers and the other model with FC-256 layers are separate models, respectively, and the embodiment of the present invention trained and tested them separately. Therefore, the accuracy may be different. Columns C and D show the training and testing accuracy of the corresponding XAI-MLP model. It should be noted that when the embodiment of the present invention trains the CNN model with FC-256 layers, the XAI-MLP model uses the output of the FC-256 layers as input to the MLP. Also, the embodiment of the present invention sets XAI-MLP as a multi-label (multi-target) classification problem, and the output nodes correspond to both objects and object parts. Thus, for an image of a whole cat, embodiments of the invention set target values ​​for the "cat" object output node and the corresponding part output nodes (cat_head, cat_tail, cat_body, and cat_head) to 1. For an image of a husky's head, embodiments of the invention set target values ​​for the part output nodes "husky_head", "husky_eyes", and "husky_ears" to 1. This is how embodiments of the invention essentially teach the construction of XAI-MLP and connectivity of objects and object parts. Embodiments of the invention do not provide any location information for parts.

[0161] Column E in the table shows the difference in test accuracy between the XAI-MLP and CNN models. In most cases, the XAI-MLP model has higher accuracy. There is an inherent trade-off between prediction accuracy and explainability. Although embodiments of the present invention need to conduct more experiments to make a definitive statement on this issue, from these limited experiments, it appears that embodiments of the present invention obtain an increase in prediction accuracy with the part-based explainable model. Table 5 compares the best test accuracy of the CNN model with the best test accuracy of the XAI-MLP model. For two fine-grained problems (cat vs. dog, husky vs. wolf), the XAI-MLP model provides a significant increase in prediction accuracy.

[0162] FIG. 19 shows the number "5" and a wolf image, according to a described embodiment, where the number "5" has been modified by a fast gradient method at various epsilon values, and the wolf image has also been modified by a fast gradient method at various epsilon values.

[0163] Robustness of Explainable AI against Adversarial Attacks

[0164] Using fast gradient methods, we tested our explainable AI models against adversarial attacks. In particular, we tested our explainable AI models on two problems: (1) distinguishing handwritten digits using the MNIST dataset, and (2) distinguishing huskies from wolves using the dataset from the experiments described above.

[0165] On adversarial image generation - in these tests, we focused on the smallest adversarial attacks (e.g., one-pixel attacks) that humans cannot easily detect. In other words, the altered image may cause the model to make some incorrect predictions, but humans will not see any difference from the original image. Epsilon is a hyperparameter in the fast gradient algorithm that determines the strength of the adversarial attack. Higher epsilon values ​​make the pixels more obscured, often beyond human perception.

[0166] To ensure low visual degradation, we experimented with various epsilon values ​​to determine a value that impacts the accuracy of the base CNN model but still appears roughly the same to humans. The smallest epsilon value for MNIST was found to be around 0.01 to impact the accuracy of the base CNN model.

[0167] Therefore, starting from the minimum, the following epsilon values ​​were tested for both the base CNN model and the XAI-CNN model: 0.01, 0.02, 0.03, 0.04 and 0.05.

[0168] For the husky and wolf problems, the minimum epsilon value was 0.0005. Therefore, the following epsilon values ​​were tried: 0.0005, 0.0010, 0.0015 and 0.0020.

[0169] To easily demonstrate the accuracy reduction, we used five different epsilon values ​​for MNIST, compared to four for huskies and wolves. For MNIST, the higher epsilon value was 0.05.

[0170] Note that the difference in epsilon values ​​for the two problems is due to the difference in the image backgrounds: the MNIST images have plain backgrounds, while the husky and wolf images appear in natural environments such as forests, parks, or bedrooms. Therefore, the MNIST images require more perturbations to generate incorrect classifications.

[0171] Example images from MNIST, the Husky and Wolf datasets are shown at various epsilon values. Note that rough testing does not reveal any differences between the images.

[0172] MNIST - Handwritten Digit Recognition:

[0173] Data - From the MNIST dataset of approximately 60,000 images, a subset of 6,000 images was sampled per digit. These were then split in half for training and testing. For digit regions, the top and bottom halves were cut, then the left and right halves were cut, and then each of the samples was cut diagonally. This resulted in 6 region images per digit image. This generated 6,000 images per region type (e.g., top half) for each digit class (e.g., 5), for a total of 42,000 [= (6 regions + 1 full image) x 6000] images per digit type. Including regions, there were 70 image classes for the 10 digits in the XAI model.

[0174] FIG. 20 illustrates an example base CNN model utilizing a custom convolutional neural network architecture for MNIST, according to a described embodiment.

[0175] 21 illustrates an exemplary base XAI-CNN model utilizing a custom convolutional neural network architecture for the MNIST explainable AI model, according to a described embodiment. In particular, the prediction expressed for a given digit is divided into seven regions: the lower diagonal, the lower half, the complete digit, the left half, the right half, the upper diagonal, and finally the upper half. This prediction is performed digit by digit, eventually ending with the upper half, which is the final region of the digit (the digit "9" shown in the example).

[0176] FIG. 22 shows Table 7 in element 2200, according to the described embodiment, showing the average test accuracy of the MNIST base CNN model for adversarial images generated by various epsilon values ​​across 10 different runs.

[0177] FIG. 23 shows Table 8 in element 2300, according to the described embodiment, showing the average test accuracy of the XAI-CNN model for adversarial images generated by various epsilon values ​​across 10 different runs.

[0178] FIG. 24 shows Table 9 in element 2400 according to the described embodiment, showing the average test accuracy of the base CNN models of huskies and wolves for adversarial images generated with various epsilon values ​​across 10 different runs.

[0179] FIG. 25 shows Table 10 in element 2500 according to the described embodiment, showing the average test accuracy of the XAI-CNN models of huskies and wolves for adversarial images generated with various epsilon values ​​across 10 different runs.

[0180] Model Architecture and Results - For adversarial testing, the architecture in Figure 6A was utilized for the explainable model. This model uses a multi-label CNN model and does not involve an additional MLP. The model shown in Figure 6B shows a custom-built single-label CNN model used as a base model for MNIST. This base model was trained on the global image but not on any of the part images. The model has 10 nodes with softmax activation function for the 10 digits. We compared the results of the explainable XAI-CNN model as shown in Figure 20, which shows the base CNN model. Specifically, a multi-label XAI-CNN model was trained on both global and part images of digits.

[0181] For testing, we trained the base CNN model 10 times with 30 epochs each time using the categorical cross entropy loss function and the adam optimizer. We tested the base CNN model on adversarial images generated with different epsilon values. Table 7, shown in Figure 22, shows the average test accuracy on the adversarial images across 10 different runs with different epsilon values.

[0182] The explainable AI model (XAI-CNN) shown in Figure 21 has the same network configuration as the basic model in Figure 21, with the key differences being (1) the number of nodes in the output layer is now 70 instead of just 10, (2) the activation function of the output layer (here we use sigmoid), and (3) the loss function is binary cross entropy. The other main differences are that the XAI-CNN model is a multi-label model with 70 output nodes, with 7 output nodes for each digit, and 6 of these 7 nodes belong to different parts of the digit.

[0183] We tested the model on adversarial images generated using the XAI-CNN model with various epsilon values. Table 8, shown in Figure 23, shows the average test accuracy for the XAI-CNN model across 10 different runs for different epsilon values.

[0184] Data - The same data sets for huskies and wolves were again used as in the experiments already described.

[0185] Model Architecture and Results - As usual, for adversarial testing, we used the architecture in Figure 6A for the explainable model. However, unlike MNIST, in this case we utilized the Xception model for transfer learning. For transfer learning, the process is to freeze the weights of the convolutional layers, then add a flattening layer, then add a fully connected (FC) layer, and then add an output layer. The weights of the fully connected layer were then trained for the new classification task.

[0186] The base CNN model is always a single-label classification model. The base CNN model, consisting of the Xception model plus an additional layer, was trained on the entire set of husky and wolf images. The model had an output layer with two nodes and a softmax activation function.

[0187] For the explainable AI model (XAI-CNN) in Figure 6A, i.e., the multi-label model, there were 14 output nodes with sigmoid activation function. The multi-label model was then trained on both global and local images of huskies and wolves. The loss function and optimizer used were the same as for MNIST. Both the base CNN model and the XAI-CNN model were trained 10 times with 50 epochs. The models were tested on adversarial images generated using each model with various epsilon values. Table 9 shown in Figure 24 shows the average test accuracy of the base CNN model on the adversarial images across 10 different runs for different epsilon values. Table 10 shown in Figure 25 shows the same for the XAI-CNN model.

[0188] Adversarial Attack Results - Tables 7 and 8 (see Figures 22 and 23) show that in the absence of distortion (epsilon=0), both the base CNN model and the XAI-CNN model have an accuracy of about 98% on MNIST images. However, for the base CNN, the average accuracy drops to 85.89% at epsilon 0.05. In contrast, the accuracy of the XAI-CNN model drops from 97.97% to 97.71% at epsilon 0.05. The accuracy drop of the base CNN model is 12.5%, while the accuracy drop of the XAI-CNN model is only 0.26%.

[0189] Tables 9 and 10 (see Figures 24 and 25) show the average accuracy for the husky and wolf datasets. Table 9 shows that the average accuracy of the base CNN model drops to 45.52% at epsilon 0.002 and drops to 88.01% at epsilon 0. Table 10 shows that the average accuracy of the XAI-CNN model drops to 83.35% at epsilon 0.002 and drops to 85.08% at epsilon 0. Thus, the accuracy of the base CNN model drops by 45.52% compared to the drop of only 1.73% for the XAI-CNN model.

[0190] Overall, these results show that compared to regular CNN models, DARPA-style explainable models are relatively immune to low-level adversarial attacks, primarily because multi-label models are aware of object regions and cannot be fooled as easily.

[0191] Explainability assessment

[0192] Since the object-part explainability framework described herein is constructive and user-defined, it is the user's responsibility to assess the validity of the explanation. At one extreme, the user may define an explanation using a minimum number of parts, thereby keeping the explanation simple while keeping the system's performance consistent. For example, to predict that an image is an image of a cat, it is sufficient to verify that the face is an image of a cat. At the other extreme, the user may define an explanation with many parts, with some redundancy built into it. For example, to predict that an image is an image of a cat, the user may want to verify many details - from ears, eyes, and tails to whiskers, claws, and face. For critical applications such as medical and defense, it would be reasonable to assume that the team defines which parts should be verified for a necessary and sufficient explanation. In summary, the responsibility for evaluating the explanation is on the user's side, and the user must verify that the explanation is consistent with the system's prediction. This part-based framework provides the freedom to construct explanations according to the requirements of a particular implementation and the necessary goals or desires specified by the user.

[0193] Conclusion:

[0194] The embodiments of the present invention have presented herein an approach to explainable AI for identifying parts of an object in an image and predicting the type (class) of an object only after verifying the presence of a specific part of this object type in the image. The original DARPA concept of the symbolic XAI model was this part-based model. In the embodiments described herein, the user defines (designs) the XAI model in the sense that he or she must define the parts of the object that he or she wants to verify for the prediction of the object.

[0195] An embodiment of the present invention builds an XAI symbolic model by decoding the CNN model. To generate the symbolic model, an embodiment of the present invention uses CNN and MLP models that remain black boxes. In the work presented here, an embodiment of the present invention kept the CNN and MLP models separate to understand the decoding of sites from the fully connected layer of the CNN. However, the two models may be integrated into a single model.

[0196] In this study, embodiments of the present invention have demonstrated that the configuration of an object can be easily taught from its parts simply by using a multi-label (multi-target) classification model and by indicating the parts of each object. By using a multi-label classification model, embodiments of the present invention avoid indicating the exact location of the parts. Embodiments of the present invention allow the learning system to understand the connectivity between parts and the relative locations of the parts.

[0197] The generation and annotation of object parts is currently a tedious manual process. Embodiments of the present invention are currently looking for ways to automate this process so that they can extract a large number of annotated parts from a variety of images after feeding the system a small annotated training set. If embodiments of the present invention develop such a method, they should be able to perform some large-scale testing of our approach. In this document, we have only wanted to introduce the basic ideas and demonstrate with some limited experiments that embodiments of the present invention can work and generate symbolic XAI models.

[0198] From the experiments so far, it seems that the part verification-based prediction model can increase the prediction accuracy, but more experiments are needed to confirm this claim. This speculation is reasonable if we assume that humans identify objects from their parts.

[0199] It is possible that part-based object verification may be able to protect against adversarial attacks, but this conjecture also requires experimental validation. If embodiments of the present invention could validate this conjecture, adversarial learning may not be necessary.

[0200] Overall, the part-based symbolic XAI model not only brings transparency to our CNN models for image recognition, but may also have the potential to increase prediction accuracy and protection against adversarial attacks.

[0201] Solutions to technical issues:

[0202] In the context of new AI technologies, there is a need to develop processing solutions for UAV (Unmanned Aerial Vehicle) images and video as well as CCTV (Closed Circuit Television) images and video, a need that remains unmet even with the state of the art and currently available technologies.

[0203] Deep learning is the most recent technology for video processing. However, deep learning models are difficult to understand due to their lack of transparency. Thus, there is growing concern about deploying deep learning models in rather dangerous situations where incorrect decisions can lead to legal liability. For example, fields such as medicine are hesitant to deploy the use of deep learning models and techniques to automate the reading and interpretation of images in radiology due to the obvious danger to human life in case of incorrect decisions or faulty diagnoses. The same kind of danger exists in the automation of video processing by deep learning for CCTV and UAVs, where incorrect decisions by black box (e.g. non-transparent) models can lead to undesirable consequences.

[0204] Because deep learning models have a high degree of accuracy, there is ongoing research to make them explainable and transparent. DARPA has initiated an explainable AI program because critical DoD applications have disastrous consequences and black box models cannot be used. The NSF also dedicates significant funding to explainability research.

[0205] Currently, computer vision has several interpretable methods. However, the dominant techniques such as LIME, SHAP and Grad-CAM each rely on visualization, which means that a human is needed to view the image in each case. Therefore, using these known prior art techniques, it is not possible to easily generate a system that allows automated video processing using such methods "without human intervention". Innovative solutions are therefore urgently needed to overcome the current limitations.

[0206] New AI techniques are needed:

[0207] The generation of symbolic models from deep learning models is a significant innovation for generating transparent models.

[0208] Symbolic Models: The DARPA idea of ​​part-based explanations provides a good framework for symbolic models. For example, using the DARPA framework, the logic rules for recognizing a cat might be:

[0209] IF the Fur is of a Cat AND the Whiskers are of a Cat AND the Claws are of a Cat THEN it is a Cat.

[0210] Here, cat, hair, whiskers, and claws are abstract concepts represented by their corresponding symbols of the same name, and the modified deep learning model may output TRUE / FALSE values ​​for these symbols to indicate the presence or absence of these parts in the image. The above logic rules are symbolic models that are easily processed by computer programs, do not require visualization, and do not require a human-in-the-loop. In a particular scene, there may be multiple objects in the scene. In an example video from a security camera (e.g., a bear wakes up Greenfield man sleeping by pool, YouTube), a bear is observed in the backyard and a man is observed sleeping by the pool. An intelligent security system would instantly be notified that an unknown animal is nearby. The explainable symbolic model would generate the following information for the security system:

[0211] 1. Unknown Animal(true), Face(true), Body(true), Limb(true);

[0212] 2. person(true), leg(true), foot(true), face(false), arm(false);

[0213] 3. Swimming pool (true), chaise lounge (true)...

[0214] This is the new kind of explainable symbolic system we describe here. Again, the method we disclose does not rely on any visualization and therefore does not require any human-in-the-loop. Moreover, this kind of transparent model should increase trust and confidence in the system and open the door to a wider deployment of deep learning models.

[0215] The resulting model also provides protection against adversarial attacks due to part verification – so a school bus doesn’t turn into an ostrich because of a few pixel changes.

[0216] Large-scale automated video processing with explainable AI models for certainty and trust

[0217] In addition to the above, those familiar with the field of video processing will readily recognize the problem of non-scalability, which has only gotten worse in recent years with the increasing amount of data being captured and needing processing, along with the proliferation of security cameras.

[0218] Video processing in surveillance systems, from drones and UAVs to CCTV, is quite labor intensive. Often, the video is simply stored for later review due to manpower shortages. In other cases, the video requires real-time processing. But ultimately, both cases require a human to observe and process the captured data. In the future, video processing must be fully automated due to the increasing volume. This will reduce labor costs and help in manpower limited situations. With the volume of video generated from UAVs and CCTV growing rapidly, labor-intensive video processing is a significant issue that must be addressed.

[0219] Consider the following quote, which talks about the security system of the future: "In the future, a pan-tilt-zoom camera running AI analytics at the entrance will identify weapons carried by a person, zoom in, take a closer look, and instruct the access control system to lock the door and prevent entry. At the same time, the pan-tilt-zoom camera will send an alert to the security team, occupants or authorities, and may even autonomously deploy drones to find and attack this person. In other words, the system prevents potential harmful incidents without human intervention."

[0220] To bypass "human intervention", such systems must be highly reliable and trustworthy. Deep learning is now the preeminent technology for image processing. However, the decision logic of deep learning models is difficult to understand, so NSF, DoD, and DARPA are all exploring "explainable AI" as a way to overcome the problems of traditional deep learning and non-transparent AI.

[0221] Thus, the described embodiments provide a "part-based explainable system" that meets the stated DARPA objectives. Tests have shown that the method is successful on exemplary problems such as cat and dog recognition, and will be extended to work on increasingly complex scenes, such as scenes from CCTV and UAVs. Imagine the complexity of a scene in a hospital ICU or inside a store with many different objects. The task of defining the parts of hundreds of different objects poses a problem that has never been solved before by any conventional known image recognition technique.

[0222] Explainable models are needed to address complex scenes with part definitions for thousands of objects. Ideas often work well for simple problems but fail miserably for more complex problems. However, without explainable deep learning models, these systems that operate by design "without human intervention" will produce unacceptably high false positives. Furthermore, the use of explainable AI models allows humans to guide and drive the technology towards best practices, while the AI ​​models can learn and progress through the consumption of increasingly large and accessible training datasets.

[0223] Thus, human-in-the-loop is intentionally removed from the execution of the AI ​​models obtained and implemented based on the teachings described herein, as the described AI models are expressly made to be "explainable AI models." Nonetheless, it is possible to use human thought to improve and deploy technologies that do not force human intervention into automated processes, which would prevent the large-scale use of such technologies.

[0224] FIG. 26 illustrates a flow diagram showing a method 2600 of implementing a transparency model for computer vision and image recognition utilizing a deep learning non-transparent black box model according to disclosed embodiments. The method 700 may be implemented by processing logic, including hardware (e.g., circuits, dedicated logic, programmable logic, microcode, etc.), software (e.g., instructions executed on a processing device), to perform various operations such as designing, defining, extracting, parsing, persisting, exposing, loading, executing, running, receiving, generating, storing, maintaining, generating, returning, presenting, interfacing, communicating, transmitting, querying, processing, providing, determining, triggering, displaying, updating, sending, etc., according to the systems and methods described herein. For example, the system 2701 (see FIG. 27) and machine 2801 (see FIG. 28) and other supporting systems and components described herein may implement the described methods. Some of the blocks and / or operations listed below are optional according to some embodiments. The numbering of the presented blocks is for clarity and is not intended to dictate the order of operations that the various blocks must perform.

[0225] 26, there is a method 2600 performed by a specially configured system that systematically generates and outputs transparency models for computer vision and image recognition utilizing deep learning non-transparent black box models. Such a system may be configured with at least one processor and memory that executes special instructions that cause the system to perform the following operations:

[0226] In block 2605, the processing logic of such a system generates a transparent, explainable AI model for computer vision or image recognition from a non-transparent black box AI model via the following operations.

[0227] At block 2610, processing logic trains a convolutional neural network (CNN) to classify objects from training data having a training image set.

[0228] At block 2615, processing logic trains a multi-layer perceptron (MLP) to recognize both objects and parts of objects.

[0229] At block 2620, processing logic generates an explainable AI model based on training the MLP.

[0230] At block 2625, processing logic receives an image with an object embedded therein. The image does not form part of the training data for the explainable AI model.

[0231] At block 2630, processing logic executes the CNN and the explainable AI model in the image recognition system to generate predictions of objects in the image via the explainable AI model.

[0232] At block 2635, processing logic recognizes parts of the object.

[0233] At block 2640, processing logic provides the recognized locations within the object as evidence of the prediction of the object.

[0234] At block 2645, processing logic generates an explanation for why the imaging system predicted the object in the image based on evidence including the recognized features.

[0235] According to another embodiment of method 2600, training an MLP to recognize both objects and object parts includes performing an MLP training procedure through operations including: (i) presenting training images selected from the training data to a training CNN; (ii) loading activation functions of a fully connected (FC) layer of the CNN; (iii) receiving these activation functions as inputs to the MLP; (iv) setting multi-target outputs for the training images; and (v) adjusting weights of the MLP according to one of the weight adjustment methods.

[0236] According to another embodiment, the method 2600 further includes transmitting at least a portion of the recognized features and descriptions within the object to an explanation user interface (UI) for display to a user of the image recognition system.

[0237] According to another embodiment of method 2600, identifying the part of the object includes decoding a convolutional neural network (CNN) to recognize the part of the object.

[0238] According to another embodiment of method 2600, the CNN decoding includes providing information regarding the configuration of the object, the information including parts of the object and part connectivity for the model decoding the CNN.

[0239] According to another embodiment of method 2600, the connectivity of the sites includes the spatial relationship between the sites.

[0240] According to another embodiment of method 2600, the model is a multi-layer perceptron (MLP) that is separate from or integrated with the CNN model, and the integrated model is trained to recognize both objects and parts.

[0241] According to another embodiment of the method 2600, providing information about a configuration of the object further includes providing information including a subassembly of the object.

[0242] According to another embodiment of the method 2600, recognizing the part of the object includes testing a user-defined list of parts of the object.

[0243] According to another embodiment of method 2600, training the CNN to classify the objects includes training the CNN to classify the target objects using transfer learning.

[0244] According to another embodiment of method 2600, transfer learning includes at least the following operations: freezing some or all convolutional layer weights of a pre-trained CNN pre-trained on similar object classes, adding one or more flattened fully connected (FC) layers, adding an output layer, and training both the fully connected and unfrozen convolutional layer weights for a new classification task.

[0245] According to another embodiment of method 2600, training the MLP to recognize both objects and parts of the object includes receiving inputs from activation functions of one or more fully connected layers of the CNN and providing target values ​​from a user-defined list of parts for output nodes of the MLP, the output nodes corresponding to an object defined as a target object specified by the user-defined list of parts and parts of the target object according to the user-defined list of parts.

[0246] According to another embodiment, the method 2600 further includes generating a transparent and explainable AI model from the non-transparent black box AI model through operations further including training and testing a convolutional neural network (CNN) having a set of fully connected (FC) layers using M images of C object classes, and training a multi-target MLP using a subset of a total set of images M, where M includes the original M images for CNN training and a further set M of part and sub-assembly images, where each image M in M k For training, image im kas an input to a training CNN; recording activation functions at one or more designated FC layers; receiving the activation functions of the one or more designated FC layers as input to a multi-target MLP; and recording an image IM k TR as a multi-target output vector for k and adjusting the weights of the MLP according to a weight adjustment algorithm.

[0247] According to another embodiment of method 2600, training the CNN includes training the CNN from scratch or by using transfer learning to add an FC layer.

[0248] According to another embodiment of method 2600, training a multi-target MLP using a subset of the total set of images MT, where MT includes the original M images for CNN training and a further set MP of part and sub-assembly images, includes teaching the construction of M images of C object classes from the further set MP of part and sub-assembly images and part connectivity.

[0249] According to another embodiment of method 2600, teaching the configuration of M images of the C object class from a further set MP of part and sub-assembly images and part connectivity includes identifying the parts by showing MLP individual images of the parts, and identifying the sub-assemblies and listing the parts contained therein by showing MLP images of the sub-assemblies, such that taking into account the part list for the assembly or sub-assembly and the corresponding image, the MLP learns the configuration of the object and sub-assembly, and the connectivity of the parts.

[0250] According to certain embodiments, there is a non-transitory computer readable storage medium having instructions stored therein that, when executed by a system having at least one processor and a memory therein, cause the system to perform operations including training a convolutional neural network (CNN) to classify objects from training data having a training image set; training a multi-layer perceptron (MLP) to recognize objects and object parts; generating an explainable AI model based on the training of the MLP; receiving an image having an object embedded therein, where the image does not form part of the training data for the explainable AI model; executing the CNN and the explainable AI model in an image recognition system and generating a prediction of the object in the image via the explainable AI model; recognizing the object parts; providing the recognized parts in the object as evidence for the prediction of the object; and generating an explanation for why the image system predicted the object in the image based on the evidence including the recognized parts.

[0251] 27 shows a schematic diagram of a system 2701 in which embodiments may be run, installed, integrated, or configured. According to one embodiment, there is a system 2701 having at least one processor 2790 executing implementing application code 2796 and memory 2795 therein. Such a system 2701 may be communicatively interfaced and collaboratively executed with the aid of a remote system, such as a user device transmitting instructions and data, and a user device receiving as output from the system 2701 a specially trained “explainable AI” model 2766 having extracted features 2743 therein for use and display to a user via an explainable AI user interface that provides a transparent explanation of the decisions that have been located as “sites” within a target input image 2741 for which the “explainable AI” model 2766 has rendered predictions.

[0252] According to the embodiment shown, the system 2701 includes a processor 2790 and a memory 2795 for executing instructions on the system 2701. The system 2701 shown here is specifically customized and configured to systematically generate transparency models for computer vision and image recognition utilizing deep learning non-transparent black box models. Training data 2739 is processed through an image feature learning algorithm 2791 from which determined "body parts" 2740 are extracted for a number of different objects (e.g., "cat" and "dog", etc.), and a pre-training and fine-tuning AI manager 2750 may be optionally utilized to refine predictions for a given object based on further training data provided to the system.

[0253] According to certain embodiments, there is a specially configured system 2701 that is custom configured to generate transparent and explainable AI models for computer vision or image recognition from non-transparent black box AI models. According to such embodiments, the system 2701 includes a memory 2795 that stores instructions via executable application code 2796, and a processor 2790 that executes the instructions stored in the memory 2795, the system 2701 being specially configured to execute the instructions stored in the memory via the processor, for training a convolutional neural network (CNN) 2765 to classify objects embedded in a training image set including training data 2739, training the convolutional neural network (CNN) 2765 to classify objects from the training data 2739 having a set of training images, and training a multi-layer perceptron (MLP) via an image feature learning algorithm 2791 to recognize objects and object parts. generating an explainable AI model 2766 based on training of the MLP; receiving an image (e.g., input image 2741) with an object embedded therein, where image 2741 does not form part of training data 2739 for the explainable AI model 2766; running the CNN and explainable AI model 2766 in an image recognition system and generating a prediction of the object in the image via the explainable AI model 2766; recognizing parts of the object; providing the recognized parts in the object via extracted features 2743 for the explainable AI model as evidence for the prediction of the object; and generating an explanation for why the image system predicted the object in the image based on the evidence including the recognized parts.

[0254] According to another embodiment of the system 2701, the user interface 2726 communicatively interfaces with a user client device remote from the system and communicatively interfaces with the system via the public Internet.

[0255] Bus 2716 interfaces the various components of system 2701 with each other, with any other peripheral device(s) of system 2701, and with external components such as external network elements, other machines, client devices, cloud computing services, etc. Communications may further include communications with external devices over a LAN, WAN, or public internet via the network interfaces.

[0256] FIG. 28 shows a schematic diagram of a machine 2801 in the example form of a computer system according to one embodiment, within which a set of instructions may be executed by the machine / computer system to perform any one or more of the methods described herein.

[0257] In alternative embodiments, the machine may be connected (e.g., networked) to other machines in a local area network (LAN), an intranet, an extranet, or the public Internet. The machine may operate in the capacity of a server or a client machine in a client-server network environment, as a peer machine in a peer-to-peer (or distributed) network environment, or as a server or a series of servers in an on-demand service environment. Particular embodiments of the machine may be in the form of a personal computer (PC), a tablet PC, a set-top box (STB), a personal digital assistant (PDA), a mobile phone, a web appliance, a server, a network router, a switch or bridge, a computing system, or any machine capable of executing (sequentially or otherwise) a set of instructions that specify or request the machine to take a specifically configured action in accordance with the stored instructions. Moreover, although only a single machine is depicted, the term "machine" shall be construed to include any collection of machines (e.g., computers) that individually or jointly execute a set (or sets) of instructions to implement any one or more of the methods discussed herein.

[0258] Exemplary computer system 2801 includes a processor 2802, a main memory 2804 (e.g., read only memory (ROM), flash memory, dynamic random access memory (DRAM) such as synchronous DRAM (SDRAM) or Rambus DRAM (RDRAM), flash memory, static random access memory (SRAM), static memory such as volatile high data rate RAM), and a secondary memory 2818 (e.g., persistent storage devices including hard disk drives and persistent database and / or multi-tenant database implementations), which communicate with each other via a bus 2830. Main memory 2804 includes instructions for performing a transparent learning process 2824, which provides extracted features 2823 for use by a user interface in support of the methods and techniques described herein, generates a trained explainable AI model 2825, and makes the trained explainable AI model 2825 available for execution. The main memory 2804 and sub-elements of the main memory 2804 may further operate in conjunction with the processing logic 2826 and the processor 2802 to implement the methods discussed herein.

[0259] Processor 2802 represents one or more specialized, specially configured processing devices, such as a microprocessor, a central processing unit, etc. More specifically, processor 2802 may be a complex instruction set computing (CISC) microprocessor, a reduced instruction set computing (RISC) microprocessor, a very long instruction word (VLIW) microprocessor, a processor implementing other instruction sets, or a processor implementing a combination of instruction sets. Processor 2802 may also be one or more special purpose processing devices, such as an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), a digital signal processor (DSP), a network processor, etc. Processor 2802 is configured to execute processing logic 2826 that performs the operations and functions discussed herein.

[0260] The computer system 2801 may further include a network interface card 2808. The computer system 2801 may further include a user interface 2810 (such as a visual display unit, liquid crystal display, etc.), an alphanumeric input device 2812 (e.g., a keyboard), a cursor control device 2813 (e.g., a mouse), and a signal generating device 2816 (e.g., an integrated speaker). The computer system 2801 may further include peripheral devices 2836 (e.g., wireless or wired communication devices, memory devices, storage devices, audio processing devices, video processing devices, etc.).

[0261] The secondary memory 2818 may include a non-transitory machine-readable storage medium or non-transitory computer-readable storage medium or non-transitory machine-accessible storage medium 2831 on which one or more sets of instructions (e.g., software 2822) for implementing any one or more of the methods or functions described herein are stored. The software 2822 may reside, completely or at least partially, within the main memory 2804 and / or the processor 2802 during execution of the software 2822 by the computer system 2801, the main memory 2804, and the processor 2802, which also constitute machine-readable storage media. The software 2822 may further be transmitted or received over the network 2820 via the network interface card 2808.

[0262] Although the subject matter disclosed herein has been described in terms of specific embodiments by way of example, it should be understood that the claimed embodiments are not limited to the disclosed embodiments expressly recited. Instead, the present disclosure is intended to encompass various modifications and similar arrangements as would be apparent to one skilled in the art. Therefore, the scope of the appended claims should be accorded the broadest interpretation so as to encompass all such modifications and similar arrangements. It should be understood that the above description is intended to be illustrative, and not restrictive. Numerous other embodiments will be apparent to those skilled in the art upon reading and understanding the above description. Therefore, the scope of the disclosed subject matter should be determined with reference to the appended claims, along with the full scope of equivalents to which such claims are entitled.

Claims

1. 1. A computer-implemented method performed by a system having at least one processor and memory therein for generating a transparent, explainable AI model for computer vision or image recognition from a non-transparent black box AI model, the method comprising: training a convolutional neural network (CNN) to classify objects from training data having a training image set; training a multi-layer perceptron (MLP) to recognize the object and portions of the object, said training comprising: (i) presenting training images selected from the training data to the CNN for training; (ii) reading an activation function of a fully connected (FC) layer of the CNN; (iii) receiving the activation function as an input to the MLP; (iv) setting multi-target outputs for the training images; and (v) adjusting weights of the MLP according to a weight adjustment method. training by performing an MLP training procedure via an operation including: generating the explainable AI model based on training the MLP; receiving an image having an object embedded therein, the image not forming part of the training data for the explainable AI model; Executing the CNN and the explainable AI model in an image recognition system and generating predictions of the objects in the image via the explainable AI model; Recognizing a portion of the object; providing the location recognized within the object as evidence of the prediction of the object; generating an explanation for why the imaging system predicted the object in the image based on evidence including the recognized region; and A method comprising:

2. transmitting the recognized features of the object and at least a portion of the description to a description user interface (UI) for display to a user of the image recognition system. The method of claim 1 further comprising:

3. The method of claim 1 , wherein identifying the object part comprises decoding a convolutional neural network (CNN) to recognize the object part.

4. 4. The method of claim 3, wherein the decoding of the CNN includes providing information about the configuration of the object, the information including parts of the object and connectivity of the parts for a model that decodes the CNN.

5. The method of claim 4 , wherein the connectivity of the sites comprises a spatial relationship between the sites.

6. 6. The method of claim 5, wherein the model is a multi-layer perceptron (MLP) that is separate from or integrated with the CNN model, and the integrated model is trained to recognize both the object and the part.

7. The method of claim 5 , wherein providing information about the configuration of the object further comprises providing information including subassemblies of the object.

8. The method of claim 1 , wherein recognizing the object part comprises testing a user-defined list of the object parts.

9. The method of claim 1 , wherein training the CNN to classify objects comprises using transfer learning to train the CNN to classify objects of interest.

10. Transfer learning is Freezing the weights of some or all convolutional layers of a pre-trained CNN that has been pre-trained on similar object classes; adding one or more planarizing fully connected (FC) layers; Adding an output layer; training the weights of both the fully connected and unfrozen convolutional layers for a new classification task; 10. The method of claim 9, comprising:

11. Training an MLP to recognize both the object and the object part comprises: receiving inputs from activation functions of one or more fully connected (FC) layers of the CNN; providing target values ​​from a user-defined list of locations for output nodes of said MLP; 2. The method of claim 1 , comprising:

12. Generating a transparent and explainable AI model for computer vision or image recognition from a non-transparent black box AI model and said generating further comprises: training and testing the convolutional neural network (CNN) having a set of fully connected (FC) layers using M images of C object classes; training the multi-target MLP using a subset of the total image set M; wherein the MT includes the original M images for CNN training and a further set MP of region and sub-assembly images; Each image IM in the MT ｋ Training for (i) Image IM ｋ as input to the training CNN; (ii) recording activation functions at one or more designated FC layers; (iii) receiving the one or more specified FC layer activation functions as inputs to the multi-target MLP; (iv) the image IM ｋ TR as a multi-target output vector for ｋ and (v) adjusting the weights of the MLP according to a weight adjustment algorithm; The method of claim 1 , comprising:

13. 13. The method of claim 12, wherein training the CNN comprises training the CNN from scratch or by using transfer learning to add an FC layer.

14. 13. The method of claim 12, wherein training the multi-target MLP using a subset of the total set of images M T that includes the original M images for CNN training and the further set MP of part and sub-assembly images comprises teaching the composition of M images of objects of a C object class from the further set MP of part and sub-assembly images and part connectivity.

15. Teaching the construction of M images of C object classes from said further set MP of region and sub-assembly images and region connectivity, identifying the region by indicating the region in the MLP individual image; Identifying sub-assemblies and listing the features contained therein by showing the sub-assemblies in the MLP image, so that the MLP learns the configuration of the object and the connectivity of the sub-assemblies and the features taking into account the feature list for the assembly or sub-assembly and the corresponding image; providing the site list to the MLP in the form of a multi-target output for the image; 15. The method of claim 14, comprising:

16. a memory for storing instructions; a processor for executing instructions stored in said memory; A system comprising: the system is specially configured to execute instructions stored in the memory via the processor; training a convolutional neural network (CNN) to classify objects; training a convolutional neural network (CNN) to classify objects from training data having a training image set; training a multi-layer perceptron (MLP) to recognize the object and portions of the object, said training comprising: (i) presenting training images selected from the training data to the CNN for training; (ii) reading an activation function of a fully connected (FC) layer of the CNN; (iii) receiving the activation function as an input to the MLP; (iv) setting multi-target outputs for the training images; and (v) adjusting weights of the MLP according to a weight adjustment method. training by performing an MLP training procedure via an operation including: generating the explainable AI model based on training the MLP; receiving an image having an object embedded therein, the image not forming part of the training data for the explainable AI model; Executing the CNN and the explainable AI model in an image recognition system and generating predictions of the objects in the image via the explainable AI model; Recognizing a portion of the object; providing the location recognized within the object as evidence of the prediction of the object; generating an explanation for why the imaging system predicted the object in the image based on evidence including the recognized region; and A system that causes the system to perform operations including:

17. transmitting the recognized features of the object and at least a portion of the description to a description user interface (UI) for display to a user of the image recognition system. The system of claim 16 further comprising:

18. identifying the object part includes decoding a convolutional neural network (CNN) to recognize the object part; The decoding of the CNN includes providing information about the configuration of the object, the information including parts of the object and connectivity of the parts for a model that decodes the CNN; the connectivity of the sites includes the spatial relationship between the sites; the model is a multi-layer perceptron (MLP) separate from or integrated with the CNN model, the integrated model being trained to recognize both the object and the part; The system of claim 16 , wherein providing information about the configuration of the object further comprises providing information including subassemblies of the object.

19. A non-transitory computer-readable storage medium having instructions stored therein, the instructions, when executed by a processor of a system, training a convolutional neural network (CNN) to classify objects from training data having a training image set; training a multi-layer perceptron (MLP) to recognize the object and portions of the object, said training comprising: (i) presenting training images selected from the training data to the CNN for training; (ii) reading an activation function of a fully connected (FC) layer of the CNN; (iii) receiving the activation function as an input to the MLP; (iv) setting multi-target outputs for the training images; and (v) adjusting weights of the MLP according to a weight adjustment method. training by performing an MLP training procedure via an operation including: generating the explainable AI model based on training the MLP; receiving an image having an object embedded therein, the image not forming part of the training data for the explainable AI model; Executing the CNN and the explainable AI model in an image recognition system and generating predictions of the objects in the image via the explainable AI model; Recognizing a portion of the object; providing the location recognized within the object as evidence of the prediction of the object; generating an explanation for why the imaging system predicted the object in the image based on evidence including the recognized region; and a non-transitory computer-readable storage medium that causes the system to perform operations including:

20. By the said order, transmitting the recognized features of the object and at least a portion of the description to a description user interface (UI) for display to a user of the image recognition system.

20. The non-transitory computer-readable storage medium of claim 19, further comprising:

21. identifying the object part includes decoding a convolutional neural network (CNN) to recognize the object part; The decoding of the CNN includes providing information about the configuration of the object, the information including parts of the object and connectivity of the parts for a model that decodes the CNN; the connectivity of the sites includes the spatial relationship between the sites; the model is a multi-layer perceptron (MLP) separate from or integrated with the CNN model, the integrated model being trained to recognize both the object and the part; 20. The non-transitory computer-readable storage medium of claim 19, wherein providing information about the configuration of the object further comprises providing information including subassemblies of the object.

Images