Methods, systems, and media for determining an explainability mask by a neural network

Abstract

This invention relates to a computer implementation method for determining an interpretable mask for classifying an input image by a trained neural network. The trained neural network is configured to determine the classification and classification score of an input image by determining a latent representation of the input image at an inner layer of the trained neural network. The method includes: accessing the trained neural network; obtaining the input image and its latent representation; and initializing a mask for indicating modifications to the latent representation. The mask is updated by iteratively adjusting the values ​​of the mask to optimize an objective function, which includes: i) a modification component indicating the degree of modification indicated by the mask; and ii) a classification score component, determined by applying the indicated modifications to the latent representation and determining its classification score. The mask is scaled to the spatial resolution of the input image and the output.

Description

Technical Field

[0001] This invention relates to a computer-implemented method and system for determining a mask for classifying an input image by a trained neural network. The invention further relates to a computer-readable medium comprising instructions for performing one of the above-described methods. Background Technology

[0002] Machine learning-based (“trained”) models are widely used in many real-world applications, such as autonomous driving, robotics, manufacturing, and building control. For example, machine-learnable models (such as neural networks) can be trained to infer the state of a physical system (such as an autonomous vehicle or robot) or its environment (e.g., the road on which the vehicle travels, the robot’s workspace) based on sensor data acquired by one or more sensors. Once the state has been inferred, the physical system can be controlled, for example, using one or more actuators, or its operation can be monitored.

[0003] Typically, as is known in machine learning, machine-learnable models (such as neural networks) can be trained on training data to provide outputs such as predictions or decisions. An example of a decision is classification. By training a neural network, it can provide such predictions or decisions without being explicitly programmed to do so. For example, to classify images, a neural network can be trained on a training dataset that includes images whose classifications are known. Thus, a trained neural network can be trained to determine the classification of an input image. For example, a neural network can be trained to use multi-class identifiers, by which an image can be classified as, for example, "cat," "dog," or "tree." Another example is that a trained neural network can be trained to be a binary classifier (e.g., classifying an input image into only one of two complementary categories, "OK" or "NOK"). However, a trained neural network can also be trained on a dataset of images whose categories are unknown (e.g., images obtained through clustering / automatic classification).

[0004] Generally speaking, neural networks can include deep neural networks and / or convolutional neural networks (CNNs).

[0005] Classifications performed by machine learning models (such as trained neural networks) can be used in countless applications, such as optical quality inspection in manufacturing processes or hazard detection in autonomous vehicles. It will be understood that understanding how machine learning models arrive at their conclusions is relevant in these and other application areas.

[0006] Therefore, explainability is a core element of both model validation (whether we have correctly built the model) and verification (whether we have built the correct model). It is well known that large, existing-technology network models can easily be fooled into providing false, high-confidence predictions; and consequently, their true generalization performance is often questionable. As with any other software component intended for field deployment, it is crucial that the model can be quantitatively validated and verified, for example, to establish the extent to which the model has learned the expected input-output relationships. Current state-of-the-art techniques in machine and deep learning lack the metrics and practices for measuring this effect, often working with limited datasets (and therefore datasets with inherent biases), and producing over-parameterized models compared to the amount of available data. Consequently, there is often a significant gap between what the model has actually learned and what the implementer believes the model has learned.

[0007] Over the past few years, the need to provide explanations for model predictions has been recognized, and several methods have been proposed and used with varying degrees of academic success. For example, in the context of image classification networks, most explanation methods are a hybrid of black-box (i.e., model-independent) and white-box methods, which compute attributes on pixels of the input image. That is, these methods study the local behavior of the classifier on a given input. These attribution maps are often also called explanation maps, saliency maps, or attention maps, and are often represented as heatmaps overlaid on the input image, highlighting the input image regions that enable the model to classify the input into one class or another.

[0008] In Ruth Fong and Andrea Vedaldi’s “Interpretable explanations of blackboxes by meaningful perturbation” (arXiv: 1704.03296, 2017 (https: / / arxiv.org / pdf / 1704.03296.pdf)), the model is treated as a black box and an optimization process is used to learn a meaningful mask on the input image, which forces the model to classify its input differently.

[0009] This issue is explored differently in Ramprasaath R. Selvaraju et al.'s "Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization" (arXiv: 1610.02391, 2017 (https: / / arxiv.org / pdf / 1610.02391.pdf)). In that work, the argument is that, to understand where the model is looking at the input image, it is often sufficient to examine the data representations in the deepest convolutional layers.

[0010] However, the inventors have recognized the need for a more quantitative assessment of how masks actually affect the classification of inputs. This is particularly important for real-world deployment scenarios where a good understanding of why saliency is generated or recommended is crucial. A decrease in model classification accuracy due to saliency perturbations is one such quantitative measure of this impact. Therefore, a more accurate quantitative measure of the interpretability of the model to be determined is needed. Summary of the Invention

[0011] According to a first aspect of the invention, a computer implementation method for determining an interpretability mask as defined in claim 1 is provided. According to another aspect of the invention, a corresponding computer-readable medium as defined in claim 14 is provided. According to an aspect of the invention, a corresponding system as defined in claim 15 is provided.

[0012] This invention relates to a computer-implemented method for determining an interpretable mask for classifying an input image by a trained neural network. The trained neural network can be configured to determine the classification of an input image by determining a latent representation of the input image at an inner layer of the trained neural network and by determining a classification of the input image based on the latent representation. Determining the classification of the input image can include determining a classification score for the input image. The method can include: accessing the trained neural network; obtaining the input image and its latent representation; and initializing a mask to indicate modifications to the latent representation. The method further includes: updating the mask by iteratively adjusting the values ​​of the mask to optimize an objective function. The objective function includes a modification component indicating the degree of modification indicated by the mask. The classification score component can be determined by applying the modifications indicated by the mask to the latent representation to obtain a perturbed latent representation; and by determining a classification score for the perturbed latent representation based on the trained neural network. The method further includes: scaling the mask to the spatial resolution of the input image to obtain an interpretable mask, and outputting the interpretable mask.

[0013] The above aspects relate to obtaining a latent representation of an input image. A latent representation (also known as an activation volume) is the output of an inner (e.g., hidden) layer of a trained neural network. An inner layer is a layer in the trained neural network between the input and output layers. For example, the latent representation of layer k can be viewed as the output of the k-th layer of the trained neural network. A latent representation can include the activations (e.g., weights) of nodes in the inner layers of the neural network. Therefore, a latent representation will typically have a coarser spatial resolution than the input image and often involves a deeper, more complex structure of the input image. A latent representation can provide an abstract, more semantic, and / or compressed representation of the input image while preserving its spatial relationships.

[0014] The above aspects relate to masks of the latent representation of an input image. A mask can also be referred to as a heatmap. A mask can indicate modifications to the latent representation of an input image. For example, a mask can indicate the portion of the input image to which modifications of the latent representation can be applied. Modifications can involve means for completely or partially obscuring regions of the latent representation, and / or means for blurring regions of the latent representation. Modifications can be viewed as “blocking” or inhibiting the activation of layers in a trained neural network from which the latent representation is output. The amount or extent of modification to be applied to which portion of the latent representation of the input image can be determined through an optimization process.

[0015] A mask can be determined to indicate modifications to the latent representation of the input image that are relevant to the resulting classification. For example, the mask can indicate portions of the latent representation to be modified to influence the classification indicated by the trained neural network. Modifications may result in different classifications or a significant drop in confidence scores / measures for the classification (e.g., as measured by the classification score). The mask is updated by optimizing an objective function (e.g., the mask value is changed), which will be further described below, to balance the importance of minor modifications with the importance of low classification scores. In other words, the mask can be updated to identify smaller or minimal regions of the latent representation, and thus smaller or minimal regions of the input image that the trained neural network relies on most when determining its classification. Doing so identifies regions containing features of the input image that the trained neural network weights more heavily. For example, if the input image is an image of a cat, the objective function can be optimized to identify a mask that covers the smallest possible region (e.g., the cat's ears) that, when masked, would result in a classification score corresponding to the "cat" class falling below a threshold, or until the trained neural network no longer determines that the input image contains a cat. This would instruct the trained neural network to consider the cat's ears as a primary indicator defining the content of "cat." Therefore, the mask can indicate the region in the input image that is most responsible for the trained neural network to achieve its determined classification. In the above respect, the mask is applied to the latent representation of the input image. Since the latent representation preserves the spatial relationship with the input image, this concept remains unchanged—the smallest possible region of the latent representation to be masked (which provides the lowest possible classification score for the corresponding class) will be identified by the proxy from the corresponding small region of the input image. Applying a mask to the latent representation of the input image results in a perturbed latent representation.

[0016] The above measures also involve determining a classification score for the perturbed latent representation based on a trained neural network. The trained neural network can be configured to provide a classification and a confidence level for that classification. The classification score is typically used to indicate this confidence level.

[0017] The above measures also involve performing optimization to determine the mask. This optimization is carried out by optimizing an objective function that includes a modification component and a classification score component. The modification component can indicate the amount or degree of modification indicated by the mask. The optimization takes into account both the amount or degree of modification indicated by the mask and the classification score obtained from the perturbation of the latent representation by the mask.

[0018] The above measures also involve scaling the mask to the spatial resolution of the input image to obtain an interpretable mask. This scaling can be achieved through various scaling techniques, including upsampling. For example, bilinear upsampling can be used to obtain a spatial resolution that matches the spatial resolution of the input image. Other techniques for scaling masks also exist and can be used, such as a series of transposed convolutions or deconvolutions.

[0019] The above measures also involve outputting an interpretability mask. An interpretability mask may also be referred to as a saliency mask or saliency map. The interpretability mask can be output in a sensorily perceptible form (such as via a display) that allows the user to see it. In some embodiments, the interpretability mask is reversed before being displayed to show a more intuitive image. In some embodiments, the interpretability mask can be output to a processor subsystem (such as an external processor subsystem) or to a function or computer program within the same system in which the method is implemented, such as for use in an automated system. In some embodiments, the interpretability mask is output to an external server or cloud-based storage device, or to a remote database.

[0020] The inventors have discovered that determining the interpretability mask, as described in the independent claim, produces a mask that provides improved interpretability to the trained neural network. By considering the amount or degree of modification applied to the perturbed latent representation, and the resulting classification score of the perturbed latent representation, smaller but important regions or features of the input image can be identified as primarily responsible for the resulting classification.

[0021] Specifically, adversarial solutions can be avoided or reduced by applying modifications to the latent representations of the inner layers of a trained neural network. An adversarial solution is one where the mask is small and the change to the input image is essentially minimal (e.g., almost imperceptible), yet still leads to a large change in the model's classification output. Since the resolution of the latent representations of the inner layers of a trained neural network is typically coarser than the resolution of the input image, adversarial solutions are less likely to be translated. Furthermore, adversarial attacks can often be generated by blocking individual pixels. Such adversarial attacks can be avoided or mitigated by blocking more complex concepts, as achieved by applying masks to the latent representations.

[0022] Specifically, by identifying the interpretability mask described in the independent claim, a user can determine whether the trained neural network has learned as much as he or she expects. For example, if the interpretability mask reveals that the trained neural network relies heavily on detecting grass in order to classify horses, the user might conclude that the trained neural network is not yet sufficiently trained and may require a more diverse training dataset. Therefore, the interpretability map can also reveal hidden biases and can prompt the user to address such biases by retraining or further developing the neural network.

[0023] This optimization also provides a quantitative measure of the importance of the interpretation identified by the mask. For example, the difference between the classification score of the perturbed latent representation and the classification score of the input image can quantitatively describe the relevance of the interpretation, with respect to the degree or amount of modification.

[0024] Optionally, updating the mask includes terminating the update operation if the classification score of the perturbed latent representation meets a classification threshold. For example, the classification threshold can be determined a priori as a fixed score value or a relative amount of the classification score of the input image. The optimization process can terminate when the classification score reaches the threshold. This avoids unnecessary further iterations of the optimization process, thereby reducing the runtime of the method and the power consumed by the system implementing the method. Since the threshold is predetermined, the user may be able to tailor the method to their needs—for example, to obtain a mask that highlights the deep neurons most responsible for activating a particular output class, the user could set the threshold to a value close to zero.

[0025] Optionally, updating the mask involves iteratively adjusting the mask values ​​for up to a predetermined number of iterations. The user can determine a priori the number of iterations by which the mask will be optimized. In some cases, this may refer to a maximum number of iterations, and the optimization process may terminate, for example, if the classification score reaches a classification threshold. However, it is also possible to set the predetermined number of iterations based on the classification score without a termination condition. By setting a predetermined number of iterations by which the mask will be optimized, it is possible to troubleshoot the system without entering an infinite loop. Additionally, if the mask optimization does not meet the necessary criteria within the predetermined number of iterations, a problem can be alerted to the user. Termination of optimization after the predetermined number of iterations can further enable the method to be performed in different ways, such as by extending the mask to previous layers.

[0026] Optionally, the method further includes: after updating the mask, determining whether the classification score of the perturbed latent representation meets a classification threshold. If the classification score of the perturbed latent representation does not meet the classification threshold, the method further includes: expanding the mask to additionally cover the input image with an additional layer representation at a previous layer of the trained neural network, and repeating the update step. In other words, it is possible that the initially selected layer does not include the filter most relevant to the classification. By expanding the mask to the previous latent representation (i.e., to the latent representation of the previous layer of the trained neural network), the filters of the previous layer can also be explored. In some cases, the mask can be expanded to additionally cover the previous latent representation after the update step has iterated for up to a predetermined number of iterations.

[0027] Optionally, initializing the mask involves assigning mask values ​​that replace the latent representations with zero-activation latent representations. That is, the mask can be initialized such that the activations of the layers generating the latent representations are effectively "blocked." During optimization, the mask can then be updated such that the activations of the layers generating the latent representations are gradually "unblocked," and an objective function can be formulated to maximize the classification score while maximizing the modified component of the objective function. Conceptually, this method identifies the smallest region in the input image that is most confident in generating the classification and should be "unblocked" (in other words, the largest complementary region remains "blocked").

[0028] Optionally, initializing the mask includes assigning mask values ​​that do not perturb the latent representation. In this case, the mask is initialized to be effectively "invisible," meaning it has no effect on any activations of layers in the trained neural network that produce the latent representation. During optimization, the mask can then be updated to gradually cover (or block or remove) portions of the latent representation, and an objective function can be formulated to minimize the classification score while also minimizing the modified components of the objective function.

[0029] Optionally, the method further includes: obtaining a dataset comprising multiple first input images; obtaining multiple latent representations corresponding to the multiple first input images; and clustering the first input images in the dataset into multiple clusters. For each cluster, the method may further include: determining an aggregate mask across the latent representations of the first input images within the clusters. The method may further include: determining the cluster to which the input image belongs from the multiple clusters; and initializing the mask as an aggregate mask corresponding to the determined cluster. By doing so, information from previous input samples can be used to improve the efficiency of the optimization process. Compared to initializing mask values ​​to be fully perturbed or fully unperturbed, an aggregate mask determined for the cluster to which the input image belongs can provide a better “starting point” for the mask, as discussed above. This can reduce the number of iterations required to achieve an appropriate threshold for the classification score, thus speeding up the method and reducing the required processing resources and power. Additionally, this can help avoid local optima, thus providing better optimal values.

[0030] Alternatively, an aggregate mask is determined by computing an average mask across the latent representations of the first input images in the cluster. Using the average mask provides an improved starting point for mask initialization by employing information already obtained through applying optimization to multiple similar input images. This reduces the number of iterations required to achieve an appropriate classification score threshold, thus speeding up the method and reducing the required processing resources and power.

[0031] Optionally, the method further includes: obtaining a dataset comprising a plurality of first input images; obtaining a plurality of corresponding latent representations of the plurality of first input images; and clustering the first input images of the dataset into a plurality of clusters. For each cluster, the method may be further configured to compute a set of at least one of the following: the most frequently masked neurons, the most frequently masked filters, and the most frequently masked layers of a trained neural network. The method further includes: determining the cluster to which the input image belongs among the plurality of clusters. A mask can then be determined to indicate modifications to a subregion of the latent representation, which is determined by the computed set corresponding to the cluster to which the input image belongs. By doing so, the optimization process can focus on a smaller region on which the mask is applied, thereby accelerating the optimization process. Information relating to the most frequently masked neurons, the most frequently masked filters, and the most frequently masked layers of the trained neural network of the first input image in the cluster can be additionally or alternatively used to initialize the mask, thereby providing a better starting point for the optimization process compared to the previously discussed fully perturbative or fully non-perturbative initialization.

[0032] Alternatively, the inner layers are convolutional layers in a trained neural network, such as those immediately preceding the first dense layer. Deeper convolutional layers in a trained neural network should strongly respond to concepts likely to be highly localized in the image. By using the last convolutional layer in the trained neural network, the receptive field of the inner layers will cover more (if not all) of the image and can determine information about highly localized and complex features. Additionally, adversarial solutions can be avoided, as these solutions are unlikely to be translated into deeper convolutional layers of the trained neural network.

[0033] Optionally, the input image includes an image-based representation of at least one of the following: image data, audio data, video data, text data, radar data, LiDAR data, and time-series data. This method is not limited to image data and, once represented in an image-based form, can be used with any data. This enables the method to be used in a wide range of applications, such as those involving the interpretability of classifying audio data.

[0034] Optionally, obtaining the input image includes acquiring an image from a sensor. For example, sensor data represented as an image can be obtained via an input interface. Such sensor data can include, for example, image / video / radar / LiDAR / ultrasound sensor data. The classification of sensor data is being used in an increasing number of critical applications, such as the optimization of automated manufacturing processes, medical image analysis, or diagnostics, making it increasingly important to explain how trained neural networks classify inputs.

[0035] Optionally, the output interpretability mask includes at least one of the following: outputting the interpretability mask to a user in a sensorily perceptible manner, and outputting the interpretability mask to a processor subsystem. For example, a user can examine the interpretability mask to understand how a trained neural network arrives at its classification. For example, this could indicate to the user whether the trained neural network has been adequately trained, or it could warn the user of hidden biases caused by an insufficient training dataset. The interpretability mask can additionally or alternatively be output to a processor subsystem that allows the interpretability mask to be used in automated processes. For example, the interpretability mask can be used in a manufacturing process involving automated optical quality inspection, and the location of highlighted features in the interpretability mask corresponding to an image of the product can determine how the product is routed or subsequently classified or processed.

[0036] Those skilled in the art will appreciate that two or more of the above embodiments, implementations and / or optional aspects of the present invention can be combined in any manner deemed useful.

[0037] Based on this specification, those skilled in the art can make modifications and variations to any system and / or any computer-readable medium, which correspond to modifications and variations of the described corresponding computer implementation methods. Attached Figure Description

[0038] These and other aspects of the invention will become apparent from the embodiments and the accompanying drawings, and will be further explained with reference to the embodiments and the accompanying drawings, in which the embodiments are described by way of example in the following description, and in the accompanying drawings: Figure 1 A system for determining an interpretability mask for classifying an input image by a trained neural network is shown; Figure 2a A schematic diagram of a trained neural network is shown, and Figure 2b A non-limiting example of a method for accessing a trained neural network is shown; Figure 3 A detailed example of a method for determining a mask by optimizing an objective function is shown; Figure 4 An illustrative process for clustering multiple input images to improve mask initialization is shown; Figure 5 A computer implementation method for determining an interpretable mask for classifying an input image by a trained neural network is shown. Figure 6 A computer-readable medium containing data is shown.

[0039] It should be noted that the accompanying drawings are purely schematic and not drawn to scale. In the drawings, elements corresponding to those already described may have the same reference numerals. Detailed Implementation

[0040] Figure 1 A system 100 is shown for determining an interpretability mask for classifying an input image by a trained neural network. Figures 2a to 5 The method is further described with further explanation.

[0041] System 100 may include a data interface 120 and a processor subsystem 140, which can communicate internally via data communication 124. Data interface 120 may be used to access a trained neural network 040 and, optionally, an input image in the form of data 060. In some embodiments, data interface 120 may also access a latent representation of the input image. For example, such as... Figure 1As shown, data interface 120 can provide access 122 to an external data storage device 020 that may include the data 040, 060. Alternatively, data 040, 060 can be accessed from an internal data storage device that is part of system 100. Alternatively, data 040, 060 can be received from another entity via a network. Generally, data interface 120 can take various forms, such as a network interface to a local area network or wide area network (e.g., the Internet), a storage interface to an internal or external data storage device, etc. Data storage device 020 can take any known and suitable form.

[0042] Processor subsystem 140 can be configured to obtain an input image and a latent representation of the input image during operation of system 100 and using data interface 120. In some embodiments, the latent representation of the input image can be obtained by feeding the input image into a trained neural network 040 and determining the latent representation of the input image output from the k-th layer of the trained neural network 040. In other embodiments, the latent representation of the input image may have already been determined and can be readily accessed, for example, in memory.

[0043] Processor subsystem 140 can be configured to initialize a mask used to indicate modifications to the latent representation. The mask can have the same spatial dimensions and depth as the latent representation. Several ways exist for initializing the mask, for example, by setting all values ​​of the mask so that the latent representation is completely unaffected (i.e., a mask without perturbative effects). Conceptually, this can be considered a “blank” mask. In another example, the mask can be initialized using values ​​that replace the latent representation with a zero-activation latent representation, which is the latent representation where the activation of said layer is blocked or hidden. In other examples, the mask can be initialized based on information from multiple input images to determine the set of initial mask values—referencing... Figure 4 Describe two such processes.

[0044] The processor subsystem 140 may be further configured to update the mask by iteratively adjusting the mask values ​​to optimize an objective function. The objective function includes a modification component and a classification score component. The modification component indicates the degree of modification indicated by the mask. In some embodiments, the modification component penalizes masks with large degrees or amounts of modification, for example, by using constraints. Such constraints may be hard constraints that ensure the degree or amount of modification is below a threshold amount, or they may be soft constraints that penalize higher amounts or degrees of modification. In other embodiments, the modification component rewards large degrees or amounts of modification. This will be referred to... Figure 3Let's discuss this in more detail. The classification score component can be determined as follows: applying the modification indicated by the mask to the latent representation (in other words, applying the mask to the latent representation) to obtain a perturbed latent representation; and determining the classification score of the perturbed latent representation based on the trained neural network 040. In other words, the perturbed latent representation is used as input to the next layer of the trained neural network 040, thereby continuing through the trained neural network 040 to obtain a classification score. This is in Figure 3 The diagram is shown and will be described in more detail in the relevant section of the specification. Classification scores can be obtained using any known method, such as the softmax probability layer of a neural network, the distance to the decision boundary of a support vector machine, etc. For example, a binary classifier can assign a confidence score *s* for identifying a class, such as a value between 0 and 1. If the input instance belongs to that class, the classification score can be equal to *s*, and if the input instance does not belong to that class, the value *s* can be reversed, for example, 1 - *s*. Classification scores can also be determined for multi-class classifiers; for example, some multi-class classifiers provide confidence scores for each possible class, in which case the classification score of the perturbed latent representation can be chosen as the classification score for the class to which the input image belongs. The classification score can also simply indicate whether the trained neural network classifies the perturbed latent representation into the same class as the input image.

[0045] Once the mask has been updated by optimizing the objective function, the processor subsystem 140 can be configured to scale the mask to the spatial resolution of the input image to obtain an interpretable mask. That is, the scaled mask can have the same spatial resolution as the input image. If the updated mask has a smaller spatial resolution than the input image, it can be scaled up using any known technique, such as bilinear upsampling or using a series of transposed convolutions or deconvolutions, although many alternative scaling methods are known and can be used to implement a scaled mask.

[0046] Processor subsystem 140 may be further configured to output an interpretable mask. In some embodiments, the processor subsystem may output the interpretable mask to another processing system, network entity, computer program, or function, thereby allowing the interpretable mask to be used, for example, in an automated process or to further refine a trained neural network 040. As an optional component, system 100 may include display output interface 160 or any other type of output interface for outputting the determined mask, and / or a perturbed instance obtained by applying the determined mask to input instance 142, to a rendering device (such as display 180). For example, display output interface 160 may generate display data 162 for display 180, which causes display 180 to present the determined mask and / or perturbed instance 142 in a sensory-perceptible manner, for example, as a visualization 182 on a screen. In some embodiments, processor subsystem 140 outputs the interpretable mask to a user in a sensory-perceptible manner, such as on a screen or via sound.

[0047] As an optional component, system 100 may include image input interface 150 or any other type of input interface for acquiring sensor data from a sensor, such as camera 170. Sensor data may be included in the input instance. For example, the camera may be configured to capture image data 172, and processor subsystem 140 may be configured to obtain input instance 152 from the image data 172 acquired via input interface 150.

[0048] System 100 may also include a communication interface 190 configured to communicate 194 with another system or entity, such as an actuator or automation system. For example, if system 100 is used to determine an interpretability mask that can then be used to determine how to handle a product in a manufacturing process, communication interface 190 may communicate with a system or actuator that controls the manufacturing process. Communication interface 190 may communicate internally with processor subsystem 140 via data communication 192. Communication interface 190 may be arranged for direct communication with another system, for example, using a USB, IEEE 1394, or similar interface. Communication interface 190 may also communicate via a computer network, such as a wireless personal area network, the Internet, an intranet, a LAN, a WLAN, etc. For example, communication interface 190 may include connectors, such as wireless connectors, Ethernet connectors, Wi-Fi, 4G or 4G antennas, ZigBee chips, etc., suitable for computer networks. Communication interface 190 may also be an internal communication interface, such as a bus, API, storage interface, etc.

[0049] Refer to Figure 2 and Figure 3This further clarifies the various details and aspects of the operation of System 100, including its optional aspects.

[0050] Generally, system 100 may be embodied as a single device or apparatus, or embodied in a single device or apparatus, such as a workstation (e.g., a workstation based on a laptop or desktop computer) or a server. The device or apparatus may include one or more microprocessors executing appropriate software. For example, a processor subsystem may be embodied by a single central processing unit (CPU), but may also be embodied by a combination or system of such CPUs and / or other types of processing units. The software may have been downloaded and / or stored in a corresponding memory, such as volatile memory like RAM or non-volatile memory like flash memory. Alternatively, the functional units of the system (e.g., data interfaces and processor subsystems) may be implemented in the device or apparatus as programmable logic, for example, as a field-programmable gate array (FPGA). Generally, each functional unit of the system may be implemented as a circuit. It should be noted that system 100 may also be implemented in a distributed manner, for example, involving different devices or apparatuses, such as a distributed server employing cloud computing.

[0051] The trained neural network 040 can be parameterized by a set of parameters. This set of parameters can include the weights of the nodes of the trained neural network. For example, the number of layers in the trained neural network can be at least 5 or at least 10, and the number of nodes and / or weights can be at least 1000 or at least 10000. Depending on the specific application, various known architectures for neural networks can be used. From a training efficiency perspective, it is beneficial to use a generative model suitable for gradient-based optimization, for example, a generative model that is continuous and / or differentiable in its set of parameters.

[0052] Figure 2a It shows that it can be made by Figure 1 The system 100 accesses trained neural networks (such as...) Figure 1 A schematic diagram of the trained neural network (040), and Figure 2b The diagram illustrates the access method. Figure 1 A non-limiting example of a trained neural network 040 method. The system for image classification is explained with reference to this example. Figure 2a The technology can be applied to various features, but as those skilled in the art will understand, this is not a limitation, as the technology is readily applicable to other kinds of instances. For example, various types of data, such as video data, audio data, sensor data (e.g., data obtained from sensors), radar / LiDAR data, and / or time-series data, can be provided in image-based representations.

[0053] For the purpose of explanation, Figure 2a The process illustrated in the diagram is a high-level and greatly simplified description of the classification of an input image in a convolutional neural network (CNN). As those skilled in the art will understand, other types of neural networks (such as deep neural networks) can also be used, and the following examples are non-limiting and merely exemplary.

[0054] CNNs can generally be conceptually divided into two components: a convolutional part, which includes at least one convolutional layer; and a dense (e.g., fully connected) part, which includes at least one dense layer.

[0055] Input image II 210 is input to Figure 1 In the trained neural network 040, more specifically, the first layer L1 212-1 is fed into filters (or in some cases, a single filter). Each filter can be convolved on the input image II 210 to produce an activation volume or "latent representation" LR1 222-1. Each "slice" of the activation volume can be the output of one of the filters. The latent representation LR1 222-1 can have a different spatial resolution than the input image II 210. The latent representation LR1 222-1 can then be used as input to the subsequent layer L2, 212-2. One or more filters of L2 212-2 can then be convolved on the latent representation LR1 222-1 to produce a second latent representation LR2, 222-2. This process can continue for the additional layers 212.

[0056] It should be noted that throughout the convolutional part of a CNN, the spatial relationship between the latent representation and the input image is preserved, even though the spatial dimensions of the latent representation may vary depending on the latent representation and the spatial dimensions of the input image.

[0057] After the last layer Lk 212-k of the convolutional part of the CNN (often referred to as the last convolutional layer, although in some embodiments the last layer Lk of the convolutional part is not necessarily a convolutional layer, as long as it preserves the spatial relationships discussed above), the resulting latent representation LRk is input into the first dense layer DL1, 214-1. The dense layer 214 can be a fully connected layer, where each neuron in the fully connected layer is connected to all neurons in another layer. After passing through the dense part of the CNN, the classification score CS 230 (or the set of classification scores) can be output.

[0058] In order to explain Figure 1To determine the output classification score for a trained neural network 040, it is useful to define an interpretable mask that highlights the regions or areas in the input image that contribute most to the output classification. This mask can be applied to the latent representation of the input image at a given layer of the trained neural network 040. In other words, to determine which regions(s) of the input image are most relied upon in determining its classification, a mask can be defined that modifies the latent representation of the input image to the minimum possible degree. Figure 1 The activation of filters within the trained neural network 040 simultaneously results in sufficiently different (or, in some embodiments, pre-specified) classification scores or outcomes. The mask may have the same spatial dimensions and depth as the latent representation. In some embodiments, the mask may be optimized such that its application to the latent representation results in a classification score below a threshold.

[0059] The mask indicates a modification that effectively ( wholly or partially) removes or prevents information from being passed forward to subsequent layers. That is, when a mask is applied to a latent representation, specific activations indicated by the mask are at least partially ignored in subsequent (i.e., deeper) layers. The classification score generated by such a perturbed latent representation may differ from that following... Figure 2a The classification score of the input image obtained during the process.

[0060] Figure 2b The diagram illustrates the perturbation of the latent representation and the subsequent classification score output. Figure 2b and Figure 2a The difference lies in that, at a specific layer (in this case, layer Lk 212-k), the resulting latent representation LRk222-k is perturbed by applying a mask in the mask application Ma 240. The perturbed latent representation LRk'222-k' is then used as input to the deeper layer 214.

[0061] exist Figure 2b In this model, the latent representation LRk 222-k is perturbed in the last layer of the convolutional part of the network, immediately before any non-convolutional layers (such as pooling layers, dense layers, fully connected layers, etc.). This is advantageous because it reduces or even prevents adversarial solutions from having an unwanted impact on the classification score. Adversarial solutions are features of an image that, despite being incorrect, still trigger classification with a high confidence level. Deeper layers of a CNN typically have a coarser spatial resolution than the input image and are therefore less likely to translate adversarial solutions. However, the latent representation to be perturbed does not need to be the output of the last layer of the convolutional part of the network. In some embodiments, the latent representation to be perturbed can be the output of one or more earlier layers. In these cases, the perturbed latent representation continues through the rest of the CNN, including any remaining layers of the CNN's convolutional part.

[0062] Figure 3 A detailed, but non-limiting, example of a computer implementation method for determining a mask by optimizing an objective function is shown, which can be, for example... Figure 1 The system 100 is implemented. For example, in... Figure 2a and 2b Similar to the system used for image classification, this was explained using... Figure 3 The technology can be applied to various features, but as those skilled in the art will understand, this is not a limitation, as the technology is readily applicable to other kinds of instances, such as various types of sensor data obtained from sensors.

[0063] Figure 3 The diagram shows a trained neural network TNN 340, trained to classify input images. The trained neural network TNN 340 is a neural network with a latent space (e.g., a space of latent features). For example, the trained neural network TNN 340 may include, for example, a deep neural network and / or a CNN.

[0064] Also shown is the input image II, 350 and the classification IIC, 351 of the input image based on the trained neural network. For example, classification IIC 351 can be obtained by evaluating the trained neural network TNN 340 on the input image II 350, or classification IIC 351 can be, for example, a ground truth classification of the training dataset from which the trained neural network TNN 340 was trained.

[0065] Input image II 350 can be fed into the first part TNNa 340a of the trained neural network. Latent representation LR 360 (such as...) Figure 2b LRk 222-k) can be identified as the output of the k-th filter of the trained neural network TNN 340. The first part TNNa340a of the trained neural network can be considered to include the input layer leading to the k-th layer of the trained neural network 040, and can be referred to as “part A” of the trained neural network 040. Then, the mask M 380 can be applied to the latent representation LR 360 in the mask application process 300 to obtain the perturbed latent representation PLR 370. The value of the mask M 380 is preferably defined by [0, 1].

[0066] Mask M 380 can be initialized in one of several ways. Some non-limiting examples of mask initialization include: initializing all values ​​of mask M 380 to have no perturbing effect on latent representation LR 360, such that the perturbed latent representation PLR 370 remains unchanged relative to latent representation LR 360; or initializing all values ​​of mask M 380 such that any activations from the layers of the output latent representation LR 360 are effectively removed, resulting in a perturbed latent representation PLR 370 that is therefore a "zero-activation" latent representation. (See also...) Figure 4 Here is a further example to describe mask initialization.

[0067] Then, the perturbed latent representation PLR 370 can be extended to the remaining layers TNNb340b of the trained neural network, also referred to as “part B” of the trained neural network 040. The trained neural network TNNb 340b can be considered to include layers of the trained neural network 040 that are not included in part A (i.e., not included in the trained neural network TNNa340a). In other words, the trained neural networks TNNa 340a and TNNb 340b, when concatenated, produce the trained neural network 040.

[0068] The trained neural network TNNb 340b can then determine the classification score of the perturbed latent representation PLR 370 used in the optimization process 320.

[0069] The optimization process Opt 320 represents the optimization of an objective function, such as a loss function or energy function, which includes modification components and classification score components. In some embodiments, such as when assigning values ​​in the range [0, 1] to a mask M, where a value of 0 indicates "blocking" (e.g., deleting) activation and a value of 1 has no perturbative effect, the objective function may take the form of Equation 1: [Equation 1] in: x It is the input image; λ 1 is the weighted term; It is a mask M The norm of the inverse; and Corresponding to the input image x Potential representation of disturbance Class of places q Classification score, perturbed latent representation It has been masked M Disturbance layer k The potential representation of the output.

[0070] The weighting term can be a hyperparameter. The weighting term can be a priori defined real-valued weighting term that balances the importance of the mask in minimizing the classification score with minimal modifications to the mask. This value can vary depending on the trained neural network and / or the input image.

[0071] The objective function can therefore include a modification component and a classification score component. In the example of Equation 1, the modification component is the first term (i.e., ), and the classification score component is determined by the second item (i.e., The regularization term is provided here as the norm of the inverse of the mask M, but other regularization terms, such as L1 or L2 regularization, can be used similarly.

[0072] An iterative process is used to determine a mask M 380 that minimizes the amount of modification (e.g., affecting as few regions of the input image / latent representation as possible) while also minimizing the classification score generated by the latent representation perturbed by the mask.

[0073] The optimization process Opt 320 identifies the modified mask M 380 from previous versions of mask M 380, and Figure 3 The iterative process shown continues. Specifically, in the mask application process Ma 300, the updated mask M 380 is applied to the latent representation LR 360, and the resulting perturbed latent representation PLR 370 is input into part B of the trained neural network O40 (i.e., the trained neural network TNNb 340b), and the classification score of the perturbed latent representation is determined. Then, the value of mask M 380 is modified accordingly, and the iterative process is repeated.

[0074] This iterative process can continue for up to a predetermined number of iterations, or until an optimal value is found (e.g., until the change in the output is sufficiently small), or until the classification score of the perturbed latent representation reaches a threshold classification value, or differs from the classification score of the input image by a specified amount or percentage. In some embodiments, the classification score of the input image can be obtained from a trained neural network 040 for comparison with the classification score generated during the mask update process. In some embodiments, it may be sufficient to iterate until the trained neural network 040 no longer determines the correct classification of the input image, i.e., until the classification score of another class becomes greater than the classification score of the input image. For example, in a simple multi-class image classification system, if the input image is determined to belong to the "cat" class, the iterative process can terminate when the perturbed latent representation is determined to belong to a different class (such as "fish"). However, these termination conditions are merely non-limiting examples, and other termination conditions may be employed.

[0075] The objective function to be optimized can depend to some extent on how the mask M 380 is initialized. For example, the objective function represented in Equation 1 is suitable in many cases (such as when initializing the mask) such that when the initialized mask is applied to the latent representation, the resulting perturbed latent representation is the same as the latent representation with the mask applied (i.e., the "invisible" mask).

[0076] However, if the mask is initialized in such a way as to block all activations from a layer (e.g., layer k) of the trained neural network 040—from which the latent representation to be perturbed outputs—the objective function can be formulated differently, for example as in Equation 2 below: [Equation 2] In Equation 2, the modification component is based on the norm of the mask M. Therefore, minimizing the modification component is equivalent to maximizing the degree or amount of activation that will be blocked by the mask M. In other words, the modification component is minimized when the value of M is close to 0. Values ​​of M close to 0 provide a larger perturbation effect (and values ​​of M close to 1 provide a smaller perturbation effect or no perturbation effect).

[0077] In this example, the objective function is optimized by minimizing the modification component—in other words, by maximizing the amount of activations blocked by the mask and maximizing the classification score component. Conceptually, this method involves starting with a mask that blocks all activations of the layer and gradually unblocking the activations. The goal is to determine the minimum amount of activations to be unblocked in order to obtain a classification. The minimum amount of activations to be unblocked is conceptually equivalent to the maximum amount of activations that are blocked by the mask. Therefore, in this case, the objective function can maximize the degree or amount of modification (thus minimizing the modification component as defined with respect to Equation 2) and also maximize the classification score component.

[0078] Although the 1-M norm is used in Equation 1 and the M norm in Equation 2, it is important to understand that these can be switched depending on how the mask value is defined. For example, if mask value 1 is interpreted as having no perturbation effect and mask value 0 as having a blocking effect, the expressions in Equations 1 and 2 above should remain unchanged. However, if mask value 0 is interpreted as having no perturbation effect and mask value 1 as having a blocking effect, the component expressions can be interchanged. In the former case, applying the mask may involve multiplying the mask by the latent representation. However, in the latter case, applying the mask may involve multiplying the mask inverse (1-M) by the latent representation.

[0079] These examples illustrate two “extreme” initializations—a first initialization in which no perturbation is performed; and a second initialization in which all activations are prevented. In the first of these initializations, the optimization process adds the modification indicated by the mask M while determining whether the classification score of the latent representation perturbed by the mask M is sufficiently low (according to at least one termination condition)—e.g., until the trained neural network can no longer satisfactorily determine the class. Conversely, in the second initialization described above, the optimization process “removes” the modification (i.e., gradually allows more activations to occur) while determining whether and to what extent the trained neural network classifies the perturbed latent representation appropriately.

[0080] In the first initialization example, termination conditions such as those previously described may involve whether the classification score component is below a threshold classification score, while in the second initialization example, termination conditions such as those previously described may instead involve whether the classification score exceeds a threshold classification score.

[0081] In some embodiments, the optimization process may preferably be configured to extend to previous layers of the trained neural network in certain situations (e.g., when the classification score component of the objective function does not meet a classification threshold). For example, for the first mask initialization method discussed earlier, this extension can be implemented if the classification score component is not lower than the classification threshold; and for the second mask initialization method discussed earlier, this extension can be implemented if the classification score component does not exceed the classification threshold.

[0082] If, after a predetermined number of iterations, the classification score components do not meet an appropriate threshold, the method can continue to expand the mask M380 to additionally cover the target area. Figure 1The mask is expanded to cover the latent representations of previous layers of the trained neural network 040. For example, if the latent representation of layer k was initially used, and after a predetermined number of iterations the classification score component does not meet the appropriate classification threshold, the mask is expanded to also cover the latent representation of layer k-1. Optionally, this process can be repeated until the classification score component of the objective function reaches (or exceeds) the classification threshold. In other words, the mask can be expanded to cover latent representations LRk and LR(k-1). If the classification score component still does not meet the classification threshold, the mask can be expanded to cover latent representations LRk, LR(k-1), and LR(k-2), and so on, until the classification threshold is met. It should be understood that if the spatial relationship with the input image is still preserved, it is also possible to expand the mask to cover subsequent or later layers. However, preferably, the initially selected layer should be a deep layer in order to identify features that are highly localized in space, contrasting with features (such as texture) that can be widely found on the image and are typically represented in lower (e.g., shallower) layers.

[0083] In some embodiments, as described above, the mask is extended to cover multiple layers, and the objective function is optimized on said multiple layers. In this case, the mask spans these multiple layers throughout the optimization process and is updated accordingly. More specifically, in the example where the mask is extended across two layers, a first portion of the mask is applied to the latent representation of the earlier of the two layers (e.g., layer k-1), and the perturbed earlier latent representation continues through layer k (i.e., the subsequent layer) to produce the next latent representation, i.e., the output from layer k. Then, a second portion of the mask is applied to this next latent representation to obtain a perturbed next latent representation, which can then continue through... Figure 1 The rest of the trained neural network 040 is used to obtain classification scores. During the optimization of the objective function, the values ​​in both parts of the mask can be updated / changed.

[0084] In some embodiments, extending the mask across multiple layers includes: first determining an updated mask M for a specific layer (e.g., layer k) by optimizing an objective function. k Despite optimization, the classification score components may not be at the appropriate threshold, so the method can continue to the previous layer. In this case, the method continues by selecting layer k-1 (or in some cases, simply selecting an earlier layer, although it is generally preferred to select the layer immediately preceding the initially used layer). The method is repeated as described above, and an updated mask M for the potential representation of layer k-1 can be determined. k-1 The mask can then be M. k and M k-1The combination of [masks] can then be used to determine the classification score by applying mask M. k-1 (i.e., the optimal mask for the latent representation of layer k-1) to obtain the perturbed latent representation of layer k-1; input the perturbed latent representation of layer k-1 into Figure 1 The trained neural network 040 then proceeds to subsequent layers (in this case, layer k); the latent representation of layer k is obtained; and the mask M is used to... k To perturb the latent representation of layer k; and to continue by feeding the perturbed latent representation of layer k into the subsequent layer of the trained neural network (in this case, layer k+1). Figure 1 The remainder of the trained neural network 040. If the classification score obtained using these two masks meets the necessary condition, the combination of mask Mk-1 and mask Mk is scaled up using any of the aforementioned known means to obtain an interpretability mask. However, if the obtained classification score still does not meet the necessary condition, the method is repeated at the previous layer k-2, and so on, until the obtained classification score meets the requirement.

[0085] Figure 4 An illustrative process is shown for clustering multiple input images to improve mask initialization.

[0086] from Figure 4 Starting with a), system 100 accesses multiple input images I1, I2, I3, I4, I5, and I6 (collectively referred to as 410). Although Figure 4 a) in the diagram describes six input images, but it should be understood that this is not a limiting number—any number of input images can be used—and they were chosen simply for ease of illustration.

[0087] The corresponding latent representation of the input image 410 can be obtained, for example, by system 100. That is, system 100 can be configured to... Figure 1 The trained neural network 040 is applied to each of the multiple input images I1-I6 410 in order to Figure 1 At a specific layer (e.g., layer k) of the trained neural network 040, corresponding latent representations R1 to R6 (collectively referred to as 420) are obtained. In some embodiments, the latent representations have been previously determined by system 100 or by another system or server, and can be accessed by system 100 without reapplying the trained neural network. Thus, latent representation R1 is the latent representation of the input image I1 at layer k of the trained neural network 040, and latent representation R2 is... Figure 1 The latent representation of the input image I2 at layer k of the trained neural network 040, etc.

[0088] Then, the multiple input images I1 to I6 and their corresponding latent representations R1 to R6 are grouped into clusters 430. The clustering of the input images 410 can be based on deep feature clustering or any known clustering method. This is in... Figure 4 As illustrated in b), input images I1 and I5 and their corresponding latent representations R1 and R5 are grouped in a first cluster 430-1, input images I4 and I6 along with their corresponding latent representations R4 and R6 are grouped in a second cluster 430-2, and input images I2 and I3 along with their corresponding latent representations R2 and R3 are grouped in a third cluster 430-3. Therefore, clusters can be formed for input images with similar characteristics. An aggregation mask can then be determined for each cluster.

[0089] Several methods exist for determining the aggregation mask 440. The first such method is based on determining the average mask.

[0090] Average mask methods include: based on... Figure 3 The described method determines a mask for each input image in a cluster. Specifically, to find the average mask 440-1 for cluster 1 430-1, interpretability masks for each of input images I1 and I5 can be determined using latent representations R1 and R5, respectively. The average mask M1 440-1 for cluster 1 is then determined by calculating the average of the masks determined for input image I1 and input image I5. In other words, the average mask for a cluster is determined by calculating the average of the masks calculated for each input image in the cluster.

[0091] By repeating this process for each of clusters 2 and 3, masks M2 and M3 can be determined respectively. In this first method, the average mask of the cluster is the aggregate mask.

[0092] A second method for determining the aggregation mask is based on identifying which filters, layers, and / or neurons in the cluster are most frequently masked. These filters, layers, and / or neurons are then reflected in a mask M1 for the associated cluster.

[0093] In this context, the method could include: computing across the cluster the most frequently masked layers(s), neurons(s), and / or filters(s). This information can then be used to initialize masks to indicate modifications to corresponding subsets of the latent representation, for example, to prevent (or partially prevent) activation of one(s) filters(s) in a particular layer. This can also reduce the number of iterations required, thus speeding up the method and reducing processing power requirements.

[0094] Both of the above methods can be used to determine the aggregation mask for clustering. Upon receiving an input image (such as input image II 350), the method may further include: determining the cluster to which input image II 350 belongs, or is most closely similar to it, and initializing mask M 380 as an aggregation mask for said cluster. Determining the cluster to which input image II 350 belongs can be accomplished by any known means, such as feature comparison, feature clustering, simple comparison, etc. For example, a distance metric can be used to identify the nearest neighbors between features of the input image and clusters. The distance metric can be, for example, L2 distance, cosine similarity, or any other distance metric known in the art. This can reduce the number of iterations required for the classification score components of the objective function to satisfy the previously described necessary conditions. Reducing the number of iterations will speed up the method and reduce processing power and resources.

[0095] In some embodiments, information about the most frequently masked layers(s), neurons(s), and / or filters(s) can be additionally or alternatively used to limit the scope of the update operation. This information can be used to limit the scope of the update process in which masks are initialized according to any described mask initialization. For example, information indicating the most frequently masked layers in a cluster can be used to determine which layer of the trained neural network should be obtained and perturbated for the latent representation. In some examples, information indicating the most frequently masked filters(s) in a particular layer can be used to restrict the optimization of the objective function to the filters(s) in that particular layer. By restricting the layers, filters, and / or neurons on which the mask is optimized, the process can be faster and can reduce processing power and resource requirements. For example, if only a few filters are masked in a particular cluster, the optimization can be restricted to those filters. In another example, if certain neurons or layers in a particular cluster are never masked, the optimization can avoid those neurons or layers.

[0096] Although this description has so far addressed images, other forms of data can also be represented in an image-based manner and can be used in the methods described above. For example, an audio file can be represented using an image-based representation; that is, the audio file can be in the form of an image file, and the methods described above can be used for the image representation of the audio file in the manner described. Similarly, video data, sensor data, radar data, time-series data, and so on.

[0097] In some embodiments, the input image can be obtained from sensors such as cameras, radar / LiDAR systems, etc. Sensor data can be represented as images and used as input to a trained neural network. In some embodiments, the input image can be obtained from internal or external storage devices via a server, downloaded from an external source, or obtained in any other way.

[0098] In some embodiments, the interpretable mask obtained via the method described above can be output to the user in a sensorily perceptible manner. Before being displayed on a monitor, the interpretable mask can be reversed to highlight areas of the input image that the method deems best explain why the input image is classified into a particular category (e.g., class). q For example, an interpretability map can be displayed on the display of system 100, allowing the user to see the regions of the input image that the trained neural network 040 relies on most when determining its classification.

[0099] In some embodiments, the interpretability mask is output to another function within system 100 or to another system. In these cases, interpretability may optionally be reversed. The interpretability mask can be used by subsequent functions or programs (e.g., in a processor subsystem) for a variety of purposes, such as automated optical inspection, etc.

[0100] For example, an interpretable mask can be used in a manufacturing process that includes an optical quality inspection system. In this case, an interpretable mask can be determined for an input image that includes measurements of a product produced during the manufacturing process, and this interpretable mask can be classified by a trained neural network as "OK" (i.e., meets a certain quality standard) or "NOK" (i.e., fails to meet the quality standard). The input image can be fed into a trained neural network, and the interpretable mask can be obtained based on the described method. The interpretable mask can then be used to appropriately classify, guide, and / or process the product, indicating the areas of the input image that the trained neural network relies on most when classifying the input image into a particular class. For example, if the interpretable mask highlights a crack in the upper left corner of the input image, it can represent a broken shell in the product. The product can then be guided to an area in the manufacturing process where the broken shell problem can be addressed. If the interpretable mask highlights an area known to be a weld point, which is the main reason the input image (and therefore the product) is classified as "NOK," the product can be guided to different parts of the manufacturing process, and so on.

[0101] As another example within the manufacturing process, if interpretive masks corresponding to many products in a batch consistently indicate a specific aspect of a product as "non-conforming," the manufacturing process can automatically adjust at least one associated parameter. For instance, if the interpretive masks for a batch of products consistently indicate a weld as "non-conforming," the parameters of the welding process can be adjusted. In some cases, such problems can be alerted to human operators, who can then make the necessary adjustments accordingly.

[0102] Other examples of using interpretability masks determined by the methods described herein include processes in the field of autonomous vehicles. For example, an autonomous vehicle may acquire images of the road ahead via one or more sensors, such as cameras. A trained neural network, such as trained neural network 040, may classify the image as “acceptable” (e.g., no anomaly detected) or “unacceptable” (e.g., an anomaly detected), or may classify the image into another category. For example, an autonomous vehicle hazard detection system may use an interpretability mask as determined by the methods described herein to ensure that the hazard detection system operates as intended. If the interpretability mask detects an anomaly (such as a speed bump or pothole) along the bottom of the image, this may indicate a speed bump or pothole in the road, such as a road where speed bumps and potholes are typically expected. If the interpretability mask instead detects a speed bump or pothole in the top of the image (e.g., in the sky), it can be determined that the hazard detection system is not operating as intended and may trigger a warning.

[0103] Figure 5 A block diagram is shown of a computer implementation method 500 for determining an interpretability mask used to indicate an interpretability metric. Method 500 may correspond to... Figure 1 The operation of system 100. However, this is not a limitation, because method 500 can also be implemented using another system, apparatus or device.

[0104] Method 500 may include, in an operation entitled "Accessing a Trained Neural Network", accessing, 510, a trained neural network, such as trained neural network 040. Method 500 may further include, in an operation entitled "Obtaining an Input Image, Classification, and Latent Representation", obtaining, 520, an input image, a classification of the input image, and a latent representation of the input image at a layer of the trained neural network. In other words, the latent representation may be an "activation volume" or output of a layer (such as layer k) of the trained neural network. In some embodiments, the latent representation is obtained by feeding the input image into the trained neural network and obtaining the latent representation of layer k after the input image has been processed by layers 0 through k. In other words, the latent representation can be obtained by applying the trained neural network to the input image. In some embodiments, the latent representation may have been previously determined and stored by the same system 100 or by another system or external server, and therefore may be accessible only by system 100. In some embodiments, a classification score of the input image for the obtained classification (i.e., class) may also be obtained. Method 500 may further include: in an operation entitled “Initializing the Mask”, initializing 530 the mask to be updated. This mask indicates a modification to the latent representation. The mask can be initialized, for example, by using any method of mask initialization described herein. For example, the mask can be initialized such that it initially has no perturbative effect on the latent representation, or the mask can be initialized such that activation of layer k of the trained neural network 040 is effectively prevented. In another example, it can be used from a clustering process (such as those described herein). Figure 4 The method uses information obtained in the described process to initialize the mask, such as by using an average masking technique, or by identifying the most frequently masked neurons(one or more), filters(one or more), and / or layers(one or more). The method may further include updating the 540 mask in an operation entitled “Update Mask”. The 540 update process may include optimizing a 542 objective function, which includes a modification component and a classification score component. The modification component indicates the degree of modification indicated by the mask of the latent representation. That is, the degree of modification may correspond to the relative area of ​​the latent representation to which the modification has been applied (or, correspondingly, to the relative area of ​​the input image) (e.g., percentage or scale). The degree of modification may also correspond to a measure of how much the latent representation has been perturbed, in order to represent the magnitude of the modification. That is, the degree of modification may correspond to how much the latent representation has been modified (spatially), but in some cases, for example, if many neurons are only slightly modified, this may be replaced by, for example, a large modification to a single neuron (in which case, the magnitude of the modification is primarily considered).

[0105] Each iteration of the optimization of the objective function 542 may include: modifying the value of the mask 544; perturbing the latent representation 546 using the mask (whose value has been modified); and determining the classification score of the perturbed latent representation 548 as previously described.

[0106] Once the mask has been updated, the method may further include scaling the mask by 550 to the spatial resolution of the input image in an operation entitled "Scaling the Mask" to obtain an interpretable mask. Scaling the mask by 550 can be done in any known manner, including bilinear upsampling, using a series of transposed convolutions, using deconvolution, deconvolution, or via any other means. The method may further include outputting an interpretable mask by 560 in an operation entitled "Outputting the Mask". This interpretable mask may be output in a manner perceptible to the user, such as being displayed on a display, or may be output to a processor subsystem, an external system or server, additional functionality, or a computer program. In some embodiments, the method may optionally include reversing the interpretable mask before or after scaling the mask, or after outputting the interpretable mask.

[0107] What will be understood is, generally speaking Figure 5 The operations of method 800 can be performed in any suitable order (e.g., sequentially, simultaneously, or a combination thereof), and these steps are subject to a specific order that must be followed, for example, due to input / output relationships, where applicable.

[0108] One or more methods can be implemented on a computer as a computer-implemented method, dedicated hardware, or a combination of both. For example... Figure 6 As illustrated, instructions for a computer (e.g., executable code) can be stored on a computer-readable medium 600, for example, in the form of a series of machine-readable physical tags 610, and / or as a series of elements with different electrical (e.g., magnetic or optical) properties or values. The executable code can be stored in a transient or non-transitory manner. Examples of computer-readable media include memory devices, optical storage devices, integrated circuits, servers, online software, etc. Figure 6 An optical disc 600 is shown. Alternatively, the computer-readable medium 600 may include temporary or non-temporary data 610 representing an interpretability mask obtained by any system or method described in this specification.

[0109] Examples, embodiments, or optional features (whether or not they are indicated as non-limiting) should not be construed as limiting the claimed invention.

[0110] It should be noted that the above embodiments are illustrative rather than limiting of the invention, and many alternative embodiments will be able to be devised by those skilled in the art without departing from the scope of the appended claims. Any reference numerals placed between parentheses in the claims should not be construed as limiting the claims. The use of the verb “comprising” and its variations does not exclude the presence of elements or stages other than those stated in the claims. The article “a” or “an” preceding an element does not exclude the presence of a plurality of such elements. Expressions such as “at least one” preceding a list or group of elements indicate a selection of all elements or any subset of elements from that list or group. For example, the expression “at least one of A, B, and C” should be understood to include: only A, only B, only C, both A and B, both A and C, both B and C, or all of A, B, and C. The invention can be implemented by means of hardware comprising several different elements, and by means of a suitably programmed computer. In device claims enumerating several components, several of these components may be embodied by the same hardware item. The mere fact that certain measures are recited in mutually different dependent claims does not indicate that a combination of these measures cannot be used advantageously.

Claims

1. A computer implementation method for determining an interpretable mask for classifying an input image by a trained neural network, the trained neural network being configured to determine a latent representation of the input image at an inner layer of the trained neural network and to determine a classification of the input image based on the latent representation of the input image, the classification determination including determining a classification score for the input image, the method comprising: - Access the trained neural network; - Obtain the input image and its latent representation; - Initialize the mask used to indicate modifications to the underlying representation: - The mask is updated by iteratively adjusting its values ​​to optimize an objective function, which includes: i) a modification component indicating the degree of modification indicated by the mask; and ii) a classification score component determined as follows: ○ Apply the modifications indicated by the mask to the latent representation to obtain the perturbed latent representation; ○ The classification score is determined based on the perturbed latent representation using a trained neural network; If a value in the range [0, 1] has been assigned to the mask, where a value of 0 indicates that activation is prevented and a value of 1 has no perturbation effect, then the objective function takes the form of Equation 1: ,in x It is the input image. λ 1 is the weighted term. It is a mask M The norm of the inverse, and Corresponding to the input image x Potential representation of disturbance Class of places q Classification score, perturbed latent representation It has been masked M Disturbance layer k The latent representation of the output; wherein if the mask has been initialized in such a way as to prevent all activations from the layers of the trained neural network—from which the latent representation to be perturbed is output—then the objective function takes the form of Equation 2: ; - Scale the mask to the spatial resolution of the input image to obtain an interpretable mask; and - Output interpretability mask.

2. The method of claim 1, wherein, The mask update process includes ending the update operation if the classification score of the perturbed latent representation exceeds a classification threshold.

3. The method of claim 1 or claim 2, wherein, Updating the mask involves iteratively adjusting the mask values ​​up to a predetermined number of iterations.

4. The method according to any of the preceding claims, further comprising: - After updating the mask, determine whether the classification score of the perturbed latent representation meets the classification threshold; as well as - If it is determined that the classification score does not meet the classification threshold: ○ then extends the mask to additionally overlay an additional latent representation of the input image on previous layers of the trained neural network, and ○ Repeat the update steps.

5. The method of any preceding claim, wherein, Initializing the mask includes: assigning mask values, which replace latent representations with zero-activation latent representations.

6. The method of any one of claims 1 to 4, wherein, Initializing the mask involves assigning mask values ​​that do not have a perturbative effect on the underlying representation.

7. The method according to any one of claims 1 to 4, further comprising: - Obtain a dataset that includes multiple initial input images; - Obtain the corresponding multiple latent representations of the multiple first input images; - Cluster the first input images of the dataset into multiple clusters; - For a cluster of multiple clusters, determine the aggregation mask across the latent representations of the first input image in the cluster; - Determine the cluster to which the input image belongs from multiple clusters; as well as - Initialize the mask as an aggregate mask corresponding to the determined cluster.

8. The method according to claim 7, wherein, The aggregate mask is determined by computing the average mask across the latent representation of the first input image in the cluster.

9. The method according to any of the preceding claims, further comprising: - Obtain a dataset that includes multiple initial input images; - Obtain the corresponding multiple latent representations of the multiple first input images; - Cluster the first input images of the dataset into multiple clusters; - Based on the first input image in the cluster, for each cluster, compute at least one of the following groups of the trained neural network: the most frequently masked neuron, the most frequently masked filter, and the most frequently masked layer. as well as - Determine the cluster to which the input image belongs among multiple clusters; The mask update process includes: determining a mask to indicate modifications to sub-regions of the latent representation, and determining the sub-regions by means of a set corresponding to the cluster to which the input image belongs.

10. The method of any preceding claim, wherein, The inner layers are convolutional layers of a trained neural network, such as convolutional layers immediately preceding the first dense layer.

11. The method of any preceding claim, wherein, The input image includes at least one image-based representation of the group consisting of: image data, audio data, video data, text data, radar data, LiDAR data, and time-series data.

12. The method of any preceding claim, wherein, Obtaining the input image involves acquiring the input image from the sensor.

13. The method of any preceding claim, wherein, The output interpretability mask includes at least one of the following: outputting the interpretability mask to a user in a sensor-perceptible manner, and outputting the interpretability mask to a processor subsystem.

14. A computer-readable storage medium comprising temporary or non-temporary data representing a computer program, said computer program comprising instructions for causing a processor system to perform the method of any one of claims 1 to 13.

15. A system for determining an interpretability mask for classifying an input image by a trained neural network, the trained neural network being configured to determine a latent representation of the input image at an inner layer of the trained neural network and to determine a classification of the input image based on the latent representation of the input image, the classification determination including determining a classification score for the input image, the system comprising: - Data interface for accessing neural networks and obtaining input images and their potential representations; - The processor subsystem is configured to: ○ Obtain the input image and its latent representation; ○ Initialize the mask used to indicate modifications to the underlying representation; ○ The mask is updated by iteratively adjusting its values ​​to optimize an objective function, which includes: i) a modification component indicating the degree of modification indicated by the mask; and ii) a classification score component determined as follows: ▪ Apply the modifications indicated by the mask to the latent representation to obtain the perturbed latent representation; ▪ Determine classification scores for the perturbed latent representations based on a trained neural network; If a value in the range [0, 1] has been assigned to the mask, where a value of 0 indicates that activation is prevented and a value of 1 has no perturbation effect, then the objective function takes the form of Equation 1: ,in x It is the input image. λ 1 is the weighted term. It is a mask M The norm of the inverse, and Corresponding to the input image x Potential representation of disturbance Class of places q Classification score, perturbed latent representation It has been masked M Disturbance layer k The latent representation of the output; wherein if the mask has been initialized in such a way as to prevent all activations from the layers of the trained neural network—from which the latent representation to be perturbed is output—then the objective function takes the form of Equation 2: ; ○ Scale the mask to the spatial resolution of the input image to obtain an interpretable mask; and ○ Output interpretability mask.

Images