Graph cutting for interpretability

Abstract

A processor-implemented method for implementing graph segmentation for interpretability using an artificial neural network (ANN) includes receiving an input via the ANN. The input is represented as a graph. The graph includes nodes connected by edges. The ANN determines a graph cut between a source node and a sink node associated with the input by solving a quadratic process with equality constraints. The ANN processes a subset of the input based on the graph segmentation to generate a prediction.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to Greek Patent Application No. 20220100795 filed on September 28, 2022 and entitled “GRAPH CUTS FOR EXPLAINABILITY”, the disclosure of which is expressly incorporated by reference in its entirety. Technical Field

[0003] Aspects of the present disclosure relate generally to artificial neural networks, and more particularly to graph cuts for interpretability. Background Art

[0004] An artificial neural network may include an interconnected group of artificial neurons (e.g., a neuron model). An artificial neural network may be a computing device or represented as a method to be performed by a computing device. A convolutional neural network (CNN) is a type of feedforward artificial neural network. A convolutional neural network may include a collection of neurons, each of which has a receptive field and collectively spells out an input space. Convolutional neural networks (CNNs), such as deep convolutional neural networks (DCNs), have numerous applications. Specifically, these neural network architectures are used in various technologies, such as image recognition, speech recognition, acoustic scene classification, keyword detection, autonomous driving, and other classification tasks.

[0005] Neural networks also have many applications in image-based video or video stream processing, such as human pose estimation, object detection, semantic segmentation, and video compression and denoising. Unfortunately, such video processing is computationally intensive, which results in a lot of time and energy consumption. Summary of the invention

[0006] The present disclosure is set out in the independent claims respectively. Some aspects of the present disclosure are described in the dependent claims.

[0007] In some aspects of the present disclosure, a processor-implemented method includes receiving an input via an artificial neural network (ANN). The processor-implemented method also includes representing the input as a graph including a plurality of nodes connected by edges. The processor-implemented method also includes determining, via the ANN, a graph cut between a source node and a sink node associated with the input by solving a quadratic process with an equality constraint. The processor-implemented method also includes processing, via the ANN, a subset of the input based on the graph cut to generate a prediction.

[0008] Aspects of the present disclosure relate to an apparatus. The apparatus includes a component for receiving an input via an artificial neural network (ANN). The apparatus also includes a component for representing the input as a graph, the graph including a plurality of nodes connected by edges. The apparatus also includes a component for determining a graph cut between a source node and a sink node associated with the input via the ANN by solving a quadratic process with an equality constraint. The apparatus also includes a component for processing a subset of the input based on the graph cut via the ANN to generate a prediction.

[0009] In some aspects of the present disclosure, a non-transient computer-readable medium having a program code recorded thereon is disclosed. The program code is executed by a processor and includes a program code for receiving an input via an artificial neural network (ANN). The program code also includes a program code for representing the input as a graph, the graph including a plurality of nodes connected by edges. The program code also includes a program code for determining a graph cut between a source node and a sink node associated with the input via the ANN by solving a quadratic process with an equality constraint. The program code also includes a program code for processing a subset of the input based on the graph cut via the ANN to generate a prediction.

[0010] Aspects of the present disclosure relate to a device. The device has at least one memory and one or more processors coupled to the at least one memory. The processor is configured to receive an input via an artificial neural network (ANN). The processor is also configured to represent the input as a graph comprising a plurality of nodes connected by edges. The processor is also configured to determine a graph cut between a source node and a sink node associated with the input via the ANN by solving a quadratic process with an equality constraint. The processor is also configured to process a subset of the input based on the graph cut via the ANN to generate a prediction.

[0011] Additional features and advantages of the present disclosure will be described below. It will be appreciated by those skilled in the art that the present disclosure can be easily used as a basis for modifying or designing other structures for implementing the same purpose as the present disclosure. It will also be appreciated by those skilled in the art that such equivalent constructions do not depart from the teachings of the present disclosure as set forth in the appended claims. The novel features considered to be characteristic of the present disclosure, both in terms of its organization and method of operation, together with further objects and advantages, will be better understood when the following description is considered in conjunction with the accompanying drawings. However, it is to be expressly understood that each of the accompanying drawings is provided for illustration and description purposes only and is not intended to be a definition of limitations of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] The features, nature and advantages of the present disclosure will become more apparent when the detailed description set forth below is read in conjunction with the accompanying drawings, in which like reference numerals are correspondingly identified throughout.

[0013] Figure 1 An example implementation of a neural network using a system on a chip (SOC) including a general purpose processor in accordance with certain aspects of the present disclosure is illustrated.

[0014] Figure 2A , Figure 2B and Figure 2C is a diagram illustrating a neural network according to aspects of the present disclosure.

[0015] Figure 2D is a diagram illustrating an exemplary deep convolutional network (DCN) according to aspects of the present disclosure.

[0016] Figure 3 is a block diagram illustrating an exemplary deep convolutional network (DCN) according to aspects of the present disclosure.

[0017] Figure 4 is a block diagram illustrating an exemplary software architecture that can modularize artificial intelligence (AI) functionality according to aspects of the present disclosure.

[0018] Figure 5A are diagrams illustrating example cuts of example graphs according to aspects of the present disclosure.

[0019] Figure 5B is a diagram illustrating an example graph for illustrating an example minimum st cut problem according to aspects of the present disclosure.

[0020] Figure 6 is a block diagram illustrating an example architecture for determining graph cuts in accordance with aspects of the present disclosure.

[0021] Fig. 7A is an example of various aspects of the present disclosure Figure 6 Block diagram of a graph partitioning module for partitioning a graph.

[0022] Figure 7B is a diagram illustrating example pseudo code for a partition map according to aspects of the present disclosure.

[0023] Fig. 8A is a diagram illustrating example object-centric feature representations according to aspects of the present disclosure.

[0024] Figure 8B is a diagram illustrating example pseudo code for object-centric representation according to aspects of the present disclosure.

[0025] Figure 8C is a diagram illustrating example pseudo code for object-centric representation according to aspects of the present disclosure.

[0026] Fig. 9is a flow chart illustrating an example processor-implemented method of implementing graph cuts for interpretability using an artificial neural network in accordance with aspects of the present disclosure. DETAILED DESCRIPTION

[0027] The specific embodiments described below in conjunction with the accompanying drawings are intended as descriptions of various configurations and are not intended to represent the only configurations in which the described concepts can be practiced. In order to provide a comprehensive understanding of the various concepts, the specific embodiments include specific details. However, it is obvious to those skilled in the art that these concepts can be practiced without these specific details. In some instances, in order to avoid obscuring such concepts, well-known structures and components are shown in block diagram form.

[0028] Based on the teachings, those skilled in the art will recognize that the scope of the present disclosure is intended to cover any aspect of the present disclosure, whether the aspect is implemented independently of any other aspect of the present disclosure or implemented in combination with any other aspect. For example, a device or a method can be implemented using any number of aspects described. In addition, the scope of the present disclosure is intended to cover such devices or methods that are practiced using other structures, functionality, or structures and functionality that are supplementary to or different from the various aspects of the present disclosure described. It should be understood that any aspect of the present disclosure disclosed can be embodied by one or more elements of the claims.

[0029] The word “exemplary” is used to mean “serving as an example, instance, or illustration.” Any aspect described as “exemplary” is not necessarily to be construed as preferred or advantageous over other aspects.

[0030] Although specific aspects are described, numerous variations and permutations of these aspects fall within the scope of the present disclosure. Although some benefits and advantages of preferred aspects are mentioned, the scope of the present disclosure is not intended to be limited to specific benefits, uses or purposes. On the contrary, various aspects of the present disclosure are intended to be broadly applicable to different technologies, system configurations, networks and protocols, some of which are illustrated as examples in the accompanying drawings and the following description of preferred aspects. The specific embodiments and drawings are merely illustrative of the present disclosure and are not limiting. The scope of the present disclosure is defined by the appended claims and their equivalents.

[0031] As discussed above, neural networks also have many applications in image-based video or video stream processing, such as human pose estimation, object detection, and semantic segmentation, as well as video compression and denoising. Unfortunately, such video processing is computationally intensive, which results in a lot of time and energy consumption.

[0032] Interpretability is an area of ​​growing interest in artificial intelligence. One motivation for interpretability is to increase the reliability of predictions generated in deep neural network models. Interpretability involves determining a subset of inputs that are more important for producing an output prediction. Therefore, by understanding the interpretability of a model, image processing optimizations can be achieved, for example, by being able to process only a subset of pixels in an image rather than the entire image to determine an output prediction.

[0033] Graph cut techniques have been applied to various problems in computer vision and graphics. Graph cut may refer to the process of partitioning a graph comprising a plurality of vertices (also referred to as nodes) coupled by edges into one or more subsets of vertices.

[0034] An example of a graph cut problem is the minimum st cut problem. A directed graph G = (V, E), where V represents vertices and E represents edges, where w u,v The weights of the edges (u, v) and two special vertices (i.e., source node s and sink node t) may represent the input. For example, the input may include (but is not limited to) an image or a feature mesh. The minimum st cut problem involves partitioning the vertex set V into subsets V 1 and Let s∈V 1 ,t∈V 2 And can be connected with V 1 To V 2 The weights w of the edges that cross the partitions u,v and minimized.

[0035] However, the problem of finding the minimum st cut in a graph is a combinatorial optimization problem, which can be related to the maximum flow problem. Solving combinatorial optimization problems can be challenging.

[0036] Therefore, aspects of the present disclosure relate to performing graph cuts for interpretability in neural network models. Aspects of the present disclosure may provide techniques for determining differentiable solutions to k-partitioning and matching problems. In addition, the disclosed techniques may also be efficient and scalable for object-centric representation learning.

[0037] In some aspects, for example, the topology-aware k-partition problem can be expressed as a problem of solving k minimum st cuts in an image graph or a problem of matching as a bipartite matching problem. A feature of the k-partition problem (e.g., minimum st cut) and the bipartite matching problem can be that the two problems can be expressed as linear processes. Such processes can be included as layers in a neural network by parameterizing the coefficients of the objective function of the linear process using a neural network. However, the linear process may not be continuously differentiable with respect to the objective function coefficients. Another problem may be that a linear process is solved in batches using a traditional solver, which may be too inefficient for a neural network model, especially when the process has a large number of variables and constraints.

[0038] To solve these and other problems, a linear process can be approximated by regularizing a quadratic process with equality constraints, and optimality conditions (e.g., Karush-Kuhn-Tucker (KKT) matrices) can be factorized in advance so that the optimality equations can be solved quickly during training.

[0039] One advantage of using a quadratic process with equality constraints may be that the process can be solved from optimality conditions. Combined with an appropriate precomputed factorization for the object-centric learning task, the optimality conditions can be solved very efficiently during training. Another advantage of using a quadratic process approximation may be that the quadratic process can be differentiated with respect to the process parameters using implicit function theorems.

[0040] In order to learn the target coefficients of the cutting problem by a neural network, it may be necessary to differentiate a linear process, such as a hidden layer. Therefore, the linear process may be relaxed to a quadratic process. Differentiable mathematical techniques may then be employed to obtain the gradient. In this way, the KKT optimality conditions may be solved, and then the gradients with respect to the parameters of the quadratic process may be generated. However, under naive relaxation, the computation of both the forward and backward pass may still be too computationally expensive (e.g., in terms of time and energy consumption) for use in object-centric representation learning applications.

[0041] In view of the fact that many techniques for solving quadratic processes differentially may be limited to smaller program sizes, aspects of the present disclosure provide optimizations for gradient calculations for problems solving graph cuts of image data. For example, potential ST flow graphs may remain unchanged on images of equal size. Therefore, large matrix factorizations may be pre-computed. In addition, forward push may be replaced with a regularized equality-constrained quadratic process constructed from a linear processing formula for the minimum ST cut problem. When combined with such task-specific optimizations, equality-constrained quadratic processes may be solved more efficiently (and in some cases, significantly more efficiently) than general quadratic processes with mixed equality and inequality constraints. Regularization of slack variables may ensure that the output of the disclosed equality-constrained quadratic process may still be interpreted as an ST cut solution. The use of sparse matrix calculations in forward and backward pushes may ensure that time and memory usage may be significantly reduced.

[0042] Thus, certain aspects of the subject matter described in this disclosure may be implemented to achieve one or more of the following potential advantages. In some examples, the techniques may reduce memory and / or power consumption and / or may be capable of performing efficient image segmentation, object-centric feature representation, and interpretability on image data. In addition, the techniques may also be capable of performing differentiable graph cuts and topology-aware feature selection on data where the structure can be represented by a graph.

[0043] Figure 1 An example implementation of a system on chip (SOC) 100 is illustrated, which may include a central processing unit (CPU) 102 or a multi-core CPU configured to implement graph cuts for interpretability. Variables (e.g., neural signals and synaptic weights), system parameters associated with a computing device (e.g., a neural network with weights), delays, frequency bin information, and task information may be stored in a memory block associated with a neural processing unit (NPU) 108, a memory block associated with CPU 102, a memory block associated with a graphics processing unit (GPU) 104, a memory block associated with a digital signal processor (DSP) 106, a memory block 118, or may be distributed across multiple blocks. Instructions executed at CPU 102 may be loaded from a program memory associated with CPU 102, or may be loaded from memory block 118.

[0044] The SOC 100 may also include additional processing blocks tailored for specific functions, such as a GPU 104, a DSP 106, a connectivity block 110 (which may include fifth generation (5G) connectivity, fourth generation long term evolution (4G LTE) connectivity, Wi-Fi connectivity, USB connectivity, Bluetooth connectivity, etc.), and a multimedia processor 112 that may, for example, detect and recognize gestures. In one specific implementation, the NPU 108 is implemented in the CPU 102, the DSP 106, and / or the GPU 104. The SOC 100 may also include a sensor processor 114, an image signal processor (ISP) 116, and / or a navigation module 120, which may include a global positioning system.

[0045] SOC 100 may be based on the ARM instruction set. In one aspect of the present disclosure, the instructions loaded into the general purpose processor 102 may include code for receiving input via an artificial neural network. The general purpose processor 102 may also include code for representing the input as a graph. The graph includes nodes connected by edges. The general purpose processor 102 may additionally include code for determining a graph cut between a source node and a sink node associated with the input via the ANN by solving a quadratic process with an equality constraint. The general purpose processor 102 may also include code for processing a subset of the input based on the graph cut via the ANN to generate a prediction.

[0046] Deep learning architectures can perform object recognition tasks by learning to represent inputs at successively higher levels of abstraction in each layer, thereby building useful feature representations of the input data. In this way, deep learning solves the main bottleneck of traditional machine learning. Before the advent of deep learning, machine learning methods for object recognition problems may rely heavily on human-designed features, possibly in conjunction with shallow classifiers. A shallow classifier can be a two-class linear classifier, for example, in which the weighted sum of feature vector components can be compared to a threshold to predict which class the input belongs to. Human-designed features can be templates or kernels customized for a specific problem domain by engineers with domain expertise. In contrast, although deep learning architectures can learn to represent features similar to those that human engineers may design, they require training. In addition, deep networks can learn to represent and recognize new types of features that humans may not have considered.

[0047] Deep learning architectures can learn hierarchies of features. For example, if presented with visual data, the first layer can learn to recognize relatively simple features in the input stream, such as edges. In another example, if presented with auditory data, the first layer can learn to recognize spectral power in specific frequencies. The second layer, taking the output of the first layer as input, can learn to recognize combinations of features, such as simple shapes for visual data or combinations of sounds for auditory data. For example, higher layers can learn to represent complex shapes in visual data or words in auditory data. Even higher layers can learn to recognize common visual objects or spoken phrases.

[0048] Deep learning architectures perform particularly well when applied to problems that have a natural hierarchical structure. For example, the classification of motorized vehicles can benefit from first learning to recognize wheels, windshields, and other features. These features can be combined in different ways at higher levels to identify cars, trucks, and airplanes.

[0049] Neural networks can be designed to have a variety of connection patterns. In a feedforward network, information is passed from a lower layer to a higher layer, where each neuron in a given layer communicates with a neuron in a higher layer. As described above, hierarchical representations can be constructed in successive layers of a feedforward network. Neural networks can also have loops or feedback (also known as top-down) connections. In a loop connection, the output from a neuron in a given layer can be communicated to another neuron in the same layer. The loop architecture can help identify patterns that span more than one input data block in the input data blocks that are sequentially delivered to the neural network. The connection from a neuron in a given layer to a neuron in a lower layer is called a feedback (or top-down) connection. When the recognition of a high-level concept can assist in discerning specific low-level features of the input, a network with many feedback connections may be helpful.

[0050] The connections between neural network layers can be fully connected or partially connected. Figure 2A An example of a fully connected neural network 202 is illustrated. In a fully connected neural network 202, a neuron in a first layer may communicate its output to each neuron in a second layer, such that each neuron in the second layer will receive input from each neuron in the first layer. Figure 2BAn example of a locally connected neural network 204 is illustrated. In the locally connected neural network 204, neurons in a first layer may be connected to a limited number of neurons in a second layer. More generally, the locally connected layers of the locally connected neural network 204 may be configured such that each neuron in the layer will have the same or similar connection pattern, but the connection strengths may have different values ​​(e.g., 210, 212, 214, and 216). The connection patterns of the local connections may produce spatially different receptive fields in higher layers because higher layer neurons in a given region may receive inputs that are tuned through training to the characteristics of a limited portion of the total input to the network.

[0051] An example of a locally connected neural network is a convolutional neural network. Figure 2C An example of a convolutional neural network 206 is illustrated. The convolutional neural network 206 may be configured such that the connection strengths associated with the inputs of each neuron in the second layer are shared (e.g., 208). Convolutional neural networks may be well suited for problems where the spatial location of the inputs is meaningful.

[0052] One type of convolutional neural network is a deep convolutional network (DCN). Figure 2D A detailed example of a DCN 200 designed to recognize visual features from an image 226 input from an image capture device 230 (such as a vehicle-mounted camera) is illustrated. The DCN 200 of the current example can be trained to identify traffic signs and numbers provided on traffic signs. Of course, the DCN 200 can be trained for other tasks, such as identifying lane markings or identifying traffic lights.

[0053] The DCN 200 may be trained by supervised learning. During training, an image (such as an image 226 of a speed limit sign) may be presented to the DCN 200, and then a forward pass may be calculated to produce an output 222. The DCN 200 may include a feature extraction portion and a classification portion. Upon receiving the image 226, a convolutional layer 232 may apply a convolutional kernel (not shown) to the image 226 to generate a first set of feature maps 218. As an example, the convolutional kernel of the convolutional layer 232 may be a 5x5 kernel that generates a 28x28 feature map. In this example, because four different feature maps are generated in the first set of feature maps 218, four different convolutional kernels are applied to the image 226 at the convolutional layer 232. A convolutional kernel may also be referred to as a filter or a convolution filter.

[0054] The first set of feature maps 218 may be subsampled by a max pooling layer (not shown) to generate a second set of feature maps 220. The max pooling layer reduces the size of the first set of feature maps 218. That is, the size of the second set of feature maps 220 (such as 14x14) is smaller than the size of the first set of feature maps 218 (such as 28x28). The reduced size provides similar information to subsequent layers while reducing memory consumption. The second set of feature maps 220 may be further convolved via one or more subsequent convolutional layers (not shown) to generate one or more sets of subsequent feature maps (not shown).

[0055] exist Figure 2D In the example of , the second set of feature maps 220 are convolved to generate a first feature vector 224. In addition, the first feature vector 224 is further convolved to generate a second feature vector 228. Each feature of the second feature vector 228 may include a number corresponding to a possible feature of the image 226, such as "sign", "60", and "100". A softmax function (not shown) may convert the numbers in the second feature vector 228 into probabilities. Therefore, the output 222 of the DCN 200 may be a probability that the image 226 includes one or more features.

[0056] In this example, the probability of "logo" and "60" in output 222 is higher than the probability of other numbers (such as "30", "40", "50", "70", "80", "90", and "100") in output 222. Before training, the output 222 generated by DCN 200 may be incorrect. Therefore, the error between output 222 and the target output can be calculated. The target output is the ground truth value of image 226 (e.g., "logo" and "60"). The weights of DCN 200 can then be adjusted so that the output 222 of DCN 200 is closer to the target output.

[0057] To adjust the weights, the learning algorithm may calculate a gradient vector for the weights. The gradient may indicate the amount by which the error will increase or decrease if the weights are adjusted. At the top layer, the gradient may correspond directly to the value of the weights connecting the activated neurons in the penultimate layer and the neurons in the output layer. In lower layers, the gradient may depend on the value of the weights and the calculated error gradients of the higher layers. The weights may then be adjusted to reduce the error. This way of adjusting weights may be referred to as "back propagation" because it involves "pushing backwards" through the neural network.

[0058] In practice, the error gradient of the weights may be calculated over a small number of examples so that the calculated gradient is close to the true error gradient. This approximation method may be referred to as stochastic gradient descent. Stochastic gradient descent may be repeated until the achievable error rate of the entire system stops decreasing or until the error rate reaches a target level. After learning, a new image (e.g., a speed limit sign of image 226) may be presented to DCN 200, and an output 222 may be generated by the forward push of DCN 200, which may be considered an inference or prediction of DCN 200.

[0059] Deep belief network (DBN) is a probabilistic model including multiple layers of hidden nodes. DBN can be used to extract hierarchical representations of training data sets. DBN can be obtained by stacking the layers of restricted Boltzmann machine (RBM). RBM is a type of artificial neural network that can learn probability distributions through a set of inputs. Because RBM can learn probability distributions without information about the categories to which each input should be classified, RBM is generally used for unsupervised learning. Using an unsupervised and supervised hybrid paradigm, the bottom RBM of DBN can be trained in an unsupervised manner and can be used as a feature extractor, while the top RBM can be trained in a supervised manner (on the joint distribution of the input and target class from the previous layer) and can be used as a classifier.

[0060] A DCN is a network of convolutional networks, configured with additional pooling and normalization layers. DCN achieves state-of-the-art performance on many tasks. DCN can be trained using supervised learning, where both the input and output targets are known for many examples and are used to modify the weights of the network by using a gradient descent method.

[0061] The DCN may be a feed-forward network. In addition, as described above, the connections from a neuron in the first layer of the DCN to a set of neurons in the next higher layer are shared across the neurons in the first layer. The feed-forward and shared connections of the DCN may be used for fast processing. For example, the computational burden of the DCN may be much smaller than that of a similarly sized neural network that includes loops or feedback connections.

[0062] The processing of each layer of the convolutional network can be considered as a spatially invariant template or basis projection. If the input is first decomposed into multiple channels, such as the red, green and blue channels of a color image, the convolutional network trained on the input can be considered to be three-dimensional, with two spatial dimensions along the axis of the image and the third dimension capturing color information. The output of the convolutional connection can be considered to form a feature map in a subsequent layer, where each element in the feature map (e.g., 220) receives input from a certain range of neurons in the previous layer (e.g., feature map 218) and from each channel in the multiple channels. The values ​​in the feature map can be further processed with nonlinearity (such as correction, max(0,x)). The values ​​from adjacent neurons can be further pooled, which corresponds to downsampling and can provide additional local invariance and dimensionality reduction. Normalization corresponding to whitening can also be applied by lateral inhibition between neurons in the feature map.

[0063] Figure 3 is a block diagram illustrating a DCN 350. The DCN 350 may include multiple different types of layers based on connection and weight sharing. Figure 3 As shown, DCN 350 includes convolution blocks 354A, 354B. Each of convolution blocks 354A, 354B may be configured with a convolution layer (CONV) 356, a normalization layer (LNorm) 358, and a maximum pooling layer (MAX POOL) 360.

[0064] Although only two of the convolution blocks 354A, 354B are shown, the present disclosure is not limited thereto, and any number of convolution blocks 354A, 354B may be included in the DCN 350 according to design preferences.

[0065] The convolution layer 356 may include one or more convolution filters that may be applied to the input data to generate a feature map. The normalization layer 358 may normalize the output of the convolution filter. For example, the normalization layer 358 may provide whitening or lateral suppression. The maximum pooling layer 360 may provide spatial downsampling aggregation to achieve local invariance and dimensionality reduction.

[0066] For example, a parallel filter bank of a deep convolutional network may be loaded on SOC 100 (e.g., Figure 1 ) on the CPU 102 or GPU 104 of the SOC 100 to achieve high performance and low power consumption. In an alternative embodiment, the parallel filter bank can be loaded onto the DSP 106 or ISP 116 of the SOC 100. In addition, the DCN 350 can access other processing blocks that may exist on the SOC 100, such as the sensor processor 114 and the navigation module 120 dedicated to sensors and navigation, respectively.

[0067] The DCN 350 may also include one or more fully connected layers 362 (FC1 and FC2). The DCN 350 may also include a logistic regression (LR) layer 364. Between each layer 356, 358, 360, 362, 364 of the DCN 350 are weights (not shown) to be updated. The output of each layer in the layers (e.g., 356, 358, 360, 362, 364) may be used as an input to a subsequent layer in the layers (e.g., 356, 358, 360, 362, 364) in the DCN 350 to learn a hierarchical feature representation from the input data 352 (e.g., images, audio, video, sensor data, and / or other input data) supplied at the first convolutional block in the convolutional block 354A. The output of the DCN 350 is a classification score 366 for the input data 352. The classification score 366 may be a set of probabilities, where each probability is a probability that the input data includes a feature from a feature set.

[0068] Figure 4 400 is a block diagram illustrating an exemplary software architecture 400 that can modularize artificial intelligence (AI) functions. According to aspects of the present disclosure, using the architecture 400, a SOC 420 (which can be similar to Figure 1 Various processing blocks (e.g., CPU 422, DSP 424, GPU 426, and / or NPU 428) of the SOC 100 support the application of graph cutting for interpretability and object-centric representation learning of the AI ​​application 402. The architecture 400 may be included in a computing device such as a smart phone, for example.

[0069] The AI ​​application 402 may be configured to call functions defined in the user space 404 that may, for example, provide detection and recognition of scenes indicating the location where the computing device (including the architecture 400) is currently operating. For example, the AI ​​application 402 may configure microphones and cameras differently depending on whether the identified scene is an office, a lecture hall, a restaurant, or an outdoor environment such as a lake. The AI ​​application 402 may make a request for compiled program code associated with a library defined in the AI ​​function application programming interface (API) 406. The request may ultimately rely on the output of a deep neural network configured to provide an inferred response based on, for example, video and positioning data.

[0070] The runtime engine 408, which may be compiled code of the runtime framework, may further be accessible to the AI ​​application 402. The AI ​​application 402 may cause the runtime engine 408 to request inference, for example, at a specific time interval or triggered by an event detected by a user interface of the AI ​​application 402. Upon causing the runtime engine 408 to provide an inference response, the runtime engine may then transmit a signal to an operating system (such as a kernel 412) in an operating system (OS) space 410 running on the SOC 420. In some examples, the kernel 412 may be a LINUX kernel. The operating system may then cause continuous quantization relaxation to be performed on a CPU 422, a DSP 424, a GPU 426, an NPU 428, or some combination thereof. The CPU 422 may be directly accessible by the operating system, while other processing blocks may be accessed through drivers, such as drivers 414, 416, or 418 for the DSP 424, the GPU 426, or the NPU 428, respectively. In an illustrative example, the deep neural network may be configured to run on a combination of processing blocks such as CPU 422 , DSP 424 , and GPU 426 , or may run on NPU 428 .

[0071] Various aspects of the present disclosure relate to determining graph cuts for interpretability of deep neural network models. According to various aspects of the present disclosure, graph cut methods can be incorporated into neural networks using, for example, linear processing formulas or specialized processes. In one application, a minimum st cut segmentation of an image is used in a post hoc interpretability method to improve the stability and quality of predictions.

[0072] According to aspects of the present disclosure, graph cuts may be applied in per-instance post hoc interpretability of an image using, for example, a learning explanation framework. That is, given a classification network, a second network may be configured to take a fragment of the input and aim to match the prediction of the classifier. In other words, the second network attempts to determine a subset of the input that is important for producing the predicted output of the classification network. The second network may be referred to as an explainer network and may generate a mask that specifies which fragment of the input is passed to the classification network.

[0073] Traditional methods use pixel-by-pixel weights and subset sampling methods to sample k element subsets. However, such traditional methods may discard more input structure, which may reduce the quality and stability of output prediction. In order to overcome such problems, aspects of the present disclosure may adopt a graph cut partitioning method that can maintain more input (e.g., image) structure than subsets. By implementing the graph cut method, the interpreter network may, for example, output parameters of a graph minimum cut problem, thereby treating the image as a graph. Solving the minimum cut problem can provide a partitioning of pixels. Negative pixels can be masked in the image and positive pixels can be supplied to downstream networks. In addition, the technology of the present disclosure can be beneficially applied to object-centric representation learning and matching tasks.

[0074] Figure 5A is a diagram illustrating an example cut of an example graph 500 according to aspects of the present disclosure. Figure 5A As shown in , the graph 500 includes a set of vertices 502 (also referred to as nodes) coupled by edges 504. For ease of illustration, only two edges (e.g., 504a, 504e) are labeled. In addition, although the example graph includes five nodes (e.g., 502a to 502e) and seven edges (e.g., 504a, 504e), this number is only for ease of illustration and is not limiting.

[0075] One or more graph cuts (e.g., 506a, 506b) may partition graph 500 into two or more portions. Each graph cut 506a, 506b may partition vertices (e.g., 502a to 502e) into disjoint subsets. For example, graph cut 506a may partition graph 500, thereby separating vertex 502a from vertices 502b to 502e. On the other hand, graph cut 506b may further partition the graph, thereby separating vertices 502b and 502c from vertices 502d and 502e. Additional cuts may also be included based on design preferences.

[0076] Figure 5B 5 is a diagram illustrating an example graph 550 for illustrating an example minimum st cut problem according to aspects of the present disclosure. Figure 5B As shown in , a graph 550 may include a set of vertices 552 (e.g., 552a to 552z) coupled together by a set of edges 554 (e.g., 554a to 552z). For ease of illustration, only element numbers of two instances of vertices 552 and edges 554 are provided. For example, graph 550 may represent an image (e.g., an image map) or an input feature grid. Edges 554 may represent relationships between vertices 552.

[0077] Source node s 556 may be coupled to each of vertices 552 via edge 560. Sink node t 558 (also referred to as a sink node) may be coupled to each of vertices 552 via edge 562. One goal of the minimum st cut problem may be to determine a cut or partition of graph 550 such that the sum of the weights of the edges between a partition defined by edge 560 coupled to source node s 556 and a partition defined by edge 562 coupled to sink node t 558 may be minimized. Partitioning may be achieved by removing edges (e.g., 560 or 562) that couple a node (e.g., 552) to source node s 556 or sink node t 558.

[0078] As described, for example, the minimum ST cut problem has been used in computer vision and graphics for image segmentation. The image segmentation (e.g., segmentation into foreground and background) problem can be reduced to a minimum ST cut by representing the image as a weighted grid graph (e.g., 550) with vertices (e.g., 552) representing pixels, where the weights of the edges (e.g., 554) can represent the similarity of neighboring pixels. Source vertices s (e.g., 556) and sink vertices t (e.g., 558) can be introduced so that each pixel can include vertex v 552, edge (s, v) 560, and edge (v, t) 562. The weight w of edge 560 is s,v and the weight w of edge 562 t,v The relationship between the pixels of each partition (e.g., relative background and foreground weights) can be represented separately for the pixel vertex v. For example, the vertex w of vertex 556 s,v >w t,v It may indicate that the pixel vertex v may be more likely to be associated with the partition defined by the source vertex 556 than with the partition defined by the sink vertex t 558 (e.g., more likely to be in the foreground than in the background). u,v It may be indicated that similar pixels should be included in the same partition (e.g., the partition defined by source vertex 556). Solving the minimum cut problem on the graph may result in image segmentation (e.g., foreground and background). Using such a process, the underlying image topology may be encoded into a graph structure.

[0079] The graph cut problem can be formulated as a linear process. In this way, the variable p can be defined for each vertex u in the linear process. u Similarly, for each edge (u, v), we can also define a variable for the edge weight, denoted by w u,v .

[0080] Edge weight w u,v It can be used as the objective function parameter in the linear process. The linear process can be written as follows.

[0081]

[0082] where p s 、p t p u 、p v Corresponding to vertices s, t, u, and v respectively and d uv Corresponds to the edge (u,v).

[0083] For example, the process may be supplied to a linear process solver to determine a solution.While other methods may exist for solving graph cut problems, the linear process formulation may provide a flexible solution that may be approximated by a differentiable surrogate.

[0084] The minimum st cut in the image map for partitioning the image feature map x′=g(x) into k disjoint partitions can be determined, for example, using weights that can be parameterized by a neural network. For image x, w u,v =f(x). Solving the standard minimum st cut problem can partition the vertex set of the input graph into two partitions. The minimum st cut problem can be generalized as partitioning a given graph into a fixed k number of partitions by solving k parallel minimum st cut problems and then normalizing the vertex (or edge) variables to sum to one over k copies of the graph. For example, the two partitions can also be formulated as a subset selection problem. One advantage of using the minimum st cut for partitioning can be that the minimum st cut method depends on the underlying topology and adjacency of the image data represented in the image graph.

[0085] This can be done by solving k parallel bipartition problems and using the flexible maximum function in i to convert the vertex variables from the optimal solution of Eq. 1 Normalization is used to generalize the minimum st cut of two partitions to k-partitions, where i denotes the partition index. In some aspects, the k-partition problem can be solved by a single graph cut formula. However, such a formula may have higher memory usage as k increases.

[0086] There may be some obstacles associated with including linear processes as hidden layers in neural networks when extended to solving large computer vision problems. One obstacle is that the linear process may not be continuously differentiable with respect to the linear process parameters. Some traditional methods utilize quadratic programming relaxation techniques. However, the gradient calculations of such large quadratic processes perform large matrix factorizations that may be infeasible in terms of computational time and memory when applied to image-level data.

[0087] Another obstacle may be that established quadratic process solvers can operate on CPUs and cannot solve programs in large batches associated with fast neural network optimization. Some traditional methods may attempt to accelerate the solution of general quadratic processes through GPUs. However, in practice, such traditional methods may not scale well to quadratic process solving in batch settings with tens of thousands of variables and constraints that may occur when applied to image data.

[0088] Therefore, aspects of the present disclosure approximate the st-cut problem as a quadratic process that can be solved differentiably. The disclosed techniques can advantageously enable solving large batches of quadratic processes on a GPU and further applied to image data.

[0089] Given an equality constrained quadratic process, it can be solved directly by computing the Karush-Kuhn-Tucker (KKT) matrix factorization and solving the resulting triangular system.

[0090] A linear process or a quadratic process (e.g., for determining the st cut) may have non-negativity constraints on variables that may not be represented in an equality-constrained quadratic process. Alternatively, a regularization term may be applied in the objective function, which may ensure that the variables are within a reasonable range. After solving the regularized equality-constrained process, the solutions may be transformed, for example, by a sigmoid function or a soft maximum function, for example, to bring them within a desired range.

[0091] In some aspects, the minimum st-cut linear process defined in Equation 1 can be approximated by the following quadratic process with equality constraints:

[0092] Minimize:

[0093]

[0094] Comply with:d uv -s uv =p u -p v (u,v)∈E,p s -p t -s st =1,

[0095] The variable s uv is the slack variable and γ is the regularization coefficient. Each inequality constraint can add a slack variable s uv to convert the inequality constraints into equality constraints plus any non-negativity constraints. Next, a diagonal quadratic regularization term can be added for all variables including the slack variables in the objective function. Then, just any non-negativity constraints can be removed to obtain a quadratic process with equality constraints.

[0096] Given an input feature x, the edge weights w can be transformed into u,v The parameterization is w = f(x) to include the regularized equality-constrained training process of the st-cut in the artificial neural network.

[0097] refer to Figure 5B , in one example, each vertex 552 may be connected to four neighboring vertices as well as a source node s 556 and a sink node t 558, for a total of six edges per vertex 552 (e.g., five outgoing edges and one incoming edge). Given that the graph 550 has a height H and a width W, the weights of the k partitions may be parameterized using a ConvNet (f) with an output size of (6k,H,W), thereby assigning six edge weights per partition per pixel. The output of the regularized equality constrained quadratic process for minimum st cuts may be interpreted as a cut after a soft maximum and may be applied to image-level data.

[0098] The computation of the regularized equality-constrained quadratic process and the corresponding gradients for solving the equality-constrained quadratic process can be limited with respect to the objective function parameters. The equality-constrained quadratic process can have the following form:

[0099]

[0100] in is a constraint matrix consisting of l constraints, and For a quadratic proxy to a linear process, G represents a multiple of the identity, G = γI, corresponding to the regularization term γ, and c is a learnable parameter. In some aspects, the constraint matrix A and the variables b can be fixed and the slack variables s can be used. uv Encode the graph constraints converted into equality constraints. Then, the non-negativity constraints can be removed.

[0101] The quadratic process can be solved directly by solving the following KKT optimality condition for the dual vector λ:

[0102]

[0103] The leftmost matrix is ​​the KKT matrix. In the case of G = γI, the inverse of the KKT matrix can be written as:

[0104]

[0105] in And F=-(AA t ) -1 In equation 5, (AA t ) -1 It is in the form of an inverse and does not change. Therefore, it can be pre-calculated (AA t ) -1 In some aspects, Cholesky factorization can be used to precompute (AA t ) -1 According to the trigonometric factor L, AA t Factorize into AA t =LL t Then, during training, the KKT system by computing Cz or Ez (Equation 5) for a value z can be efficiently solved by using forward and back substitution for the triangular factors L.

[0106] Having determined the KKT matrix factorization of Equation 5, the KKT matrix can be used for gradient calculation. Determine the gradient with respect to the parameters c, where l(.) is the loss function and z is the solution to the quadratic process. The Cholesky factor L can be used for the gradient, making the gradient computation more efficient.

[0107] Another challenge is that since the image map can scale quadratically with the image resolution, the constraint matrix A can be very large for object-centric applications. For example, 64x64 images can produce an ST-cut quadratic procedure with more than 52,000 variables and more than 23,000 constraints (e.g., can make the constraint matrix have a size of more than four gigabytes (4GB)), which can be solved in batches for each training example. Since the constraint matrix can be very sparse (about 6% non-zero elements for 64x64 images), sparse matrices and sparse Cholesky factorization can be used.

[0108] During training, a batch sparse triangular solver can be used for back- and forward-generation to solve a different batch of quadratic procedures on the right-hand side of Equation 4. Given a graph size of N, directly solving the quadratic procedure (e.g., with dense Cholesky factorization) can have a cost of approximately O(N 3 ) complexity, and may be even greater for general quadratic processes. Applying the optimization techniques disclosed in this disclosure can reduce the complexity to O(Nn z ), where n z is the number of non-zero elements, which may be the time taken by the sparse triangular solver.

[0109] Figure 6 6 is a block diagram illustrating an example architecture 600 for determining graph cuts in accordance with aspects of the present disclosure. The example architecture 600 may include an encoder 602 and a graph partitioning module (shown as a k-partitioning module) 604. The encoder 602 may receive an input 606. The input 606 may include a graph. The graph may, for example, represent (but is not limited to) an image or a feature grid. In an example where the input 606 represents an image, the vertices of the graph may represent pixels of the image and the edges may represent relationships between pixels of the image.

[0110] The encoder 602 may extract features of the input 606 to generate a feature map 608. The feature map 608 may be supplied to a graph partitioning module 604. The graph partitioning module 604 may then partition the feature map 608 into two or more subsets or segments of features 610.

[0111] Fig. 7A is an example of various aspects of the present disclosure Figure 6 Graph partitioning module 604 for partitioning a graph. Graph partitioning module 604 may include an artificial neural network (ANN) 702, a quadratic process (QP) solver 704, and a multi-layer perceptron (MLP) 708. ANN 702 may include, but is not limited to, a convolutional neural network (CNN) (e.g., Figure 3For example, the ANN 702 may be configured to perform a quadratic process, such as a quadratic process with only the equality constraints provided in Equation 2, or an equality-constrained quadratic process defined in Equation 3, or another quadratic process.

[0112] The ANN 702 may receive the feature map 608. The ANN 702 may apply a quadratic process to the feature map 608 to determine a set of quadratic process parameters. The quadratic process parameters may be provided to a quadratic process solver 704. In some aspects, for example, the quadratic process solver 704 may include, but is not limited to, a serial solver (e.g., a Gurobi solver) or a parallel quadratic process solver. The quadratic process solver 704 may solve the quadratic process based on the quadratic process parameters to generate a partition mask 706.

[0113] In some aspects, the partition mask 706 can be a binary mask indicating features to keep (e.g., indicated by 1) or to discard (e.g., indicated by 0). The binary mask can then be applied to the feature grid 710. The partition mask 706 can be applied to the feature grid 710 to partition the feature grid 710 into two or more sub- or fragments of the features 610.

[0114] Figure 7B is a diagram illustrating example pseudo code 750 for a partition map according to aspects of the present disclosure. Figure 7B As shown in , an input (e.g., an image or a feature grid) can be represented as a graph. A set of quadratic process parameters can be determined based on the input, for example using a CNN (line 1). A regularized quadratic process can be calculated to determine one or more cut segments or partitions in the graph (line 3). The regularized quadratic process can include equality constraints and can solve a minimum cut problem. Vertex variables of the cut set can be extracted. For example, the extracted vertex variables can be normalized across each pixel cut by applying a temperature-scaled flexible maximum function (line 4). The normalized vertex variables can be multiplied by the spatial feature x f To generate K masked feature maps r i (e.g., K partition features) (line 5).

[0115] In some aspects, the regularized quadratic process framework can be extended to the matching problem. Given two sets of k partitions P 1 and P 2 , the task can be in P 1 and P 2 The matching partitions in , which can be viewed as a bipartite matching problem. The bipartite matching problem can be formulated according to a linear process given by:

[0116]

[0117] The node variable can be labeled as p for node u u, for edge {u,v}, label the edge variable as d u,v , and the edge cost is c u,v .

[0118] Unlike the st-cut problem, the bipartite matching problem may involve a bipartite graph, which may include, for example, 2x12=24 nodes or fewer. For such smaller problems, a CPU-based solver or parallel batch solver (e.g., after suitable relaxation) may be employed. However, in some aspects, the same method of regularized equality-constrained quadratic procedures for approximating the graph cut problem may be used for the bipartite matching problem.

[0119] In some aspects, the k-partitioning scheme can also be used for object-centric representation learning. Object-centric representation learning aims to learn the representation of individual objects in a scene given in the form of a static image or video. One way to formulate object-centric representation learning is to regard it as an input partitioning problem. In the input partitioning problem, a set of spatial scene features is given, and the features can be partitioned into k object-by-object features, also known as slots, for a given number of objects k. A useful property of the partitioning scheme is topological awareness. For example, the partitioning scheme should be aware that points close together in space may be related and may form part of the same object. A related problem is to match object representations in two closely related scenes (such as frames in a video) to learn object permanence in space and time.

[0120] Fig. 8A is a diagram illustrating an example object-centric feature representation 800 according to aspects of the present disclosure. Fig. 8A , a set of different images 802a to 802f may be supplied to an artificial neural network, such as Figure 6 802a-802f. In addition, the k-partitions of features of the respective input images may be further processed to determine one or more features of each object in the images 802a-802f. By determining the features of each object in the respective images 802a-802f, objects in the scene of each image may be reconstructed (e.g., as shown in slots 820a-820k). Slots 820a-820k provide one slot for each object, such that each slot 820a-820k may reconstruct an object discovered by the network (e.g., the example architecture 600).

[0121] Figure 8B830 is a diagram illustrating an example pseudocode 830 for object-centric representation according to aspects of the present disclosure. Given a scene feature x of size C x H x W as input (where C represents channel, H represents height, and W represents width), the scene feature x may be transformed into k vectors, referred to as slots. Each of the slots may correspond to and represent a specific object in the scene. The goal may be to determine a graph cut that preserves the topological properties of the pixel space, so that a neural network may learn to group objects together in a spatially coherent manner, rather than arbitrarily assigning distant regions to the same slot.

[0122] An encoder-decoder structure including a minimum st cut layer may be employed. Based on scene features that may be extracted, for example, by a CNN backbone, k minimum st cut quadratic process parameters may be computed using a smaller CNN. The k quadratic processes may be solved and soft max normalization may be applied to the vertex variables to produce a normalized (soft) spatial mask z across i i . The spatial mask z i Multiply (optionally transform) the scene feature x broadcasted across the channel dimension. This produces a masked feature r of size D x H x W forming k partitions i ( Figure 8B Line 2).

[0123] To obtain the slot object representation, the localization encoding is added to the scene feature x before partitioning (line 1). After partitioning, each partition r of size D x H x W across the spatial dimensions can be calculated i The k vectors may then be transformed, for example, by a multi-layer perceptron to obtain slots (line 4). Each slot may then be decoded individually. Separate output channels may combine individual slot images. The combined image may be trained to reconstruct the original scene features x and / or the original scene.

[0124] In some aspects, the object-centric framework described can also be extended to a dedicated background slot. A set of masks for the two partitions and a set of masks for the k-1-partition can be calculated, denoted as mask foreground (mask_fg), mask background (mask_bg), and object mask (mask_object). The mask foreground and object masks can then be multiplied as mask_fg*mask_object to calculate a dedicated slot for the background.

[0125] Figure 8C is a diagram illustrating an example pseudocode 850 for object-centric representation according to aspects of the present disclosure. Object representation learning can be extended to moving objects in video by combining an object-centric learning model with the matching process of pseudocode 850. Given a pair of frames of a video, for each frame, k per-frame slot representations can be computed, e.g., Figure 8BAs shown in pseudo code 830 (line 1). Using the linear matching process defined in equation 6, the matching matrix m can be determined i,j (Line 3). The flexible maximum function can be applied to the matching matrix M along the j dimension i,j Then, using the matching process of pseudo code 850, the two slot sets can be matched to generate a new slot r i (Line 5). In some aspects, the new slot may be transformed, for example, by an MLP. A decoder (not shown) may process the new slot in a manner reciprocal to that of encoder 602, for example to reconstruct an input pair of frames by optimizing the sum of the reconstruction errors of the two frames.

[0126] Fig. 9 9 is a flow chart illustrating an example processor-implemented method 900 for implementing graph cuts for interpretability using an artificial neural network in accordance with aspects of the present disclosure. For example, the processor-implemented method 900 may be performed by one or more processors such as a CPU (e.g., 102, 422), a GPU (e.g., 104, 426), a DSP (e.g., 106, 424), and / or an NPU (e.g., 108, 428).

[0127] like Fig. 9 As shown in FIG. 9 , at block 902, one or more processors receive input via an artificial neural network. For example, the input may include an image (which may also include but is not limited to a frame of a video) or a feature grid.

[0128] At block 904, one or more processors represent the input as a graph that includes nodes connected by edges. For example, an image may be represented as a graph where each pixel is represented by a node of the graph. In addition, the connection between each node may be weighted based on whether the pixel is connected.

[0129] At block 906, the one or more processors determine, via the ANN, a graph cut between a source node and a sink node associated with the input by solving a quadratic process with an equality constraint. As discussed, graph cuts can be applied in a per-instance post hoc interpretability of an image using, for example, a learning explanation framework. In some aspects, a graph cut can be a graph minimum source node-sink node cut. Other examples of graph cuts include multi-cut problems and multi-way cuts and minimum k-cut problems, all of which can be solved or approximated by linear processes.

[0130] At block 908, one or more processors process a subset of the input based on the graph cut via the ANN to generate a prediction. As discussed, given a classification network, a second network can be configured to take a fragment of the input and aim to match the prediction of the classifier. In other words, the second network attempts to determine a subset of the input that is important for producing the predicted output of the classification network.

[0131] Specific implementation examples are provided in the following numbered clauses.

[0132] 1. A device, comprising:

[0133] at least one memory; and

[0134] at least one processor, the at least one processor coupled to the at least one memory, the at least one processor configured to:

[0135] Receiving input via an artificial neural network (ANN);

[0136] Representing the input as a graph comprising a plurality of nodes connected by edges;

[0137] determining, via the ANN, a graph cut between source nodes and sink nodes associated with the input by solving a quadratic process with equality constraints; and

[0138] A subset of the inputs is processed via the ANN based on the graph cuts to generate predictions.

[0139] 2. An apparatus according to clause 1, wherein the graph cut partitions the input such that the weight of the cross-edges between segments is minimized.

[0140] 3. An apparatus according to clause 1 or 2, wherein the input is an image or a feature grid.

[0141] 4. An apparatus according to any one of clauses 1 to 3, wherein the plurality of nodes correspond to pixels of the image or the feature grid.

[0142] 5. An apparatus according to any one of clauses 1 to 4, wherein the feature grid corresponds to image features of a scene including one or more objects in the image, and the at least one processor is further configured to divide the image features of the scene among the one or more objects.

[0143] 6. An apparatus according to any one of clauses 1 to 5, wherein the graph cut is differentiable.

[0144] 7. An apparatus according to any one of clauses 1 to 6, wherein the ANN comprises one or more differentiable optimization layers.

[0145] 8. A processor-implemented method executed by at least one processor, the processor-implemented method comprising:

[0146] Receiving input via an artificial neural network (ANN);

[0147] Representing the input as a graph comprising a plurality of nodes connected by edges;

[0148] determining, via the ANN, a graph cut between source nodes and sink nodes associated with the input by solving a quadratic process with equality constraints; and

[0149] A subset of the inputs is processed via the ANN based on the graph cuts to generate predictions.

[0150] 9. A processor-implemented method according to clause 8, wherein the graph cut partitions the input such that the weight of the crossing edges between segments is minimized.

[0151] 10. A processor-implemented method according to clause 8 or 9, wherein the input is an image or a feature grid.

[0152] 11. A processor-implemented method according to any of clauses 8 to 10, wherein the plurality of nodes correspond to pixels of the image or the feature grid.

[0153] 12. A processor-implemented method according to any one of clauses 8 to 11, wherein the feature grid corresponds to image features of a scene including one or more objects in the image, and the processor-implemented method further comprises dividing the image features of the scene among the one or more objects.

[0154] 13. A processor-implemented method as recited in any one of clauses 8 to 12, wherein the graph cut is differentiable.

[0155] 14. A processor-implemented method according to any of clauses 8 to 13, wherein the ANN comprises one or more differentiable optimization layers.

[0156] 15. A non-transitory computer readable medium having program code recorded thereon, the program code being executed by at least one processor and comprising:

[0157] Program code for receiving input via an artificial neural network (ANN);

[0158] program code for representing the input as a graph, the graph comprising a plurality of nodes connected by edges;

[0159] program code for determining, via the ANN, a graph cut between source nodes and sink nodes associated with the input by solving a quadratic process with equality constraints; and

[0160] Program code for processing, via the ANN, a subset of the inputs based on the graph cuts to generate predictions.

[0161] 16. The non-transitory computer-readable medium of clause 15, wherein the graph cut partitions the input such that a weight of a cross edge between segments is minimized.

[0162] 17. A non-transitory computer readable medium according to clause 15 or 16, wherein the input is an image or a feature grid.

[0163] 18. The non-transitory computer-readable medium of any one of clauses 15 to 17, wherein the plurality of nodes correspond to pixels of the image or the feature grid.

[0164] 19. A non-transitory computer-readable medium according to any one of clauses 15 to 18, wherein the feature grid corresponds to image features of a scene including one or more objects in the image, and the program code also includes program code for dividing the image features of the scene among the one or more objects.

[0165] 20. The non-transitory computer-readable medium of any one of clauses 15 to 19, wherein the graph cut is differentiable.

[0166] 21. The non-transitory computer-readable medium of any one of clauses 15 to 20, wherein the ANN comprises one or more differentiable optimization layers.

[0167] 22. A device, comprising:

[0168] means for receiving input via an artificial neural network (ANN);

[0169] means for representing the input as a graph, the graph comprising a plurality of nodes connected by edges;

[0170] means for determining, via the ANN, a graph cut between source nodes and sink nodes associated with the input by solving a quadratic process with equality constraints; and

[0171] Means for processing, via the ANN, a subset of the inputs based on the graph cut to generate predictions.

[0172] 23. An apparatus according to clause 22, wherein the graph cut partitions the input such that the weight of the crossing edges between segments is minimized.

[0173] 24. An apparatus according to clause 22 or 23, wherein the input is an image or a feature grid.

[0174] 25. An apparatus according to any of clauses 22 to 24, wherein the plurality of nodes correspond to pixels of the image or the feature grid.

[0175] 26. An apparatus according to any one of clauses 22 to 25, wherein the feature grid corresponds to image features of a scene including one or more objects in the image, and the apparatus further comprises means for dividing the image features of the scene at the one or more objects.

[0176] 27. An apparatus according to any of clauses 22 to 26, wherein the graph cut is differentiable.

[0177] 28. An apparatus according to any of clauses 22 to 27, wherein the ANN comprises one or more differentiable optimization layers.

[0178] In one aspect, the receiving means, representing means, determining means and / or processing means may be a GPU 104 configured to perform the functions, a program memory associated with the GPU 104, a fully connected layer 362, an NPU 428 and / or a routing connection processing unit 216. In another configuration, the aforementioned means may be any module or any device configured to perform the functions recited by the aforementioned means.

[0179] The various operations of the above methods may be performed by any suitable components capable of performing the corresponding functions. These components may include various hardware and / or software components and / or modules, including but not limited to circuits, application specific integrated circuits (ASICs), or processors. In general, where operations are illustrated in the accompanying drawings, these operations may have corresponding paired components plus function components with similar numbers.

[0180] As used, the term "determining" encompasses a wide variety of actions. For example, "determining" may include calculating, computing, processing, deriving, investigating, searching (e.g., searching in a table, database, or another data structure), ascertaining, etc. Additionally, "determining" may include receiving (e.g., receiving information), accessing (e.g., accessing data in a memory), etc. Furthermore, "determining" may include resolving, selecting, choosing, establishing, etc.

[0181] As used, a phrase referring to "at least one of" a list of items refers to any combination of those items, including single members. As an example, "at least one of a, b, or c" is intended to cover: a, b, c, ab, ac, bc, and abc.

[0182] The various illustrative logical blocks, modules, and circuits described in conjunction with the present disclosure may be implemented or performed with a general purpose processor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array signal (FPGA) or other programmable logic device (PLD), discrete gate or transistor logic components, discrete hardware components, or any combination thereof designed to perform the functions described. A general purpose processor may be a microprocessor, but in an alternative, the processor may be any commercially available processor, controller, microcontroller, or state machine. The processor may also be implemented as a combination of computing devices, such as a combination of a DSP and a microprocessor, a plurality of microprocessors, one or more microprocessors combined with a DSP core, or any other such configuration.

[0183] The steps or algorithms of the methods described in conjunction with the present disclosure may be directly embodied in hardware, software modules executed by a processor, or a combination of the two. The software module may reside in any form of storage medium known in the art. Some examples of usable storage media include random access memory (RAM), read-only memory (ROM), flash memory, erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), registers, hard disks, removable disks, CD-ROMs, and the like. The software module may include a single instruction, perhaps multiple instructions, and may be distributed over several different code segments, distributed between different programs, and distributed across multiple storage media. The storage medium may be coupled to a processor so that the processor can read information from the storage medium and write information to the storage medium. In an alternative, the storage medium may be integral with the processor.

[0184] The disclosed methods include one or more steps or actions for implementing the described methods. The steps and / or actions of the methods may be interchanged with each other without departing from the scope of the claims. In other words, unless a specific order of steps or actions is specified, the order and / or use of specific steps and / or actions may be modified without departing from the scope of the claims.

[0185] The functions described may be implemented in hardware, software, firmware, or any combination thereof. If implemented in hardware, an example hardware configuration may include a processing system in a device. The processing system may be implemented using a bus architecture. Depending on the specific application and overall design constraints of the processing system, the bus may include any number of interconnecting buses and bridges. The bus may link various circuits together, including a processor, a machine-readable medium, and a bus interface. The bus interface may be used to connect a network adapter, etc., to the processing system via the bus. The network adapter may be used to implement signal processing functions. For some aspects, a user interface (e.g., a keypad, a display, a mouse, a joystick, etc.) may also be connected to the bus. The bus may also link various other circuits, such as timing sources, peripherals, voltage regulators, power management circuits, etc., which are well known in the art and will not be described further.

[0186] The processor may be responsible for managing the bus and general processing, including executing software stored on the machine-readable medium. The processor may be implemented using one or more general-purpose processors and / or special-purpose processors. Examples include microprocessors, microcontrollers, DSP processors, and other circuits that can execute software. Software should be broadly interpreted as meaning instructions, data, or any combination thereof, whether referred to as software, firmware, middleware, microcode, hardware description language, or other. By way of example, a machine-readable medium may include a random access memory (RAM), a flash memory, a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a register, a disk, an optical disk, a hard disk drive, or any other suitable storage medium, or any combination thereof. The machine-readable medium may be embodied in a computer program product. The computer program product may include packaging materials.

[0187] In hardware implementation, machine-readable media can be part of a processing system separate from a processor. However, as will be readily appreciated by those skilled in the art, machine-readable media or any part thereof can be outside a processing system. By way of example, machine-readable media can include a transmission line, a carrier modulated by data, and / or a computer product separate from a device, all of which can be accessed by a processor through a bus interface. Alternatively or in addition, machine-readable media or any part thereof can be integrated into a processor, such as with a cache and / or a general register stack. Although the various components discussed can be described as having specific locations, such as local components, they can also be configured in various ways, such as certain components being configured as a part of a distributed computing system.

[0188] The processing system can be configured as a general processing system having one or more microprocessors providing processor functionality and an external memory providing at least a portion of a machine-readable medium, all of which are linked together with other supporting circuit systems through an external bus architecture. Alternatively, the processing system may include one or more neuromorphic processors for implementing the described neuron model and neural system model. As another alternative, the processing system may be implemented with an application specific integrated circuit (ASIC) having a processor, a bus interface, a user interface, a supporting circuit, and at least a portion of a machine-readable medium integrated in a single chip, or with one or more field programmable gate arrays (FPGAs), programmable logic devices (PLDs), controllers, state machines, gated logic components, discrete hardware components, or any other suitable circuits, or circuits capable of performing the various functionalities described throughout this disclosure. Those skilled in the art will recognize how to best implement the functionality of the processing system depending on the specific application and the overall design constraints imposed on the entire system.

[0189] The machine-readable medium may include multiple software modules. These software modules include instructions that cause the processing system to perform various functions when executed by the processor. The software module may include a sending module and a receiving module. Each software module may reside in a single storage device or be distributed across multiple storage devices. By way of example, when a triggering event occurs, the software module may be loaded from a hard drive into a RAM. During the execution of the software module, the processor may load some of the instructions into a cache to increase access speed. Then one or more cache lines may be loaded into a general register stack for execution by the processor. When the functionality of a software module is mentioned below, it will be understood that such functionality is implemented by the processor when executing instructions from the software module. In addition, it should be understood that various aspects of the present disclosure produce improvements in the functionality of processors, computers, machines, or other systems that implement such aspects.

[0190] If implemented in software, each function may be stored as one or more instructions or codes on or sent through a computer-readable medium. Computer-readable media include both computer storage media and communication media, including any media that facilitates the transmission of a computer program from one place to another. Storage media can be any available media that can be accessed by a computer. By way of example and not limitation, such computer-readable media may include RAM, ROM, EEPROM, CD-ROM or other optical disk storage devices, disk storage devices or other magnetic storage devices, or any other media that can be used to carry or store the desired program code in the form of instructions or data structures and can be accessed by a computer. In addition, any connection is also appropriately referred to as a computer-readable medium. For example, if the software is sent from a website, server, or other remote source using a coaxial cable, optical cable, twisted pair, digital subscriber line (DSL), or wireless technologies such as infrared (IR), radio, and microwaves, the coaxial cable, optical cable, twisted pair, DSL, or wireless technologies such as infrared, radio, and microwaves are included in the definition of the medium. As used, disks and optical disks include compact disks (CDs), laser disks, optical disks, digital versatile disks (DVDs), floppy disks, and optical disks. Optical disks, where magnetic disks typically reproduce data magnetically, and optical disks reproduce data optically with lasers. Thus, in some aspects, computer-readable media may include non-transitory computer-readable media (e.g., tangible media). Additionally, for other aspects, computer-readable media may include transient computer-readable media (e.g., signals). Combinations of the above should also be included within the scope of computer-readable media.

[0191] Thus, some aspects may include a computer program product for performing the operations presented. For example, such a computer program product may include a computer-readable medium having stored (and / or encoded) thereon instructions that can be executed by one or more processors to perform the described operations. For some aspects, the computer program product may include packaging materials.

[0192] In addition, it should be understood that the modules and / or other appropriate components for performing the described methods and techniques can be downloaded and / or otherwise obtained by the user terminal and / or base station where applicable. For example, such a device can be coupled to a server to facilitate the transmission of the components for performing the described methods. Alternatively, the various methods described can be provided via a storage component (e.g., RAM, ROM, a physical storage medium such as a compact disc (CD) or a floppy disk) so that once the storage component is coupled to or provided to the device, the user terminal and / or base station can obtain the various methods. In addition, any other suitable technology suitable for providing the described methods and techniques to the device can be used.

[0193] It is to be understood that the claims are not limited to the precise configuration and components illustrated above. Various modifications, changes and variations may be made in the arrangement, operation and details of the methods and apparatus described above without departing from the scope of the claims.

Claims

1. A device, comprising: at least one memory; and at least one processor, the at least one processor coupled to the at least one memory, the at least one processor configured to: Receiving input via an artificial neural network (ANN); Representing the input as a graph comprising a plurality of nodes connected by edges; determining, via the ANN, a graph cut between source nodes and sink nodes associated with the input by solving a quadratic process with equality constraints; as well as A subset of the inputs is processed via the ANN based on the graph cuts to generate predictions. 2 . The apparatus of claim 1 , wherein the graph cut segments the input such that the weight of the cross-edges between segments is minimized.

3. The apparatus of claim 1, wherein the input is an image or a feature grid. The apparatus of claim 3 , wherein the plurality of nodes correspond to pixels of the image or the feature grid.

5. The apparatus of claim 3, wherein the feature grid corresponds to image features of a scene including one or more objects in the image, and the at least one processor is further configured to divide the image features of the scene among the one or more objects. The apparatus of claim 1 , wherein the graph cut is differentiable.

7. The apparatus of claim 1, wherein the ANN comprises one or more differentiable optimization layers.

8. A processor-implemented method executed by at least one processor, the processor-implemented method comprising: Receiving input via an artificial neural network (ANN); Representing the input as a graph comprising a plurality of nodes connected by edges; determining, via the ANN, a graph cut between source nodes and sink nodes associated with the input by solving a quadratic process with equality constraints; as well as A subset of the inputs is processed via the ANN based on the graph cuts to generate predictions.

9. The processor-implemented method of claim 8, wherein the graph cut partitions the input such that the weight of crossing edges between segments is minimized.

10. The processor-implemented method of claim 8, wherein the input is an image or a feature grid.

11. The processor-implemented method of claim 10, wherein the plurality of nodes correspond to pixels of the image or the feature grid.

12. The processor-implemented method of claim 10, wherein the feature grid corresponds to image features of a scene including one or more objects in the image, and the processor-implemented method further comprises dividing the image features of the scene among the one or more objects.

13. The processor-implemented method of claim 8, wherein the graph cut is differentiable.

14. The processor-implemented method of claim 8, wherein the ANN comprises one or more differentiable optimization layers.

15. A non-transitory computer readable medium having program code recorded thereon, the program code being executed by at least one processor and comprising: Program code for receiving input via an artificial neural network (ANN); program code for representing the input as a graph, the graph comprising a plurality of nodes connected by edges; program code for determining, via the ANN, a graph cut between source nodes and sink nodes associated with the input by solving a quadratic process with equality constraints; and Program code for processing, via the ANN, a subset of the inputs based on the graph cuts to generate predictions.

16. The non-transitory computer readable medium of claim 15, wherein the graph cut segments the input such that a weight of a cross edge between segments is minimized.

17. The non-transitory computer readable medium of claim 15, wherein the input is an image or a feature grid.

18. The non-transitory computer readable medium of claim 17, wherein the plurality of nodes correspond to pixels of the image or the feature grid.

19. The non-transitory computer-readable medium of claim 17, wherein the feature grid corresponds to image features of a scene including one or more objects in the image, and the program code further comprises program code for dividing the image features of the scene among the one or more objects.

20. The non-transitory computer readable medium of claim 15, wherein the graph cut is differentiable.

21. The non-transitory computer readable medium of claim 15, wherein the ANN comprises one or more differentiable optimization layers.

22. A device, comprising: means for receiving input via an artificial neural network (ANN); means for representing the input as a graph, the graph comprising a plurality of nodes connected by edges; means for determining, via the ANN, a graph cut between source nodes and sink nodes associated with the input by solving a quadratic process with equality constraints; and Means for processing, via the ANN, a subset of the inputs based on the graph cut to generate predictions.

23. The apparatus of claim 22, wherein the graph cut segments the input such that the weight of the cross-edges between segments is minimized.

24. The apparatus of claim 22, wherein the input is an image or a feature grid.

25. The apparatus of claim 24, wherein the plurality of nodes correspond to pixels of the image or the feature grid.

26. The apparatus of claim 24, wherein the feature grid corresponds to image features of a scene including one or more objects in the image, and the apparatus further comprises means for dividing the image features of the scene among the one or more objects.

27. The apparatus of claim 22, wherein the graph cut is differentiable.

28. The apparatus of claim 22, wherein the ANN comprises one or more differentiable optimization layers.

Images