Preprocessing for deep neural network compilation using graph neural network

Abstract

A processor-implemented method for preprocessing of deep neural network compilation includes receiving a representation of an artificial neural network (ANN) model. The ANN includes a plurality of nodes coupled by edges. Location information for each node of the ANN is determined. An operator embedding is generated based on the location information to represent an operator of the ANN model in an embedding space. A graph neural network (GNN) processes operator embedding according to the learned distance metric and based on the location information to generate a graph embedding corresponding to the ANN model. The GNN determines a set of hyper-parameters of the ANN model based on graph embedding.

Description

[0001] Cross Reference to Related Applications

[0002] This application claims priority to U.S. Patent Application No. 18 / 500,014, filed November 1, 2023, entitled “PRE-PROCESSING FOR DEEP NEURAL NETWORK COMPILATION USING GRAPH NEURAL NETWORKS,” which is a continuation-in-part of U.S. Patent Application No. 18 / 330,253, filed June 6, 2023, entitled “PRE-PROCESSING FOR DEEP NEURAL NETWORK COMPILATION USING GRAPH NEURAL NETWORKS,” the disclosures of which are expressly incorporated by reference in their entirety. TECHNICAL FIELD

[0003] Aspects of the present disclosure generally relate to pre-processing for deep neural network compilation using graph neural networks. BACKGROUND

[0004] Artificial neural networks can include interconnected groups of artificial neurons (e.g., neuron models). Artificial neural networks can be a computing device or represent a method to be performed by a computing device. Artificial neural networks, including feed-forward neural networks, convolutional neural networks (CNNs), transformers, graph neural networks (GNNs), recurrent neural networks (RNNs), and the like, have many applications. In particular, these neural network architectures are used for various technologies, such as image recognition, speech recognition, acoustic scene classification, keyword spotting, autonomous driving, and other classification tasks.

[0005] Given the many useful applications of neural networks, there is an increasing demand to use neural networks on edge devices, such as smartphones, as well as in cloud computing. However, edge devices have limited computational resources, and generalized models can utilize more complex networks and more computations. One approach to address such issues is to optimize neural network model workloads on hardware accelerators. However, many conventional approaches for optimizing such workloads on hardware accelerators can depend on prior knowledge of the configuration of the model architecture.

[0006] Conventional graph neural network (GNN)-based techniques can generate embeddings of neural networks, but can not be able to automatically identify models. Such conventional GNN-based techniques can also not be able to pass embeddings of structural and semantic information of a computational task to downstream tasks without retraining. SUMMARY

[0007] The present disclosure is set forth in independent claims. Some aspects of the present disclosure are described in dependent claims.

[0008] In some aspects of the present disclosure, a processor-implemented method for preprocessing of deep neural network compilation includes receiving a representation of an artificial neural network (ANN) model. The ANN includes a plurality of nodes coupled by edges. The method further includes determining position information for each node of the ANN model. The method additionally includes generating operator embeddings to represent operators of the ANN model in an embedding space based on the position information for each node. The method further includes processing, by a graph neural network (GNN), the operator embeddings to generate a graph embedding corresponding to the ANN model according to a learned distance metric and based on the position information. The method also includes determining, by the GNN, a set of hyperparameters for the ANN model based on the graph embedding.

[0009] Various aspects of the present disclosure relate to an apparatus comprising means for receiving a representation of an artificial neural network (ANN) model. The ANN includes a plurality of nodes coupled by edges. The apparatus further includes means for determining position information for each node of the ANN model. The apparatus additionally includes means for generating operator embeddings to represent operators of the ANN model in an embedding space based on the position information for each node. The apparatus further includes means for processing, by a graph neural network (GNN), the operator embeddings to generate a graph embedding corresponding to the ANN model according to a learned distance metric and based on the position information. The apparatus also includes means for determining, by the GNN, a set of hyperparameters for the ANN model based on the graph embedding.

[0010] In some aspects of the present disclosure, a non-transitory computer-readable medium having non-transitory program code recorded thereon is disclosed. The program code is executed by a processor and includes program code to receive a representation of an artificial neural network (ANN) model. The ANN includes a plurality of nodes coupled by edges. The program code further includes program code to determine position information for each node of the ANN model. The program code additionally includes program code to generate operator embeddings to represent operators of the ANN model in an embedding space based on the position information for each node. The program code further includes program code to process, by a graph neural network (GNN), the operator embeddings to generate a graph embedding corresponding to the ANN model according to a learned distance metric and based on the position information. The program code also includes program code to determine, by the GNN, a set of hyperparameters for the ANN model based on the graph embedding.

[0011] Various aspects of this disclosure relate to an apparatus having a memory and one or more processors coupled to the memory. The processors are configured to receive a representation of an artificial neural network (ANN) model. The ANN includes a plurality of nodes coupled by edges. The processors are further configured to determine location information for each node of the ANN model. The processors are further configured to generate operator embeddings based on the location information of each node to represent operators of the ANN model in an embedding space. The processors are additionally configured to process the operator embeddings by a graph neural network (GNN) based on a learned distance metric and the location information to generate graph embeddings corresponding to the ANN model. Furthermore, the processors are configured to determine a set of hyperparameters of the ANN model by the GNN based on the graph embeddings.

[0012] Additional features and advantages of this disclosure will be described below. Those skilled in the art will understand that this disclosure can be readily used as the basis for modifying or designing other structures for implementing the same purposes as this disclosure. Those skilled in the art will also recognize that such equivalent constructions do not depart from the teachings of this disclosure as set forth in the appended claims. Novel features considered characteristic of this disclosure, in both their organization and manner of operation, along with further objects and advantages, will be better understood when considered in conjunction with the accompanying drawings. However, it is to be clearly understood that each drawing is provided for illustrative and descriptive purposes only and is not intended to be a definition of a limitation of this disclosure. Attached Figure Description

[0013] The features, substance, and advantages of this disclosure will become more apparent when understood in conjunction with the accompanying drawings, in which the same reference numerals are consistently used for identification.

[0014] Figure 1 Examples of specific implementations of neural networks using a system-on-a-chip (SoC) (including a general-purpose processor) according to certain aspects of this disclosure are illustrated.

[0015] Figure 2A , Figure 2B and Figure 2C These are illustrations of neural networks according to various aspects of this disclosure.

[0016] Figure 2D This is a diagram illustrating exemplary deep convolutional networks (DCNs) according to various aspects of this disclosure.

[0017] Figure 3 This is a block diagram illustrating exemplary deep convolutional networks (DCNs) according to various aspects of this disclosure.

[0018] Figure 4 This is a block diagram illustrating exemplary software architectures that enable modularization of artificial intelligence (AI) functions according to various aspects of this disclosure.

[0019] Figure 5 This is a block diagram illustrating an example architecture for preprocessing for deep neural network compilation using graph neural networks according to various aspects of this disclosure.

[0020] Figure 6 This is a block diagram illustrating an example architecture of a metric learning module for training a graph neural network (GNN) using metric learning, according to various aspects of this disclosure.

[0021] Figure 7 This is a block diagram illustrating an example architecture for training a GNN using a relative size preservation module based on various aspects of this disclosure.

[0022] Figure 8 This is a block diagram illustrating the use of reconstruction loss to train a GNN according to various aspects of this disclosure.

[0023] Figure 9 This is a block diagram illustrating example architectures for training GNNs according to various aspects of this disclosure.

[0024] Figure 10 This is a block diagram illustrating an example architecture for training a GNN using node location information according to various aspects of this disclosure.

[0025] Figure 11A and Figure 11B This is a block diagram illustrating examples of shortest distance positioning (SDP) and longest distance positioning (LDP) according to various aspects of this disclosure.

[0026] Figure 12 This is a flowchart illustrating a method for a processor implementation of preprocessing for deep neural network compilation using graph neural networks, according to various aspects of this disclosure. Detailed Implementation

[0027] The detailed description following, taken in conjunction with the accompanying drawings, is intended as a description of various configurations and not as representing only configurations in which the described concepts can be practiced. To provide a comprehensive understanding of the various concepts, the detailed description includes specific details. However, it will be apparent to those skilled in the art that these concepts can be practiced without these specific details. In some instances, to avoid obscuring such concepts, well-known structures and components are shown in block diagram form.

[0028] Based on the teachings, those skilled in the art will recognize that the scope of this disclosure is intended to cover any aspect of this disclosure, whether implemented independently of or in combination with any other aspect of this disclosure. For example, an apparatus or method may be implemented using any number of the aspects described. Furthermore, the scope of this disclosure is intended to cover such apparatuses or methods practiced using other structures, functionalities, or structures and functionalities that complement or differ from the various aspects of this disclosure described. It should be understood that any aspect of this disclosure may be embodied by one or more elements of the claims.

[0029] The word “exemplary” is used to mean “serving as an example, instance, or illustration.” Any aspect described as “exemplary” need not be construed as superior to or better than other aspects.

[0030] While specific aspects have been described, numerous variations and substitutions of these aspects fall within the scope of this disclosure. Although some benefits and advantages of preferred aspects have been mentioned, the scope of this disclosure is not intended to be limited to a particular benefit, use, or purpose. Rather, aspects of this disclosure are intended to be broadly applicable to different technologies, system configurations, networks, and protocols, some of which are illustrated by way of example in the accompanying drawings and the following description of preferred aspects. The detailed description and drawings are merely illustrative and not limiting of this disclosure, the scope of which is defined by the appended claims and their equivalents.

[0031] Deep neural networks have a large footprint and can suffer from significant latency and power consumption. One approach to reduce latency in neural networks is through machine learning accelerators. Machine learning accelerators consist of dedicated processors designed to accelerate machine learning computations, such as multiplication and accumulation operations in matrix-matrix and matrix-vector operations. To utilize machine learning accelerators, machine learning compilers provide a mapping from the trained machine learning model to a given machine learning accelerator. The machine learning compiler aims to optimize this mapping from the machine learning model to the accelerator.

[0032] Conventional approaches to optimizing artificial intelligence (AI) workloads on hardware (HW) accelerators rely on prior knowledge of the optimal configuration of the model architecture or brute-force tools. These conventional methods may not be able to scale to any model or architecture to automatically identify the closest match to the known optimal configuration without requiring retraining of the AI-driven tool. While conventional graph neural network (GNN)-based techniques can generate embeddings of neural networks, they cannot automatically identify the model.

[0033] Additionally, the embeddings of structural and semantic information of the computation graph may not be able to be passed to downstream tasks without retraining. Many conventional methods can only generate embeddings for specific tasks.

[0034] Therefore, aspects of this disclosure relate to generating embeddings that capture the semantic (operator) and structural information of deep neural networks. Each generated embedding can be unique and can be beneficial for backbone and architecture classification and understanding, as well as similarity searching of computation graphs. Additional use cases include recommendations for neural network architecture changes and / or adaptations and optimal performance parameters.

[0035] According to various aspects of this disclosure, graph machine learning methods can be coupled with metric learning to produce model embeddings. Model embeddings can be unique for each model. Models with similar network architectures can cluster closer together in the embedding space, unlike graphs (models) which can be more distant. Embeddings can preserve semantic operator characteristics while transforming a graph into the embedding space. In some aspects, the distance between embeddings of models with similar substructures belonging to the same network architecture can be proportional to their relative size (e.g., the number of operators).

[0036] Hybrid methods can be used to train networks based on different objective functions. The proposed method is an open-set solution, so the same trained model can be used for unseen neural network architectures.

[0037] The described technique provides improvements in performance tuning time by accurately representing neural networks in the embedding space. Additionally, the described technique can advantageously provide good performance in separating embeddings of models belonging to different classes. However, capturing intra-class separation can be challenging in some aspects. That is, it may be difficult to distinguish embeddings of neural network models within the same class that are significantly close to each other. Furthermore, graph embeddings can suffer from information over-compression in some aspects, where deep neural network models (such as transformer models) have similar embeddings to shallower neural network models.

[0038] To address these additional challenges, aspects of this disclosure also relate to generating model embeddings based on sinusoidal localization information. The sinusoidal localization information can be computed and combined with operator embeddings.

[0039] Specific aspects of the subject matter described in this disclosure can be implemented to achieve one or more of the following potential advantages. In some examples, the described techniques can also improve backbone and architecture classification, as well as recommendations for architecture changes and neural network parameter selection, to improve accuracy and reduce memory consumption and latency.

[0040] Figure 1An example implementation of a System-on-a-Chip (SOC) 100 is illustrated, which may include a Central Processing Unit (CPU) 102 or a multi-core CPU configured to perform preprocessing for compiling a deep neural network. Variables (e.g., neural signals and synaptic weights), system parameters associated with the computing device (e.g., a neural network with weights), latency, frequency slot information, and task information may be stored in a memory block associated with a Neural Processing Unit (NPU) 108, a memory block associated with the CPU 102, a memory block associated with a Graphics Processing Unit (GPU) 104, a memory block associated with a Digital Signal Processor (DSP) 106, a memory block 118, or may be distributed across multiple blocks. Instructions executed at the CPU 102 may be loaded from the program memory associated with the CPU 102 or from memory block 118.

[0041] SOC 100 may also include additional processing blocks tailored for specific functions, such as GPU 104, DSP 106, connectivity block 110 (which may include fifth-generation (5G) connectivity, fourth-generation LTE (4G) connectivity, Wi-Fi connectivity, USB connectivity, Bluetooth connectivity, etc.), and multimedia processor 112 capable of, for example, detecting and recognizing gestures. In one specific implementation, NPU 108 is implemented within CPU 102, DSP 106, and / or GPU 104. SOC 100 may also include sensor processor 114, image signal processor (ISP) 116, and / or navigation module 120, which may include a global positioning system.

[0042] The SOC 100 may be based on the ARM instruction set. In one aspect of this disclosure, instructions loaded into the general-purpose processor 102 may include code for receiving a representation of an artificial neural network (ANN) model. The ANN includes multiple nodes coupled at the edges. The general-purpose processor 102 may also include code for determining location information for each node of the ANN model. The general-purpose processor 102 may also include code for generating operator embeddings based on the location information of each node to represent operators of the ANN model in an embedding space. The general-purpose processor 102 may additionally include code for processing the operator embeddings by a graph neural network (GNN) based on a learned distance metric and location information to generate graph embeddings corresponding to the ANN model. The general-purpose processor 102 may also include code for determining a set of hyperparameters of the ANN model by the GNN based on the graph embeddings.

[0043] Deep learning architectures perform object recognition tasks by learning to represent inputs at progressively higher levels of abstraction in each layer, thereby constructing useful feature representations of the input data. In this way, deep learning addresses a major bottleneck of traditional machine learning. Before deep learning, machine learning methods for object recognition problems often relied heavily on human-designed features, possibly in conjunction with shallow classifiers. Shallow classifiers could be two-class linear classifiers, where a weighted sum of feature vector components is compared to a threshold to predict which class the input belongs to. Human-designed features could be templates or kernels customized for a specific problem domain by engineers with domain expertise. In contrast, while deep learning architectures can learn to represent features similar to those that human engineers might design, this requires training. Furthermore, deep networks can learn to represent and recognize novel types of features that humans might not have considered.

[0044] Deep learning architectures can learn hierarchical structures of features. For example, if presented with visual data, the first layer can learn to recognize relatively simple features in the input stream, such as edges. In another example, if presented with auditory data, the first layer can learn to recognize spectral power at specific frequencies. The second layer, taking the output of the first layer as input, can learn to recognize combinations of features, such as simple shapes in visual data or combinations of sounds in auditory data. For example, higher layers can learn to represent complex shapes in visual data or words in auditory data. Even higher layers can learn to recognize common visual objects or spoken phrases.

[0045] Deep learning architectures perform particularly well when applied to problems with a natural hierarchical structure. For example, the classification of motorized vehicles can benefit from first learning to identify features such as wheels, windshields, and others. These features can then be combined in different ways at higher levels to identify cars, trucks, and airplanes.

[0046] Neural networks can be designed to have multiple connectivity patterns. In feedforward networks, information is passed from lower layers to higher layers, where each neuron in a given layer communicates with neurons in higher layers. As described above, hierarchical representations can be built in successive layers of a feedforward network. Neural networks can also have recurrent or feedback (also known as top-down) connections. In recurrent connections, the output from a neuron in a given layer can be passed to another neuron in the same layer. Recurrent architectures can help identify patterns across more than one block of input data that is sequentially delivered to the neural network. Connections from neurons in a given layer to neurons in lower layers are called feedback (or top-down) connections. Networks with many feedback connections can be helpful when the recognition of higher-level concepts can aid in discerning specific lower-level features of the input.

[0047] The connections between layers in a neural network can be fully connected or locally connected. Figure 2A An example of a fully connected neural network 202 is shown. In the fully connected neural network 202, neurons in the first layer can pass their outputs to each neuron in the second layer, so that each neuron in the second layer will receive inputs from each neuron in the first layer. Figure 2B An example of a locally connected neural network 204 is shown. In the locally connected neural network 204, neurons in the first layer can connect to a finite number of neurons in the second layer. More generally, the locally connected layers of the locally connected neural network 204 can be configured such that each neuron in the layer will have the same or similar connectivity pattern, but the connection strength can have different values ​​(e.g., 210, 212, 214, and 216). The connectivity pattern of locally connected layers can produce spatially different receptive fields in higher layers because higher-layer neurons in a given region can receive inputs that are tuned to the characteristics of a restricted portion of the total input to the network through training.

[0048] An example of a locally connected neural network is a convolutional neural network. Figure 2C An example of a convolutional neural network 206 is shown. Convolutional neural network 206 can be configured such that the connection strength associated with the input for each neuron in the second layer is shared (e.g., 208). Convolutional neural networks may be well-suited for problems where the spatial location of the input is meaningful.

[0049] One type of convolutional neural network is the deep convolutional network (DCN). Figure 2D A detailed example of a DCN 200 designed to recognize visual features from an image 226 input from an image capture device 230 (such as an in-vehicle camera) is provided. The DCN 200 in this example can be trained to identify traffic signs and the numbers provided on them. Of course, the DCN 200 can be trained for other tasks, such as identifying lane markings or traffic lights.

[0050] Supervised learning can be used to train the DCN 200. During training, an image (such as image 226 of a speed limit sign) can be presented to the DCN 200, and then forward passes can be computed to produce output 222. The DCN 200 may include a feature extraction part and a classification part. Upon receiving image 226, convolutional layer 232 can apply a convolutional kernel (not shown) to image 226 to generate a first set 218 of feature maps. As an example, the convolutional kernel used for convolutional layer 232 may be a 5x5 kernel that generates 28x28 feature maps. In this example, because four different feature maps are generated in the first set of feature maps 218, four different convolutional kernels are applied to image 226 at convolutional layer 232. Convolutional kernels may also be referred to as filters or convolutional filters.

[0051] The first set of feature maps 218 can be subsampled by a max-pooling layer (not shown) to generate a second set of feature maps 220. The max-pooling layer reduces the size of the first set of feature maps 218. That is, the size of the second set of feature maps 220 (e.g., 14x14) is smaller than the size of the first set of feature maps 218 (e.g., 28x28). The reduced size provides similar information to subsequent layers while reducing memory consumption. The second set of feature maps 220 can be further convolved via one or more subsequent convolutional layers (not shown) to generate one or more subsequent sets of feature maps (not shown).

[0052] exist Figure 2D In the example, the second set of feature maps 220 is convolved to generate a first feature vector 224. Furthermore, the first feature vector 224 is further convolved to generate a second feature vector 228. Each feature of the second feature vector 228 may include a number corresponding to a possible feature of image 226, such as "sign", "60", and "100". A softmax function (not shown) can convert the numbers in the second feature vector 228 into probabilities. Thus, the output 222 of DCN 200 can be the probability that image 226 includes one or more features.

[0053] In this example, the probabilities for "sign" and "60" in output 222 are higher than the probabilities for other numbers in output 222 (such as "30", "40", "50", "70", "80", "90", and "100"). Before training, output 222 generated by DCN 200 may be incorrect. Therefore, the error between output 222 and the target output can be calculated. The target output is the ground truth value of image 226 (e.g., "sign" and "60"). The weights of DCN 200 can then be adjusted so that output 222 of DCN 200 is more closely aligned with the target output.

[0054] To adjust the weights, the learning algorithm computes the gradient vector of the weights. The gradient indicates by how much the error will increase or decrease as the weights are adjusted. At the top layers, the gradient corresponds directly to the values ​​of the weights connecting the activated neurons in the penultimate layer to the neurons in the output layer. In lower layers, the gradient depends on the values ​​of the weights and the error gradient computed in the higher layers. The weights can then be adjusted to reduce the error. This method of adjusting weights is called "backpropagation" because it involves "passing backward" through the neural network.

[0055] In practice, the error gradient of the weights can be calculated using a small number of examples to make the calculated gradient approximate the true error gradient. This approximation method is called stochastic gradient descent. Stochastic gradient descent can be repeated until the achievable error rate of the entire system stops decreasing or until the error rate reaches a target level. After learning, DCN 200 can present a new picture, and the forward pass of DCN 200 can produce an output that can be considered an inference or prediction of DCN 200.

[0056] Deep Belief Networks (DBNs) are probabilistic models that include multiple layers of hidden nodes. DBNs can be used to extract hierarchical representations of training datasets. DBNs are obtained by stacking layers of Restricted Boltzmann Machines (RBMs). An RBM is a type of artificial neural network that learns a probability distribution from a set of inputs. Because RBMs can learn a probability distribution without information about the class each input should be classified into, they are often used for unsupervised learning. Using a hybrid paradigm of supervised and unsupervised learning, the bottom RBM of a DBN can be trained unsupervised and used as a feature extractor, while the top RBM can be trained supervisedly (on the joint distribution of inputs from the previous layer and the target class) and used as a classifier.

[0057] Deep convolutional networks (DCNs) are convolutional networks configured with additional pooling and normalization layers. DCNs have achieved state-of-the-art performance on many tasks. DCNs can be trained using supervised learning, where both the input and output targets are known for many paradigms and are used to modify the network's weights using gradient descent.

[0058] DCNs can be feedforward networks. Furthermore, as described above, connections from neurons in the first layer of a DCN to a set of neurons in the next higher layer are shared across neurons in the first layer. The feedforward and shared connections of a DCN can be used for fast processing. For example, the computational cost of a DCN may be much smaller than that of a similarly sized neural network that includes recurrent or feedback connections.

[0059] The processing at each layer of a convolutional network can be thought of as a spatially invariant template or base projection. If the input is first decomposed into multiple channels, such as the red, green, and blue channels of a color image, then a convolutional network trained on that input can be considered three-dimensional, with two spatial dimensions along the image's axes and a third dimension capturing color information. The output of the convolutional connections can be viewed as forming a feature map in the next layer, where each element in the feature map (e.g., 220) receives input from a range of neurons in the previous layer (e.g., feature map 218) and from each of the multiple channels. The values ​​in the feature map can be further processed with nonlinearities (such as correction, max(0,x)). Values ​​from neighboring neurons can be further pooled, which corresponds to downsampling and provides additional local invariance and dimensionality reduction. Normalization corresponding to whitening can also be applied through lateral inhibition between neurons in the feature map.

[0060] Figure 3 This is a block diagram illustrating a DCN 350. A DCN 350 can include multiple layers of different types based on connectivity and weight sharing. For example... Figure 3 As shown, the DCN 350 includes convolutional blocks 354A and 354B. Each convolutional block in 354A and 354B can be configured with a convolutional layer (CONV) 356, a normalization layer (LNorm) 358, and a max pooling layer (MAX POOL) 360. Although only two convolutional blocks from 354A and 354B are shown, this disclosure is not limited thereto, and any number of convolutional blocks 354A and 354B can be included in the DCN 350 according to design preferences.

[0061] Convolutional layer 356 may include one or more convolutional filters that can be applied to the input data to generate feature maps. Normalization layer 358 can normalize the output of the convolutional filters. For example, normalization layer 358 can provide whitening or lateral suppression. Max pooling layer 360 can provide spatial downsampling aggregation to achieve local invariance and dimensionality reduction.

[0062] For example, a parallel filter bank of a DCN can be loaded into a SOC 100 (e.g., Figure 1 The CPU 102 or GPU 104 of the SOC 100 can be used to achieve high performance and low power consumption. In an alternative implementation, a parallel filter bank can be loaded onto the DSP 106 or ISP 116 of the SOC 100. Additionally, the DCN 350 can access other processing blocks that may exist on the SOC 100, such as the sensor processor 114 and navigation module 120, which are dedicated to sensors and navigation, respectively.

[0063] The DCN 350 may also include one or more fully connected layers 362 (FC1 and FC2). The DCN 350 may also include logistic regression (LR) layers 364. Weights (not shown) to be updated are located between each of the layers 356, 358, 360, 362, and 364 of the DCN 350. The output of each layer (e.g., 356, 358, 360, 362, and 364) can be used as input to the next layer in the DCN 350 to learn hierarchical feature representations from the input data 352 (e.g., images, audio, video, sensor data, and / or other input data) supplied at the first convolutional block in convolutional block 354A. The output of the DCN 350 is a classification score 366 of the input data 352. The classification score 366 may be a set of probabilities, where each probability is the probability that the input data includes a feature from the feature set.

[0064] Figure 4 This is a block diagram illustrating an exemplary software architecture 400 that allows for modularization of artificial intelligence (AI) functionality. According to various aspects of this disclosure, by using architecture 400, various processing blocks of an SOC 420 (e.g., CPU 422, DSP 424, GPU 426, and / or NPU 428) can be designed to support preprocessing applications for deep neural network compilation for AI application 402. Architecture 400 can, for example, be included in a computing device such as a smartphone.

[0065] AI application 402 can be configured to invoke functions defined in user space 404, which may, for example, provide the detection and recognition of a scene indicating the current location of the computing device (including architecture 400). For example, AI application 402 may configure microphones and cameras differently depending on whether the recognized scene is an office, lecture hall, restaurant, or outdoor environment such as a lake. AI application 402 may make requests to compiled program code associated with libraries defined in the AI ​​Function Application Programming Interface (API) 406. This request may ultimately rely on the output of a deep neural network configured to provide inferred responses based on, for example, video and location data.

[0066] The runtime engine 408 (which may be compiled code of the runtime framework) may further be accessible to the AI ​​application 402. The AI ​​application 402 may cause the runtime engine 408 to request inferences, for example, at specific time intervals or triggered by events detected by the user interface of the AI ​​application 402. Upon causing the runtime engine 408 to provide an inference response, the runtime engine may then signal to the operating system (OS) space 410 running on the SOC 420, such as kernel 412. In some examples, kernel 412 may be a LINUX kernel. The operating system may then enable sequential quantization relaxation to be performed on the CPU 422, DSP 424, GPU 426, NPU 428, or some combination thereof. The CPU 422 may be directly accessible by the operating system, while other processing blocks may be accessed via drivers, such as drivers 414, 416, or 418 for the DSP 424, GPU 426, or NPU 428, respectively. In an exemplary example, the deep neural network may be configured to run on a combination of processing blocks such as CPU 422, DSP 424 and GPU 426, or on NPU 428.

[0067] Various aspects of this disclosure relate to using graph neural networks for preprocessing for compiling deep neural networks. Various aspects of this disclosure relate to using graph neural networks (GNNs) to generate position-induced graph embeddings for neural network models.

[0068] Figure 5 This is a block diagram illustrating an example architecture 500 for preprocessing for deep neural network compilation using graph neural networks according to various aspects of this disclosure. Example architecture 500 may include a preprocessing module 504, an operator embedding module 506, and a GNN 508. The graph neural network may be, for example, a graph convolutional neural network (GCN), a graph autoencoder network, a recurrent GNN, or other types of graph neural network models. GNN 508 may include a set of convolutional layers 510, activation layers 512a, 512b, and a max-pooling layer 514. For example, activation layers (e.g., 512a, 512b) may include rectified linear units (ReLU), Gaussian error linear units (GISU), or sigmoid function layers.

[0069] Additionally, the example architecture 500 may include a metric learning module 516, a relative size preservation module 518, and an operator reconstruction module 520. The example architecture 500 may accept an artificial neural network (ANN) model 502 as input. The ANN model 502 may include a residual neural network, a transformer neural network, a recurrent convolutional neural network (RCNN), an autoencoder, or other types of artificial neural networks. The ANN model 502 may be, for example, in the form of a computation graph, a matrix, or other forms. The example architecture 500 may be configured to receive ANN models 502 of varying sizes (e.g., the number of operators).

[0070] An ANN model 502 can be received at preprocessing module 504. Preprocessing module 504 extracts a set of operators from the ANN model 502. In some aspects, preprocessing module 504 can generate an adjacency matrix corresponding to the ANN model 502. The extracted operators are supplied to operator embedding module 506. Operator embedding module 506 extracts features of the operators and generates representations of the operators of the ANN model 502 in a higher-dimensional space. The operator embeddings can be supplied to GNN 508.

[0071] GNN 508 can process operator embeddings to capture graph dependencies using messages passed between operators (e.g., nodes) in the graph. That is, GNN 508 can determine the relationships between operators and topology of ANN model 502 through message passing. For example, operator embeddings can be processed by convolutional layer 510 to extract features from the operator embeddings. Activation layers 512a, 512b and max-pooling layer 514 can be used to aggregate and process the extracted features to determine one or more edges or connections between operators and parameters (e.g., weights) associated with one or more edges. Therefore, GNN 508 can generate graph embeddings 524 corresponding to ANN model 502. GNN 508 can generate graph embeddings 524 corresponding to each ANN model 502 such that graph embeddings 524 are unique (different from graph embeddings of other ANN models). Furthermore, GNN 508 can generate graph embeddings (e.g., 524) that preserve semantic operator characteristics when transforming the computation graph of ANN model 502 into the embedding space. Each generated graph embedding can be stored in a graph embedding database along with the set of training samples. The database can be maintained in memory (such as...). Figure 1 (in the memory 118 or another data storage device).

[0072] The metric learning module 516 can be used to train the GNN 508 to generate graph embeddings. The metric learning module 516 implements metric learning techniques. Metric learning is the task of learning a distance function over objects. According to various aspects of this disclosure, the GNN 508 can be trained to cluster positive samples to the vicinity of anchors. That is, similar samples (e.g., ANNs) can be clustered together in the embedding space, while dissimilar samples can be spaced further apart in the embedding space. Metric learning can be performed based on a triplet loss, which is given by the following equation:

[0073]

[0074] in It is a distance function. These are samples selected from the training dataset (e.g., an ANN model). Indicates and Similar (or identical) ANN models, Indicates and Different ANN models, and Indicates belonging to different categories or The distance between the two classes of ANN models in the sample.

[0075] In some respects, the relative size preservation module 518 can also be used to train the GNN 508. The relative size preservation module 518 can be used to train the GNN 508 such that the distance between graph embeddings is proportional to the relative size of the ANN model according to the RelativeSizePreservationLoss objective, which is given by the following equation:

[0076]

[0077] in This represents the estimated count of operators in an ANN model (e.g., 502), and Represents the actual count of operators in an ANN model (e.g., 502).

[0078] By training GNN 508 based on RelativeSizePreservationLoss, GNN 508 can maintain the size (e.g., number of operators) ratio between similar architectures. For example, in the case where ANN model 502 has an unseen architecture (no matching in the training dataset), GNN 508 can determine that this ANN model 502 has a substructure similar to the sample ANN. GNN 508 trained according to RelativeSizePreservationLoss can identify the ratio of such architectures to each other (e.g., ANN model 502 relative to the identified sample ANN). Therefore, GNN 508 can generate graph embeddings 524 for the unseen ANN model 502, which maintains the size ratio of the identification.

[0079] In some respects, the operator reconstruction module 520 can also be used to train the GNN 508 based on the reconstruction loss. The operator reconstruction module 520 can take the graph embedding 524 output by the GNN 508 as input. The operator reconstruction module 520 processes the graph embedding 524 to determine the accuracy (e.g., correspondence) of the graph embedding (524) to the ANN model 502. The accuracy of the graph embedding can be determined based on the operator reconstruction loss, which can be expressed as:

[0080]

[0081] in The label represents the operator, and This represents the probability of the operator.

[0082] In some respects, the metric learning module 516 can operate concurrently with the relative size preservation module 518 and / or the operator reconstruction module 520 during the training of the GNN 508. For example, each of the metric learning module 516, the relative size preservation module 518, and the operator reconstruction module 520 can concurrently receive graph embeddings and compute the corresponding loss, as shown in Equations 1 to 3. The corresponding losses can then be aggregated to produce a total loss.

[0083] After training the GNN 508, the metric learning module 516, the relative size preservation module 518, and the operator reconstruction module 520 can be removed, and the architecture 500 can be operated without such modules during inference.

[0084] Therefore, aspects of this disclosure can advantageously employ an open-set approach. That is, example architectures can be scaled to generate embeddings for unseen neural networks, thereby avoiding retraining.

[0085] Additionally, aspects of this disclosure can be trained to use computational graphs as inputs for downstream tasks with limited data. For example, graph embeddings can be used to identify the architecture or backbone of an input ANN to determine the fusion of operators and other downstream tasks within the ANN model.

[0086] Figure 6 This is a block diagram illustrating an example architecture 600 of a metric learning module 516 for training a GNN (e.g., 508) using metric learning, according to various aspects of this disclosure. Figure 6 As shown, the metric learning module 516 can be configured as a Siamese network for comparing sets of inputs.

[0087] The graph embeddings 524 from example architecture 500 can be learned in such a way that the generated embeddings 524 can be projected into a high-dimensional embedding space, allowing ANN-like nodes to be clustered and projected more closely together. On the other hand, dissimilar ANNs can be projected further apart from each other.

[0088] During training, graph embedding anchor 602 can be selected as a reference input by the miner 610. For example, anchor 602 can be selected through random sampling or other sampling techniques. Graph embedding positive examples 604 (e.g., the same identity as the anchor) and graph embedding negative examples 606 (e.g., different identities from the anchor) can be received as input by the metric learning module 516. Equation 1 shows... It can be used to minimize the distance between graph embedding anchor 602 and graph embedding positive example 604, while maximizing the distance between graph embedding anchor 602 and graph embedding negative example 606.

[0089] Figure 7 This is a block diagram illustrating an example architecture 700 for training a GNN (e.g., 508) using a relative size-preserving loss for a relative size-preserving module 518, according to various aspects of this disclosure. References Figure 7 The relative size preservation module 518 can receive a graph embedding 524 and a graph embedding norm 702. The relative size preservation module 518 can compare the graph embedding 524 with graph embeddings having similar substructures. The relative size preservation module 518 can determine the RelativeSizePreservationLoss (as shown in Equation 2) based on the graph embedding 524 and graph embeddings having similar substructures. The GNN (e.g., 508) can adapt each graph embedding in the graph embeddings based on the RelativeSizePreservationLoss, such that the norm of the graph embedding (e.g., 702) is proportional to the relative size of each graph embedding (e.g., based on the number of operators).

[0090] Therefore, by using a relative size preservation module 518 for training, a GNN (e.g., 508) can generate graph embeddings for ANNs with similar substructures (e.g., residual networks such as Resnet18, Resnet34, and Resnet101), such that the corresponding graph embeddings have individual values ​​and the distance between the corresponding graph embeddings can be linear.

[0091] Figure 8 This is a block diagram illustrating the use of reconstruction loss to train a GNN (e.g., 508) according to various aspects of this disclosure. References Figure 8 The operator reconstruction module 520 can receive graph embedding 524 from the example architecture 500.

[0092] In some respects, when generating graph embeddings 802, the GNN (e.g., 508) may lose some operator information. To reduce the occurrence of operator information loss, the operator reconstruction module 520 can be trained concurrently with the GNN (e.g., 508) to preserve the operator information in the graph embedding.

[0093] The operator reconstruction module 520 may include a classification network that uses a sigmoid activation function based on graph embedding 524 to estimate operators present in the ANN model (e.g., 502). The operator reconstruction module 520 may compare the estimated operators with the actual operators in the ANN model (e.g., 502) to compute... As shown in Equation 3. Using... GNNs (e.g., 508) can be adapted to graph embeddings of 524 to preserve operator information, such as information about unique and less frequent operators.

[0094] Figure 9 This is a block diagram illustrating an example architecture 900 for training a GNN (e.g., 508) according to various aspects of this disclosure. Reference Figure 9 It can receive an input graph (e.g., 502) from a training dataset (e.g., for a known ANN). The input graph (e.g., 502) can be preprocessed to generate operator embeddings that can be fed into the GNN 508. The GNN 508 can process the operator embeddings to generate graph embeddings 524.

[0095] Graph embedding 524 can be supplied to one or more of the metric learning module 516, the relative size preservation module 518, and the operator reconstruction module 520, which can be used to train the GNN 508. Figure 9 In the example, the calculation by operator reconstruction module 520 This can be used to calculate the binary cross-entropy (BCE) loss 902. The BCE loss 902 can be compared with the loss calculated by the metric learning module 516. This loss is combined with the RelativeSizePreservationLoss calculated by the RelativeSizePreservation module 518 to produce a total loss of 904. The total loss of 904 can be used to train the GNN 508 (e.g., backpropagation) to generate graph embeddings.

[0096] Figure 10 This is a block diagram illustrating an example architecture 1000 for training a GNN (e.g., 508) using sinusoidal position information according to various aspects of this disclosure. Example architecture 1000 is similar to... Figure 5 The example architecture shown is 500. However, Figure 10 It also includes a node location module 1002. The node location module 1002 can receive the ANN model 502. The node location module 1002 can determine the location information of each node in the ANN model 502. The node location information may include (but is not limited to) shortest distance localization (SDP) or longest distance localization (LDP) of the node. The node location module 1002 can calculate a sinusoidal position embedding based on the node location information.

[0097] Sinusoidal positional embeddings can be combined with operator embeddings. Both operator embeddings and positional embeddings can be provided to GNN 508. Furthermore, GNN 508 can generate graph embeddings based on position-induced operator embeddings.

[0098] Figure 11A and Figure 11B This is a block diagram illustrating examples of location assignment using Shortest Distance Positioning (SDP) and Longest Distance Positioning (LDP) according to various aspects of this disclosure. Reference Figure 11A Two examples of SDP (1100 and 1120) are shown. In SDP, a position can be assigned to each node in the neural network graph relative to the root node. In example 1100, a position or index can be assigned. ReLU 1102 can be determined as the root node (idx 0). Each subsequent node can be assigned a position in ascending order. For example, position idx 1 can be assigned to convolution block 1106, position idx 3 can be assigned to batch normalization block, and so on. Addition block 1104 has two possible paths relative to the root node (idx 0). In the first path, nodes assigned idx 1 to idx 5 can be traversed. In the second path, root node 1102 is directly coupled to addition block 1104. Since the distance between the root node and the addition block in the second path is shorter than in the first path, addition block 1104 can be assigned the position idx 1 (the next node after the root node).

[0099] Similarly, in Example 1120, the position given by the next subsequent index of the shortest path from the root node (e.g., ReLU 1122) can be assigned to adder block 1124. Therefore, in Example 1120, position idx2 can be assigned to adder block 1124.

[0100] refer to Figure 11B Examples 1140 and 1160 illustrate location assignment using LDP. Figure 11B In Example 1140, the location of adder block 1104 is assigned based on the path with the longest distance from the root node (e.g., ReLU 1102). In Example 1140, location idx 6 can be assigned to adder block 1104. Similarly, in Example 1160, location idx 5 can be assigned to adder block 1124.

[0101] In some respects, node location information can be used to compute sinusoidal location embeddings. That is, a sinusoidal function (e.g., sine or cosine) can be computed based on one or more of the shortest or longest distance localization determined for each node in the ANN model. This is achieved by combining the location information with operator embeddings and training a GNN 508 (e.g., ...) as described. The GNN 508 can generate position-aware graph embeddings 524. In doing so, it is possible to distinguish ANN models that have only minor differences (e.g., models with different numbers of parameters). Therefore, inference accuracy can be improved.

[0102] Figure 12 This is a flowchart illustrating a processor implementation of method 1200 for preprocessing for deep neural network compilation using graph neural networks, according to various aspects of this disclosure. For example, the processor implementation of method 1200 may be executed by one or more processors, such as CPUs (e.g., 102, 422), GPUs (e.g., 104, 426), DSPs (e.g., 106, 424), and / or NPUs (e.g., 108, 428)).

[0103] like Figure 12 As shown, at box 1202, the processor receives a representation of an artificial neural network (ANN) model. This ANN comprises multiple nodes coupled at the edges. See, for example, a reference... Figure 10 As described, node location module 1002 can receive ANN model 502. ANN model 502 may include residual neural networks, transformer neural networks, recurrent convolutional neural networks (RCNN), autoencoders, or other types of artificial neural networks. ANN model 502 may be, for example, a computation graph, a matrix, or other forms. Example architecture 500 can be configured to receive ANN models 502 of varying sizes (e.g., the number of operators).

[0104] At box 1204, the processor determines the location information of each node in the ANN model. For example, as referenced... Figure 10As described, the node location module 1002 can determine the location information of each node in the ANN model 502. The node location information may include (but is not limited to) shortest distance localization (SDP) or longest distance localization (LDP) of the node. By generating node location information based on SDP and / or LDP, the node location module 1002 can generate permutation-invariant node location information. The node location module 1002 can compute sinusoidal position embeddings based on the node location information.

[0105] At box 1206, the processor generates operator embeddings based on the location information of each node to represent the operators of the ANN model in the embedding space. See, for example, the reference... Figure 10 As described, sinusoidal position embedding can be combined with operator embedding.

[0106] At box 1208, the processor uses a graph neural network (GNN) to process operator embeddings based on learned distance metrics and location information to generate graph embeddings corresponding to the ANN model. For example, as referenced... Figure 10 As described, operator embeddings and position embeddings can be provided to the GNN 508. The GNN 508 can generate graph embeddings based on position-induced operator embeddings.

[0107] Furthermore, at box 1210, the processor uses GNN to determine the set of hyperparameters of the ANN model based on graph embedding.

[0108] Specific implementation examples are provided in the following numbered clauses.

[0109] 1. A preprocessing apparatus for compiling deep neural networks, the apparatus comprising:

[0110] At least one memory; and

[0111] At least one processor, coupled to the at least one memory, is configured to:

[0112] Receive a representation of an artificial neural network (ANN) model, wherein the ANN comprises multiple nodes coupled at the edges;

[0113] Determine the location information of each node in the ANN model;

[0114] Based on the location information of each node, an operator embedding is generated to represent the operator of the ANN model in the embedding space;

[0115] The operator embedding is processed by a graph neural network (GNN) based on a learned distance metric and the location information to generate a graph embedding corresponding to the ANN model; and

[0116] The set of hyperparameters of the ANN model is determined by the GNN based on the graph embedding.

[0117] 2. The apparatus according to Clause 1, wherein the position information includes a sinusoidal position embedding.

[0118] 3. The apparatus according to Clause 1 or 2, wherein the sinusoidal position embedding is calculated based on the relative node distance to the root node.

[0119] 4. The apparatus according to any one of clauses 1 to 3, wherein the relative node distance includes the shortest distance position or the longest distance position.

[0120] 5. The apparatus according to any one of clauses 1 to 4, wherein nodes having the same node type have different location information.

[0121] 6. The apparatus according to any one of clauses 1 to 5, wherein the at least one processor is further configured to generate the location information such that the location information is invariant to node permutations.

[0122] 7. A processor-implemented method for preprocessing for compiling a deep neural network, executed by at least one processor, the processor-implemented method comprising:

[0123] Receive a representation of an artificial neural network (ANN) model, wherein the ANN comprises multiple nodes coupled at the edges;

[0124] Determine the location information of each node in the ANN model;

[0125] Based on the location information of each node, an operator embedding is generated to represent the operator of the ANN model in the embedding space;

[0126] The operator embedding is processed by a graph neural network (GNN) based on a learned distance metric and the location information to generate a graph embedding corresponding to the ANN model; and

[0127] The set of hyperparameters of the ANN model is determined by the GNN based on the graph embedding.

[0128] 8. The processor-implemented method according to Clause 7, wherein the position information includes a sinusoidal position embedding.

[0129] 9. The processor implementation method according to Clause 7 or 8, wherein the sinusoidal position embedding is calculated based on the relative node distance to the root node.

[0130] 10. The method implemented by the processor according to any one of Clauses 7 to 9, wherein the relative node distance includes the shortest distance position or the longest distance position.

[0131] 11. The method implemented by the processor according to any one of Clauses 7 to 10, wherein nodes having the same node type have different location information.

[0132] 12. The processor-implemented method according to any one of Clauses 7 to 11, the processor-implemented method further comprising generating the location information such that the location information is invariant to node permutations.

[0133] 13. A non-transitory computer-readable medium having program code recorded thereon, the program code being executed by a processor and comprising:

[0134] Program code for receiving a representation of an artificial neural network (ANN) model, the ANN comprising multiple nodes coupled at the edges;

[0135] Program code used to determine the location information of each node in the ANN model;

[0136] Program code for generating operator embeddings based on the location information of each node to represent the ANN model in the embedding space;

[0137] Program code for processing the operator embeddings by a graph neural network (GNN) based on a learned distance metric and the location information to generate graph embeddings corresponding to the ANN model; and

[0138] Program code for determining the set of hyperparameters of the ANN model by the GNN based on the graph embedding.

[0139] 14. The non-transitory computer-readable medium as described in Clause 13, wherein the position information includes a sinusoidal position embedding.

[0140] 15. The non-transitory computer-readable medium according to Clause 13 or 14, wherein the sinusoidal position embedding is calculated based on the relative node distance to the root node.

[0141] 16. The non-transitory computer-readable medium according to any one of Clauses 13 to 15, wherein the relative node distance includes the shortest distance position or the longest distance position.

[0142] 17. A non-transitory computer-readable medium according to any one of Clauses 13 to 16, wherein nodes having the same node type have different location information.

[0143] 18. The non-transitory computer-readable medium according to any one of clauses 13 to 17, wherein the program code further comprises program code for generating the location information such that the location information is invariant to node permutations.

[0144] 19. A preprocessing apparatus for compiling deep neural networks, the apparatus comprising:

[0145] A component for receiving a representation of an artificial neural network (ANN) model, the ANN comprising multiple nodes coupled at the edges;

[0146] A component used to determine the location information of each node in the ANN model;

[0147] Components for generating operator embeddings based on the location information of each node to represent the operators of the ANN model in the embedding space;

[0148] A component for processing the operator embedding by a graph neural network (GNN) based on a learned distance metric and the location information to generate a graph embedding corresponding to the ANN model; and

[0149] A component used by the GNN to determine the set of hyperparameters of the ANN model based on the graph embedding.

[0150] 20. The apparatus according to Clause 19, wherein the position information includes a sinusoidal position embedding.

[0151] 21. The apparatus according to Clause 19 or 20, wherein the sinusoidal position embedding is calculated based on the relative node distance to the root node.

[0152] 22. The apparatus according to any one of clauses 19 to 21, wherein the relative node distance includes the shortest distance position or the longest distance position.

[0153] 23. The apparatus according to any one of clauses 19 to 22, wherein nodes having the same node type have different location information.

[0154] 24. The apparatus according to any one of clauses 19 to 23, the apparatus further comprising a component for generating the location information such that the location information is invariant to node permutations.

[0155] The various operations of the methods described above can be performed by any suitable component capable of performing the corresponding function. These components may include various hardware and / or software components and / or modules, including but not limited to circuits, application-specific integrated circuits (ASICs), or processors. Generally, in the cases where operations are illustrated in the accompanying drawings, these operations may have corresponding paired components with similar numbering plus functional components.

[0156] As used, the term "determine" encompasses a wide variety of actions. For example, "determine" can include calculation, computation, processing, derivation, research, searching (e.g., looking in a table, database, or other data structure), assertion, etc. Additionally, "determine" can include receiving (e.g., receiving information), accessing (e.g., accessing data in memory), etc. Furthermore, "determine" can include parsing, selecting, choosing, building, etc.

[0157] As used, the phrase "at least one of" in a list of items refers to any combination of these items, including a single member. As an example, "at least one of: a, b, or c" is intended to cover: a, b, c, ab, ac, bc, and abc.

[0158] The various exemplary logic blocks, modules, and circuits described in this disclosure can be implemented or executed using a general-purpose processor, digital signal processor (DSP), application-specific integrated circuit (ASIC), field-programmable gate array (FPGA) or other programmable logic device (PLD), discrete gate or transistor logic component, discrete hardware component, or any combination thereof designed to perform the described functions. While the general-purpose processor may be a microprocessor, in alternative embodiments, the processor may be any commercially available processor, controller, microcontroller, or state machine. The processor may also be implemented as a combination of computing devices, such as a combination of a DSP and a microprocessor, multiple microprocessors, one or more microprocessors combined with a DSP core, or any other such configuration.

[0159] The steps or algorithms of the methods described in this disclosure may be directly embodied in hardware, a software module executed by a processor, or a combination of both. The software module may reside in any form of storage medium known in the art. Some examples of usable storage media include random access memory (RAM), read-only memory (ROM), flash memory, erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), registers, hard disks, removable disks, CD-ROMs, and the like. The software module may include a single instruction or multiple instructions and may be distributed across several different code segments, across different programs, and across multiple storage media. The storage medium may be coupled to the processor, enabling the processor to read information from and write information to the storage medium. Alternatively, the storage medium may be integral with the processor.

[0160] The disclosed method includes one or more steps or actions for implementing the described method. The steps and / or actions of the method may be interchanged without departing from the scope of the claims. In other words, unless a specific order of steps or actions is specified, the order and / or use of a particular step and / or action may be modified without departing from the scope of the claims.

[0161] The described functionality can be implemented in hardware, software, firmware, or any combination thereof. If implemented in hardware, an example hardware configuration may include a processing system within the device. This processing system may utilize a bus architecture. Depending on the specific application and overall design constraints of the processing system, the bus may include any number of interconnect buses and bridges. The bus can link various circuits together, including processors, machine-readable media, and bus interfaces. The bus interface can be used to connect network adapters, etc., to the processing system via the bus. The network adapter can be used to implement signal processing functions. In some respects, user interfaces (e.g., keypads, displays, mice, joysticks, etc.) may also be connected to the bus. The bus may also link various other circuits, such as timing sources, peripherals, voltage regulators, power management circuits, etc., which are well known in the art and will not be described further.

[0162] A processor may be responsible for managing the bus and general-purpose processing, including executing software stored on a machine-readable medium. A processor may be implemented using one or more general-purpose processors and / or special-purpose processors. Examples include microprocessors, microcontrollers, DSP processors, and other circuitry that can execute software. Software should be interpreted broadly as instructions, data, or any combination thereof, whether referred to as software, firmware, middleware, microcode, hardware description language, or others. By way of example, a machine-readable medium may include random access memory (RAM), flash memory, read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), registers, disks, optical disks, hard disks, or any other suitable storage medium, or any combination thereof. A machine-readable medium may be embodied in a computer program product. A computer program product may include packaging material.

[0163] In a hardware implementation, machine-readable media can be part of a processing system separate from the processor. However, as those skilled in the art will readily understand, machine-readable media, or any portion thereof, can be external to the processing system. By way of example, machine-readable media may include transmit lines, carrier waves modulated by data, and / or computer components separate from the device, all accessible to the processor via a bus interface. Alternatively or additionally, machine-readable media, or any portion thereof, may be integrated into the processor, such as in the case of a cache and / or a general-purpose register file. Although the various components discussed may be described as having a specific location, such as local components, they may also be configured in various ways, such as certain components being configured as part of a distributed computing system.

[0164] The processing system may be configured as a general-purpose processing system having one or more microprocessors providing processor functionality and external memory providing at least a portion of machine-readable medium, all of which are linked together with other supporting circuitry via an external bus architecture. Alternatively, the processing system may include one or more neuromorphic processors for implementing the described neuron and nervous system models. As another alternative, the processing system may be implemented using an application-specific integrated circuit (ASIC) having a processor, bus interface, user interface, supporting circuitry, and at least a portion of machine-readable medium integrated on a single chip, or using one or more field-programmable gate arrays (FPGAs), programmable logic devices (PLDs), controllers, state machines, gated logic components, discrete hardware components, or any other suitable circuitry, or any combination of circuitry capable of performing the various functionalities described throughout this disclosure. Those skilled in the art will recognize how best to implement the described functionality of the processing system depends on the specific application and the overall design constraints imposed on the system as a whole.

[0165] Machine-readable media may include multiple software modules. These software modules include instructions that, when executed by a processor, enable the processing system to perform various functions. Software modules may include send and receive modules. Each software module may reside in a single storage device or be distributed across multiple storage devices. For example, when a triggering event occurs, a software module may be loaded from a hard disk drive into RAM. During the execution of a software module, the processor may load some of the instructions into a cache to improve access speed. One or more cache lines may then be loaded into a general-purpose register file for processor execution. When the functionality of a software module is referred to below, it will be understood that such functionality is implemented by the processor when executing the instructions from that software module. Furthermore, it should be understood that aspects of this disclosure result in improvements to the functionality of a processor, computer, machine, or other system implementing such aspects.

[0166] If implemented in software, the functions may be stored as one or more instructions or codes on or transmitted through a computer-readable medium. Computer-readable media includes both computer storage media and communication media, including any medium that facilitates the transfer of a computer program from one location to another. Storage media can be any available medium accessible to a computer. By way of example, and not limitation, such computer-readable media may include RAM, ROM, EEPROM, CD-ROM or other optical disc storage devices, magnetic disk storage devices or other magnetic storage devices, or any other medium that can be used to carry or store the desired program code in the form of instructions or data structures and is accessible to a computer. Additionally, any connection is also appropriately referred to as a computer-readable medium. For example, if the software is transmitted from a website, server, or other remote source using coaxial cable, optical fiber, twisted pair, digital subscriber line (DSL), or wireless technologies such as infrared (IR), radio, and microwave, these coaxial cable, optical fiber, twisted pair, DSL, or wireless technologies such as infrared, radio, and microwave are included in the definition of medium. As used, disks and optical discs include compact optical discs (CDs), laser discs, optical discs, digital versatile discs (DVDs), floppy disks, and Blu-ray discs. ® Optical discs, where magnetic disks typically reproduce data magnetically, and optical discs reproduce data optically using lasers. Therefore, in some aspects, computer-readable media may include non-transitory computer-readable media (e.g., tangible media). Furthermore, in other aspects, computer-readable media may include transient computer-readable media (e.g., signals). Combinations of the above should also be included within the scope of computer-readable media.

[0167] Therefore, some aspects may include a computer program product for performing the presented operations. For example, such a computer program product may include a computer-readable medium on which instructions are stored (and / or encoded) that can be executed by one or more processors to perform the described operations. In some aspects, the computer program product may include packaging material.

[0168] Furthermore, it should be understood that modules and / or other suitable components for performing the described methods and techniques may be downloaded and / or otherwise obtained by the user terminal and / or base station where applicable. For example, such devices can be coupled to a server to facilitate the delivery of components for performing the described methods. Alternatively, the various methods described can be provided via storage components (e.g., RAM, ROM, physical storage media such as CDs or floppy disks) so that the user terminal and / or base station can obtain the various methods once the storage component is coupled to or provided to the device. In addition, any other suitable techniques suitable for providing the described methods and techniques to the device may be utilized.

[0169] It should be understood that the claims are not limited to the precise configurations and components illustrated above. Various modifications, variations, and alterations may be made to the arrangement, operation, and details of the methods and apparatus described above without departing from the scope of the claims.

Claims

1. A preprocessing apparatus for compiling deep neural networks, the apparatus comprising: At least one memory; and At least one processor, coupled to the at least one memory, is configured to: Receive a representation of an artificial neural network (ANN) model, wherein the ANN comprises multiple nodes coupled at the edges; Determine the location information of each node in the ANN model; Based on the location information of each node, an operator embedding is generated to represent the operator of the ANN model in the embedding space; The operator embedding is processed by a graph neural network (GNN) based on the learned distance metric and the location information to generate a graph embedding corresponding to the ANN model; as well as The set of hyperparameters of the ANN model is determined by the GNN based on the graph embedding.

2. The apparatus of claim 1, wherein the position information includes a sinusoidal position embedding.

3. The apparatus of claim 2, wherein the sinusoidal position embedding is calculated based on the relative node distance to the root node.

4. The apparatus according to claim 3, wherein the relative node distance includes the shortest distance position or the longest distance position.

5. The apparatus of claim 1, wherein nodes having the same node type have different location information.

6. The apparatus of claim 1, wherein the at least one processor is further configured to generate the location information such that the location information is node permutation invariant.

7. A processor-implemented method for preprocessing for compiling a deep neural network, executed by at least one processor, the processor-implemented method comprising: Receive a representation of an artificial neural network (ANN) model, wherein the ANN comprises multiple nodes coupled at the edges; Determine the location information of each node in the ANN model; Based on the location information of each node, an operator embedding is generated to represent the operator of the ANN model in the embedding space; The operator embedding is processed by a graph neural network (GNN) based on the learned distance metric and the location information to generate a graph embedding corresponding to the ANN model; as well as The set of hyperparameters of the ANN model is determined by the GNN based on the graph embedding.

8. The processor-implemented method of claim 7, wherein the position information includes sinusoidal position embedding.

9. The processor-implemented method of claim 8, wherein the sinusoidal position embedding is calculated based on the relative node distance to the root node.

10. The processor-implemented method of claim 9, wherein the relative node distance includes the shortest distance position or the longest distance position.

11. The processor-implemented method of claim 7, wherein nodes having the same node type have different location information.

12. The processor-implemented method of claim 7, further comprising generating the location information such that the location information is node permutation invariant.

13. A non-transitory computer-readable medium having program code recorded thereon, the program code being executed by a processor and comprising: Program code for receiving a representation of an artificial neural network (ANN) model, the ANN comprising multiple nodes coupled at the edges; Program code used to determine the location information of each node in the ANN model; Program code for generating operator embeddings based on the location information of each node to represent the ANN model in the embedding space; Program code for processing the operator embeddings by a graph neural network (GNN) based on a learned distance metric and the location information to generate graph embeddings corresponding to the ANN model; and Program code for determining the set of hyperparameters of the ANN model by the GNN based on the graph embedding.

14. The non-transitory computer-readable medium of claim 13, wherein the position information includes a sinusoidal position embedding.

15. The non-transitory computer-readable medium of claim 14, wherein the sinusoidal position embedding is calculated based on the relative node distance to the root node.

16. The non-transitory computer-readable medium of claim 15, wherein the relative node distance includes the shortest distance position or the longest distance position.

17. The non-transitory computer-readable medium of claim 13, wherein nodes having the same node type have different location information.

18. The non-transitory computer-readable medium of claim 13, wherein the program code further comprises program code for generating the location information such that the location information is invariant to node permutations.

19. A preprocessing apparatus for compiling deep neural networks, the apparatus comprising: A component for receiving a representation of an artificial neural network (ANN) model, the ANN comprising multiple nodes coupled at the edges; A component used to determine the location information of each node in the ANN model; Components for generating operator embeddings based on the location information of each node to represent the operators of the ANN model in the embedding space; A component for processing the operator embedding by a graph neural network (GNN) based on a learned distance metric and the location information to generate a graph embedding corresponding to the ANN model; and A component used by the GNN to determine the set of hyperparameters of the ANN model based on the graph embedding.

20. The apparatus of claim 19, wherein the position information includes a sinusoidal position embedding.

21. The apparatus of claim 20, wherein the sinusoidal position embedding is calculated based on the relative node distance to the root node.

22. The apparatus of claim 21, wherein the relative node distance includes the shortest distance position or the longest distance position.

23. The apparatus of claim 19, wherein nodes having the same node type have different location information.

24. The apparatus of claim 19, further comprising components for generating the location information such that the location information is node permutation invariant.

Images