System, apparatus, method and non-transitory computer readable storage device for optimizing artificial neural network

Abstract

A computerized method has the steps of generating an input computational graph (CG) for representing a neural architecture, and performing a plurality of optimization steps in at least one iteration to obtain a modified neural architecture represented by the modified CG, thereby obtaining a neural network for use in one or more computing devices. The optimization step includes identifying one or more sub-graphs in the input CG, obtaining the modified CG by replacing the identified one or more sub-graphs with one or more replacement sub-graphs, respectively, evaluating one or more indicators of the modified neural network architecture represented by the modified CG, and, in accordance with the evaluation, modifying the one or more indicators of the modified neural network architecture. And selecting the modified CG or the input CG. When the optimization step is performed in multiple iterations, the selected CG obtained in one iteration is used as the input CG in the next iteration.

Description

Cross-references to related applications concerning systems, apparatus, methods for optimizing artificial neural networks and non-transitory computer-readable storage devices

[0001] This application claims the benefit of U.S. Patent Application No. 18 / 372,919, filed September 26, 2023, and U.S. Provisional Patent Application No. 63 / 539,549, filed September 20, 2023, the entire contents of which are incorporated herein by reference. Technical Field

[0002] This invention generally relates to artificial intelligence (AI) systems, apparatus, methods, and non-transitory computer-readable storage devices, and more particularly to AI systems, methods, and non-transitory computer-readable storage devices for training AI models using adaptive data sampling. Background Technology

[0003] Artificial intelligence (AI) technology is widely known. Generally speaking, through various AI models such as artificial neural networks (ANNs) widely used in machine learning (ML), AI technology has greatly improved the functionality of computing devices (e.g., computers and / or “intelligent” devices) and computer systems.

[0004] Neural architecture search (NAS) is a method for automating the design of ANN architectures. NAS improves the performance of neural network models by replacing manual trial and error with an automated search process, thus enhancing performance in many computer vision tasks. However, since the underlying search space of the architecture grows exponentially with the size of the architecture, searching for the optimal neural architecture is often very labor-intensive.

[0005] Therefore, an improved NAS method is needed to generate ANNs with better performance. Summary of the Invention

[0006] According to one aspect of the present invention, a first computerized method is provided, comprising: generating an input computational graph (CG) for representing a neural network architecture, wherein the input CG includes one or more nodes, each node corresponding to an operation; performing a plurality of optimization steps in at least one iteration to obtain a modified neural network architecture represented by a modified CG, thereby obtaining a neural network for one or more computing devices. The plurality of optimization steps include: identifying one or more subgraphs in the input CG; obtaining the modified CG by replacing the identified one or more subgraphs with one or more alternative subgraphs respectively; evaluating one or more metrics of the modified neural network architecture represented by the modified CG; and selecting either the modified CG or the input CG based on the evaluation. When the plurality of optimization steps are performed in multiple iterations, the selected CG obtained in one of the multiple iterations is used as the input CG in the next iteration of the multiple iterations.

[0007] In some embodiments, the first computerized method further includes: adjusting a first relationship between one or more inputs of each replacement subgraph and one or more outputs of the modified CG located in a first portion preceding the replaced segment of the modified CG, such that the one or more inputs match each other; and / or adjusting a second relationship between one or more outputs of each replacement subgraph and one or more inputs of the modified CG located in a second portion following the replaced segment of the modified CG, such that the one or more outputs match each other.

[0008] In some embodiments, the first computerized method further includes: using mixed-integer linear programming (MILP) to adjust a first relationship between one or more inputs of each replacement subgraph and one or more outputs of the modified CG located in a first portion preceding the replaced segment of the modified CG, such that the one or more inputs match the one or more outputs; and / or adjusting a second relationship between one or more outputs of each replacement subgraph and one or more inputs of the modified CG located in a second portion following the replaced segment of the modified CG, such that the one or more outputs match the one or more inputs.

[0009] In some embodiments, evaluating the one or more metrics of the modified neural network architecture represented by the modified CG includes: evaluating the one or more metrics of the modified neural network architecture represented by the modified CG under one or more constraints.

[0010] In some embodiments, selecting the modified CG or the input CG includes using a Pareto front to select the modified CG or the input CG.

[0011] In some embodiments, identifying the one or more subgraphs in the input CG includes: forming a triplet for each of the one or more nodes in the input CG, wherein the triplet includes the operation of the node, the operation of the node's in node, and the operation of the node's out node; encoding each triplet as a symbol according to the node traversal order to obtain a symbol sequence; and identifying one or more substrings by searching the one or more subgraphs in the symbol sequence, wherein each of the one or more substrings represents a corresponding subgraph in the one or more subgraphs.

[0012] In some embodiments, the input CG is a directed acyclic graph (DAG) having a plurality of nodes and one or more directed edges between a pair of nodes among the plurality of nodes; the first computerized method further includes: topologically indexing the input CG by assigning a number to each of the plurality of nodes, wherein for each directed edge from node i to node j among the plurality of nodes, the number of node i is less than the number of node j, and the numbers assigned to the plurality of nodes form the node traversal order.

[0013] In some embodiments, each subgraph in the identified one or more subgraphs and its corresponding subgraph in the one or more replacement subgraphs have the same number of inputs and the same number of outputs.

[0014] In some embodiments, evaluating the one or more metrics of the modified neural network architecture represented by the modified CG includes: dividing the modified CG into segment partitions including the one or more replacement subgraphs, a preceding partition before the segment partition, and a succeeding partition after the segment partition; using a graph neural network (GNN) with each of the preceding partition, the segment partition, and the succeeding partition as input to compute a fixed-length vector for each of the preceding partition, the segment partition, and the succeeding partition; concatenating the vectors of the preceding partition, the segment partition, and the succeeding partition to form a concatenated vector; and generating a performance estimate of the modified CG based on the concatenated vector.

[0015] In some embodiments, the performance estimate of generating the modified CG based on the concatenated vectors includes: generating the performance estimate of the modified CG using a multi-layer perception (MLP) artificial neural network (ANN) with the concatenated vectors as input.

[0016] According to one aspect of the present invention, one or more circuits are provided for performing the first method described above.

[0017] According to one aspect of the invention, one or more non-transitory computer-readable storage devices (also referred to as "non-transitory computer-readable storage media") are provided, comprising computer-executable instructions, wherein, when executed, the instructions cause one or more circuits to perform the first method described above.

[0018] According to one aspect of the present invention, a second computerized method is provided, comprising: generating an input CG for representing a neural network architecture, wherein the input CG includes one or more nodes, each node corresponding to an operation; forming a triplet for each of the one or more nodes of the input CG, wherein the triplet includes the operation of the node, the operation of the node's input node, and the operation of the node's output node; encoding each triplet as a symbol according to a node traversal order to obtain a symbol sequence; collecting one or more unique substrings from the symbol sequence; converting the collected substrings into a plurality of subgraphs; and storing the plurality of subgraphs to optimize one or more target neural network architectures.

[0019] In some embodiments, the input CG is a directed acyclic graph (DAG) having a plurality of nodes and one or more directed edges between a pair of nodes among the plurality of nodes; the second computerized method further includes: topologically indexing the input CG by assigning a number to each of the plurality of nodes, wherein for each directed edge from node i to node j among the plurality of nodes, the number of node i is less than the number of node j, and the numbers assigned to the plurality of nodes form the node traversal order.

[0020] According to one aspect of the present invention, one or more circuits are provided for performing the second method described above.

[0021] According to one aspect of the invention, one or more non-transitory computer-readable storage devices are provided, including computer-executable instructions, wherein, when executed, the instructions cause one or more circuits to perform the second method described above.

[0022] The above-described methods, one or more circuits, and one or more non-transitory computer-readable storage devices have various advantages, such as: The method disclosed in this paper achieves search space generalization, thereby enabling the combination of information from different search spaces and the computation of neural networks outside these search spaces and in different tasks.

[0023] The method disclosed in this paper extracts segments that have different characteristics in terms of size, topology, and / or granularity. The method disclosed in this paper also improves efficiency.

[0024] The predictor used in the method disclosed in this paper is aware of the location and context of subgraph replacements. The predictor used in the method disclosed in this paper is trained to be sensitive to changes in the neural network (e.g., computational and resolution variations), thereby enabling it to rank neural architectures with high accuracy based on their performance.

[0025] In computer vision applications, this paper discloses a method that utilizes resolution propagation mixed-integer linear programming (MILP). This method is applicable to handling subgraphs with different resolutions corresponding to different parts of a neural network and with strictly defined search spaces. This enables the combination of subgraphs in different search spaces and the utilization of knowledge across different families of neural architectures. In other words, subgraphs mined from family X can help improve the performance of architectures from family Y.

[0026] The method disclosed in this paper achieves robust optimization and can optimize both performance and resource metrics, with flexibility in the selection of performance and resource metrics. Attached Figure Description

[0027] To gain a more complete understanding of the present invention, reference is made to the following description and accompanying drawings, in which: Figure 1 is a simplified schematic diagram of an artificial intelligence (AI) system provided in some embodiments of the present invention; Figure 2 is a schematic diagram of the hardware structure of the infrastructure layer of the AI ​​system shown in Figure 1 provided in some embodiments of the present invention; Figure 3 is a schematic diagram of the hardware structure of the chip of the AI ​​system shown in Figure 1 provided in some embodiments of the present invention; Figure 4 is a schematic diagram of an AI model in the form of a deep neural network (DNN) used in the infrastructure layer shown in Figure 2; Figure 5 shows a schematic diagram of the workflow of a traditional neural architecture search (NAS) method for designing new neural network architectures; Figures 6A to 6C show examples of computation graphs, including operation types and resolutions, applicable to two-path convolutional neural networks (CNNs); Figure 7 shows a flowchart of the first stage of the automatic graph optimization (AutoGO) method provided in some embodiments of the present invention, wherein the first stage of the AutoGO method combines the frequent subgraph mining (FSM) method and the byte-pair encoding (BPE) method from the input computation graph. Figure 8A shows a simplified example of the input bipath CG processed by the first stage of the AutoGO method shown in Figure 7, wherein the input CG includes 12 nodes, each node corresponding to an operation; Figure 8B shows a schematic diagram of the topological sorting of the input CG shown in Figure 8A; Figure 9 shows a detailed flowchart of the node encoding steps of the subgraph extraction block shown in Figure 7 provided by some embodiments of the present invention; Figure 10A shows a schematic diagram of the workflow of the second stage of the AutoGO method provided by some embodiments of the present invention; Figure 10B shows a schematic diagram of the workflow of the second stage of the AutoGO method provided by other embodiments of the present invention; Figure 11A shows a partitioning schematic diagram of the input CG provided by other embodiments of the present invention; Figure 11B shows a schematic diagram of the modified CG obtained from the input CG shown in Figure 11A by replacing the recognition segment with the replacement segment provided by other embodiments of the present invention; Figure 12 shows a schematic diagram of a portion of the modified CG used to illustrate resolution propagation. Detailed Implementation

[0028] A. Artificial Intelligence Systems Artificial intelligence (AI) machines and systems typically include one or more AI models, which can be trained using a large amount of relevant data to improve the accuracy of their perception, reasoning, and decision-making.

[0029] Referring to Figure 1 below, an AI system for data clone detection provided by some embodiments of the present invention is illustrated. This AI system is generally identified by reference numeral 100. The AI ​​system 100 includes: an infrastructure layer 102 for providing the hardware foundation of the AI ​​system 100; a data processing layer 104 for processing relevant data and providing various functions 106 as needed and / or implemented; and an application layer 108 for providing intelligent products and industry applications.

[0030] Infrastructure layer 102 includes: necessary input components 112 such as sensors and / or other input devices for collecting input data; computing components 114 such as one or more smart chips, circuits and / or integrated chips (ICs) for performing necessary calculations; and a suitable infrastructure platform 116 for AI tasks.

[0031] One or more computing components 114 may be one or more central processing units (CPUs), one or more neural processing units (NPUs) (i.e., processing units with dedicated circuitry for AI-related computations and logic), one or more graphics processing units (GPUs), one or more application-specific integrated circuits (ASICs) and / or one or more field-programmable gate arrays (FPGAs), and may include necessary circuitry for hardware acceleration.

[0032] Platform 116 can be a distributed computing framework with network support capabilities, and can include cloud storage and computing, internet networks, etc.

[0033] In Figure 1, the data collected by the input component 112 is conceptually represented by the data source block 122, which may include any suitable data, such as sensor data (e.g., data collected by Internet of Things (IoT) devices), service data and / or sensing data (e.g., force, offset, liquid level, temperature and / or humidity, etc.), and may be in any suitable form, such as graphs, images, audio clips, video clips and / or text, etc.

[0034] The data processing layer 104 includes one or more programs and / or program modules 124 in the form of software, firmware and / or hardware circuits. The one or more programs and / or program modules 124 are used to process the data from the data source block 122 to achieve various purposes, such as data training, machine learning, deep learning, search, reasoning and / or decision-making.

[0035] In machine learning and deep learning, data source block 122 can be used for symbolic and formal intelligent information modeling, extraction, preprocessing, training, etc.

[0036] Reasoning refers to the process of simulating human intelligent reasoning in computers or intelligent systems, using formalized information to perform machine thinking and solve problems according to reasoning control strategies. Typical functions include search and matching.

[0037] Decision-making refers to the process of making a decision after reasoning about intelligent information. Generally, it provides functions such as classification, sorting, reasoning (or prediction).

[0038] Through programs and / or program modules 124, the data processing layer 104 typically provides various functions such as transformation, text analysis, computer vision processing, speech recognition and / or image recognition 106.

[0039] Through function 106, AI system 100 can provide various intelligent products and industry applications 108 in various fields. These intelligent products and industry applications 108 can be a complete AI solution package for productizing intelligent information decision-making and realizing applications. Examples of application areas for intelligent products and industry applications include intelligent manufacturing, intelligent transportation, smart home, smart healthcare, intelligent security, autonomous driving, safe city, and smart terminals.

[0040] Figure 2 is a schematic diagram of the hardware structure of the infrastructure layer 102 provided in some embodiments of the present invention. As shown, the infrastructure layer 102 includes a data collection device 140, which collects training data 142 for training an AI model 148 (e.g., a machine learning (ML) model or a neural network (NN) model (e.g., a convolutional neural network (CNN) model), and stores the collected training data 142 in a training database 144. In this document, the training data 142 includes multiple labeled, annotated, or otherwise categorized data samples that can be used for training (hereinafter referred to as "training samples") and their corresponding expected results (hereinafter referred to as "labels"; i.e., the target or expected prediction that the AI ​​model 148 should make based on the data samples). In this document, training samples can be any suitable data samples used to train the AI ​​model 148, such as one or more annotated images, one or more annotated text samples, one or more annotated audio clips, one or more annotated video clips, and / or one or more annotated digital data samples, etc. The desired outcome is the ideal result obtained by processing training samples using a trained or optimized AI model 148'. One or more training devices 146 (e.g., one or more server computers constituting a "computer cloud" (or simply "cloud"), and / or one or more client computing devices similar to or the same as execution device 150 (also referred to as "edge devices")) train the AI ​​model 148 using training data 142 retrieved from training database 144 to train the AI ​​model 148 for use by computing module 174 (described in more detail later).

[0041] Those skilled in the art will understand that, in practical applications, the training data 142 maintained in the training database 144 may not all be collected by the data collection device 140, but may be received from other devices. Furthermore, the training device 146 may not necessarily train entirely based on the training data 142 maintained in the training database 144 to obtain a trained AI model 148', but may obtain the training data 142 from the cloud or other sources for model training.

[0042] The trained AI model 148' obtained through training by training device 146 can be applied to various systems or devices, such as execution device 150. Execution device 150 can be an edge device, such as a mobile terminal, tablet, laptop, augmented reality (AR) device, virtual reality (VR) device, in-vehicle terminal, or server. Execution device 150 includes an I / O interface 152 for receiving input data 154 (e.g., input data provided by user 158) from external device 156 and / or outputting results 160 to external device 156. External device 156 can also provide training data 142 to training database 144. Execution device 150 can also directly receive input data 154 from user 158 using its I / O interface 152.

[0043] The execution device 150 also includes a processing module 172, which is used to preprocess the input data 154 received from the I / O interface 152. For example, if the input data 154 includes one or more images, the processing module 172 can perform image preprocessing, such as image filtering, image enhancement, image smoothing, and / or image restoration.

[0044] The processed data 142 is then sent to the computation module 174, which uses the trained AI model 148' to analyze the data received from the processing module 172 for prediction. As described above, the prediction result 160 can be output to an external device 156 via the I / O interface 152. Furthermore, the data 154 received by the execution device 150 and the prediction result 160 generated by the execution device 150 can be stored in the data storage system 176.

[0045] In the following text, for ease of description, the AI ​​model to be trained and the corresponding trained AI model are identified by the same reference numeral 148.

[0046] Figure 3 is a schematic diagram of the hardware structure of the computing component 114 provided in some embodiments of the present invention. The computing component 114 can be any processor suitable for large-scale XOR operation processing, such as a convolutional NPU, a tensor processing unit (TPU), or a GPU. The computing component 114 can be part of the execution device 150 coupled to the host CPU 202, and function as a computing module 160 under the control of the host CPU 202. Alternatively, the computing component 114 can be in the training device 146 to complete its training and output the trained AI model 148.

[0047] As shown in Figure 3, computing component 114 is coupled to external memory 204 via bus interface unit (BIU) 212 to fetch instructions and data (e.g., input data 154 and weight data) from external memory 204. Instructions are transferred to instruction fetch buffer 214. Input data 154 is transferred to input memory 216 and unified memory 218 via memory access controller (or direct memory access controller (DMAC)) 220, and weight data is transferred to weight memory 222 via DMAC 220. In these embodiments, instruction fetch buffer 214, input memory 216, unified memory 218, and weight memory 222 are on-chip memories, and input data 154 and weight data can be organized in matrix form (referred to as "input matrix" and "weight matrix," respectively).

[0048] The controller 226 fetches instructions from the instruction fetch buffer 214 and accordingly controls the arithmetic circuit 228 to perform multiplication and addition using the input matrix from the input memory 216 and the weight matrix from the weight memory 222.

[0049] In some implementations, the arithmetic circuit 228 includes multiple processing engines (PEs) (not shown). In some implementations, the arithmetic circuit 228 is a two-dimensional pulsating array. Alternatively, the arithmetic circuit 228 may be a one-dimensional pulsating array or other electronic circuits capable of performing mathematical operations such as multiplication and addition. In some implementations, the arithmetic circuit 228 is a general-purpose matrix processor.

[0050] For example, the arithmetic circuit 228 can retrieve an input matrix A (e.g., a matrix representing an input image) from the input memory 216 and a weight matrix B (e.g., a convolution kernel) from the weight memory 222, cache the weight matrix B on each PE of the arithmetic circuit 228, and then perform matrix operations on the input matrix A and the weight matrix B. Partial or final calculation results retrieved by the arithmetic circuit 228 are stored in the accumulator 230.

[0051] If needed, the output of the arithmetic circuit 228 stored in the accumulator 230 can be further processed by the vector computation unit 232, for example, performing vector multiplication, vector addition, exponentiation, logarithmic operations, and / or size comparisons. The vector computation unit 232 may include multiple computation processing engines, primarily used for computation in the non-convolutional or fully connected layers (FC) of the CNN, specifically performing pooling and normalization calculations. For example, the vector computation unit 232 can apply nonlinear functions to the output of the arithmetic circuit 228, such as a vector of accumulated values, to generate valid values. In some implementations, the vector computation unit 232 generates normalized values, combined values, or both normalized and combined values.

[0052] In some implementations, the vector computation unit 232 stores the processed vector in the unified memory 218. In some implementations, the vector processed by the vector computation unit 232 can be stored in the input memory 216 and then used as valid input to the computation circuit 228, for example, for subsequent layers in a CNN.

[0053] Data output from the arithmetic circuit 228 and / or the vector calculation unit 232 can be transferred to the external memory 204.

[0054] Figure 4 is a schematic diagram of an AI model 148 in the form of a deep neural network (DNN), which is a complex form of artificial neural network (ANN). As shown in the figure, DNN 148 includes an input layer 302, multiple cascaded hidden layers 304, and an output layer 306. The trained AI model 148 can have a set of parameters optimized through AI model training.

[0055] Input layer 302 includes multiple input nodes 312, which receive input data and output the received data to computation nodes 314 of the subsequent hidden layer 304. Each hidden layer 304 includes multiple computation nodes 314. Each computation node 304 weights and combines the outputs of the input nodes or computation nodes of the previous layer (i.e., the input nodes 312 of input layer 302 or the computation nodes 314 of the previous hidden layer 304; each arrow indicates a weighted data transmission). Output layer 306 also includes one or more output nodes 316, each of which combines the outputs of the computation nodes 314 of the last hidden layer 304 to generate output 356.

[0056] Those skilled in the art will understand that AI models such as DNN 148 shown in Figure 4 typically require optimized training. For example, training device 146 (see Figure 2) can provide training data 142 (which includes multiple training samples with corresponding expected results) to input node 312, allowing it to be processed by AI model 148 and generate output from output node 316. By comparing the output obtained from output node 316 with the expected results in training data 142, a loss function can be established, and the parameters of AI model 148 (e.g., the weights of AI model 148) can be optimized by minimizing the loss function.

[0057] B. Neural Architecture Search (NAS) B-1. Traditional NAS Methods As mentioned above, neural architecture search (NAS) is a method for automating the design of ANN architectures. A typical NAS method includes a search phase and an evaluation phase. The search phase requires designing a fixed search space (including multiple candidate neural architectures, candidate layers, and / or candidate nodes) and a search algorithm that traverses the fixed search space to discover high-performance architectures. In the evaluation phase, the discovered architectures need to be evaluated to confirm that they can indeed achieve high performance. The output of a NAS framework is typically a set of new neural network architectures with high performance (denoted as "output neural network architecture"). In the following text, for ease of description, the term "neural network" can also be simply referred to as "network," and the term "neural network architecture" can also be simply referred to as "neural architecture."

[0058] Search space design is the first crucial step in NAS methods, as it determines the possible neural network architectures that NAS methods can use to generate output neural architectures. Without loss of generality, for example, a typical CNN can include multiple convolutional operators, and the maximum number of convolutional operators defined is an upper bound in the search space of convolutional operators. Therefore, if the maximum number of convolutional operators in any search network is set to five (5), it is impossible to find a CNN with six (6) convolutions. Other common searchable properties of CNNs include the number of channels per convolutional operator, the kernel size of each convolutional operator, and / or the connection patterns between different operators. Because the search space limits the possible architectures, it also limits the possible optimal performance. For example, a CNN found in a search space that allows a maximum of five (5) convolutions may achieve an accuracy of up to 80% on some datasets, compared to a CNN with six (6) convolutions.

[0059] Once the search space is determined, efficient search algorithms can be designed. There are many types of search algorithms, including random search, sample-based search, differentiable search and optimization (e.g., DARTS, SNAS), Bayesian optimization (e.g., NASBOT, BANANAS), genetic algorithms, and reinforcement learning (e.g., ENAS, NASNet). While search algorithms may differ in their speed of finding the optimal architecture, they all operate within a predefined search space. In other words, the search algorithm is applied to a manually constructed search space with predefined computational units. The granularity of these units ranges from simple primitive operations (e.g., convolution, rectified linear unit (ReLU), and / or pooling) to more complex blocks composed of multiple primitive operations (e.g., the MBConv blocks used to build MobileNetV2 and MobileNetV3, a popular family of CNNs). Further rules can be implemented regarding how to build macro-neural networks based on these operations or blocks by restricting their location and connectivity.

[0060] As mentioned above, the search process can begin with a search space and a search algorithm or method, where the search method typically yields a new set of architectures. In traditional search methods, high-performance architectures are selected by fully training and testing each new neural network on the desired dataset. However, such search methods are often costly. Most NAS methods rely on agents for faster performance evaluation, where the agent can be a weight-shared supernet trained before or during the search, a performance predictor learned from labeled data, a score calculated based on network properties, or a network that has been partially trained for several epochs and then tested. The design of performance agents is a popular subtopic under NAS and is crucial to the overall success of the search. Like the search algorithm, some performance agent methods (e.g., performance predictors) depend on the search space. Therefore, if the search space changes, the search algorithm needs to be modified and the performance agent redesigned.

[0061] Traditional NAS methods typically focus on a single task of interest, such as finding a novel CNN with high classification accuracy. In this case, classification accuracy alone is sufficient to determine which network is considered "high-performance." Recently, hardware-aware or hardware-friendly NAS methods have supported additional constraints, such as model size, floating-point operations (FLOPs), power consumption, and inference time, aiming to find high-performance architectures under specific constraints. Therefore, architectures searched under hardware-dependent constraints are considered optimal for the underlying hardware.

[0062] Figure 5 illustrates a schematic of workflow 340 for a traditional NAS approach used to design novel neural network architectures. According to workflow 340, one or more popular, manually constructed neural networks from the literature are evaluated, and a network type with high performance (or sufficiently high performance with low computational cost) is selected from among these networks (box 342). In the example shown in Figure 5, popular neural networks such as ResNet, MobileNet, and ViT are evaluated, and MobileNet is selected due to its lightweight nature.

[0063] Then, the search space 346 (box 344) is manually designed using generic MobileNet blocks, operators, network backbone, and topology to obtain a MobileNet-based search space. This manual design of the search space is typically based on experience, and the resulting search space is often non-generalizable.

[0064] In addition to the designed search space, a search algorithm and a performance agent (boxes 348 and 350, respectively) were designed to be used to discover new architectures 352 from the search space.

[0065] However, since the search space was initially inspired by MobileNet operators and only includes MobileNet operators, the newly discovered architectures will also be highly similar to MobileNet. In other words, searching within a MobileNet-inspired search space can only produce more variants that resemble MobileNet.

[0066] Therefore, traditional NAS methods have various drawbacks, such as: Low generalization. The design of the search space (boxes 342 to 346) exhibits low generalization. As mentioned above, if the search space is manually defined around MobileNet, the NAS method can only discover new architectures similar to MobileNet. If better network families (e.g., ViT) are subsequently introduced, the NAS framework for MobileNet cannot be used to find architectures similar to the newly introduced network families. Therefore, the NAS framework has low generalization across different network families. To find architectures similar to other neural networks (e.g., ViT), the search space must be redesigned, which is extremely inefficient.

[0067] Limited portability. Because the search space is manually designed, other modules in the NAS, such as the performance proxy model designed at box 350, typically depend on the search space. Therefore, the performance proxy model used for the first search space cannot usually be directly transferred to the second search space. In other words, when manually designing a new search space, it is usually necessary to recreate the new performance proxy model, making NAS an unnecessarily labor-intensive task.

[0068] There is insufficient support for improvements to user-provided architectures. Traditional NAS methods find new architectures from a predefined search space. However, in practice, users of NAS frameworks typically have a network and only want to improve that network. For example, a user may have already designed a well-performing CNN for a camera application and want to use NAS to further optimize that CNN for specific hardware, rather than making major changes to the CNN. Therefore, while it is preferable for the NAS framework to use the user's neural network as additional input to further tune and improve that neural network, rather than recreating the architecture from scratch, existing NAS methods cannot perform the desired neural network tuning and improvement.

[0069] It relies heavily on expert knowledge. Traditional NAS workflows rely heavily on expert knowledge. For example, to manually design a search space around a network family like ViT, one must first be a ViT expert. Without human expertise, designing a robust and efficient NAS framework would be extremely challenging.

[0070] The capabilities of existing frameworks are insufficient. If a NAS framework has already been designed for MobileNet and you wish to design another NAS framework for ViT using the traditional NAS workflow 340, you must repeat workflow 340 as shown in Figure 1. Directly sharing search knowledge between different frameworks is extremely challenging, primarily because the search space is manually designed. For example, if MobileNet NAS results indicate that a channel size of 128 is always better, this knowledge cannot be utilized when designing the ViT NAS framework without human intervention, hindering the development of NAS.

[0071] Hardware awareness is limited. If the search space is designed around MobileNet, blocks / operators from MobileNet may only be hardware-friendly. Therefore, manually designed search spaces implicitly limit hardware friendliness.

[0072] B-2. Automatic graph optimization (AutoGO) method for NAS Hereinafter, various embodiments are described to address at least some of the disadvantages described above.

[0073] In some embodiments, an Autograph Optimization (AutoGO) method can be used for Neural Architectures (NAS). The AutoGO method uses multiple baseline neural architectures (referred to as a "family of baseline NAS") to optimize the input neural architecture and leverages an evolutionary algorithm to improve the performance of the input neural architecture by iteratively mutating the computational units of the input neural architecture. The optimized neural architecture output from the AutoGO method is then used to construct an ANN for use by computing devices and / or computer systems.

[0074] The AutoGO method expands the scope of NAS from a fixed search space that searches predefined blocks to progressively optimize and mutate a given input neural architecture across multiple dimensions, including computations and hyperparameters, and leverages a database of useful subgraphs discovered by the algorithm.

[0075] In these embodiments, the AutoGO method represents each baseline neural architecture as a baseline computational graph (CG). By using the baseline CG as a low-level primitive representation of the neural network, the AutoGO method achieves generalization and can compute and / or understand the architecture of many neural networks based on different search spaces and application tasks (e.g., computer vision tasks). The AutoGO method optimizes the baseline CG by performing subgraph mutation.

[0076] In this paper, CG stands for Directed Acyclic Graph (DAG) representation of a neural network. Those skilled in the art will understand that a DAG is a directed graph without directed cycles. More specifically, a DAG consists of multiple vertices and one or more edges, each pointing from one vertex to another, such that traversing the edges along their directions never forms a closed loop. In some examples, the CG of a neural network can be obtained using TensorFlow (a software library for machine learning and artificial intelligence developed by Google LLC, Mountain View, California) or PyTorch (a machine learning framework provided by Meta AI in New York City, New York, and the Linux Foundation in San Francisco, California).

[0077] Each node in a CG corresponds to a "primitive" operation or atomic computational unit, such as the "two-dimensional convolution operation (conv2d)," "fused batch normalization operation (fusedbatchnorm)," and "batch normalization operation (BN or BatchNorm)" defined in TensorFlow. Each node also includes other necessary information. For example, when a neural network is used for computer vision, each node may include information about the resolution of the input and output tensors, such as the height, width, and channel size of the captured image. The edges of the CG describe the information flow between nodes. Figures 6A through 6C (connected by page link labels 1 through 3) illustrate examples of CGs used for computer vision applications, including operation types and resolutions suitable for two-path CNNs. Those skilled in the art will understand that in image processing and computer vision, the term "resolution" for an image typically refers to "height resolution," "width resolution," and "channel resolution." The terms "height resolution" and "width resolution" refer to the number of rows and columns of pixels in an image, respectively. The term "channel" refers to a set of features of an image (e.g., in some embodiments, a color image may have red, green, and blue channels; in some other embodiments where a DNN is used for image processing and / or for computer vision processing, an image may have 100 or more channels), and the term "channel resolution" refers to the number of features in a set of images.

[0078] In these embodiments, CG uses low-level primitive operators or operations as graph nodes. Those skilled in the art will understand that low-level primitive operators or operations are building blocks for constructing networks in any search space. Because primitive operators are generalizable across search spaces, CG can universally represent neural architectures from these search spaces and supports modifications that overcome constraints imposed by expert knowledge and heuristics.

[0079] In some embodiments, the AutoGO method comprises two phases. In the first phase, the AutoGO method uses a baseline CG (corresponding to a baseline neural architecture) to construct a segment database including one or more candidate subgraphs. Hereinafter, a subgraph is also referred to as a “computational segment” or simply a “segment.” In the second phase, the AutoGO method uses one or more candidate segments from the segment database to find one or more segments in the input CG (corresponding to the target neural architecture to be optimized) and mutates one or more segments. Those skilled in the art will understand that the segment database can be used (through multiple executions of the second phase of the AutoGO method) to process multiple input CGs. The segment database can also be updated periodically, as needed, and / or upon request.

[0080] B-3. ​​The first stage of the AutoGO method: In the first stage, the AutoGO method uses a baseline CG (corresponding to the baseline neural architecture) to construct a segment database that includes one or more candidate segments.

[0081] In some embodiments, the first phase of the AutoGO method combines the FSM method with the BPE method to extract one or more CG segments from the input CG402 (which is the baseline CG in the first iteration).

[0082] In existing technologies, the FSM method is a method for finding "frequent" or "common" segments (i.e., segments that appear more than a predefined threshold) in a given graph. More specifically, the FSM method finds or otherwise identifies segments in the graph and then calculates the support (i.e., the number of times each segment appears). If a segment's support is greater than a predefined threshold, the segment is identified as a common segment.

[0083] In these embodiments, the AutoGO method combines the BPE method with the FSM method to mine or otherwise obtain multiple segments (referred to as "candidate segments") from the input CG 402.

[0084] Figure 7 shows a flowchart of the first stage 500 of the AutoGO method provided in some embodiments of the present invention. For ease of illustration, Figure 7 shows the processing of only one baseline CG 402. Those skilled in the art will understand that the first stage of the AutoGO method shown in Figure 7 can be repeated to process multiple baseline CGs.

[0085] In step 502, the input CG (which is a DAG) is topologically indexed (also referred to as "topological sorting") to obtain the node traversal order. Of course, those skilled in the art will understand that in other embodiments, other suitable node traversal orders (e.g., predefined node traversal orders) can be used. In step 504, the topologically sorted input CG is encoded into a symbol sequence according to the node traversal order. In step 506, multiple subsequences of the symbol sequence are identified. In step 508, the identified subsequences are restored to a subgraph, and then the subgraph is stored in the segment database.

[0086] More specifically, in step 502, each node of the input CG is assigned a numeric label or index, such as an integer label or index, where for each directed edge from node i to node j, the integer label of node i is less than the integer label of node j.

[0087] Figure 8A shows a simplified example of an input bipath CG 542 comprising 12 nodes 552 to 574, each corresponding to an operation. Each node in the topologically sorted CG 552 to 574 is associated with a corresponding integer label. Figure 8B shows the topologically sorted input CG 542. For ease of illustration, each node in the CG 552 to 574 is shown in the form of "<operation>-<integer label>". For example, node 552 is shown as "Conv-0" in Figure 8B. However, those skilled in the art will understand that any suitable method (e.g., using an array) can be used to associate integers with nodes.

[0088] Those skilled in the art will also understand that topological sorting is nondeterministic, meaning that topological sorting can be performed on the same DAG multiple times and different labeling schemes can be obtained. For example, there are multiple ways to label nodes 556 to 560 (“Conv-2, BN-3, ReLU-4”) and nodes 562 to 566 (“Conv-5, BN-6, ReLU-7”) in Figure 8B. However, in this example, the label (“ReLU”) of node 554 can always be “1”, and the label (“Add”) of node 568 can always be “8”.

[0089] Those skilled in the art will also understand that the associated integer labels may begin with any integer (i.e., not necessarily starting with "0") and may not be consecutive.

[0090] Referring back to Figure 7, in step 504, each node in the input CG 542 is encoded according to the associated integer label and rich tokens (with additional features) to convert the input CG 542 into a sequence. Figure 9 shows a detailed flowchart of step 504.

[0091] In these embodiments, each node of input CG 542 is represented by its operation name. In step 582, the representation of each node is enriched or expanded by including contextual information of the node (e.g., the current node and its input and output operations) to form a triple of [current node operation, input node operation, output node operation].

[0092] For example, node 552 (Conv) has no incoming nodes but one outgoing node 554 (ReLU). Therefore, the rich or expanded representation of node 552 is [Conv, N / A, ReLU], where N / A represents "not available" and can be represented in any suitable way (e.g., "NULL"). As another example, node 568 (Add) has two incoming nodes 560 (ReLU) and 566 (ReLU) and one outgoing node 570 (Conv). Therefore, the rich or expanded representation of node 568 is the triple [Add, (ReLU, ReLU), Conv]. Those skilled in the art will understand that the operations in the triple can be arranged in any order in other embodiments (e.g., in some embodiments [incoming node, current node, outgoing node], and the expanded representation of node 568 is the triple [(ReLU,ReLU), Add, Conv]).

[0093] In step 584, the triples of each node are mapped to or otherwise encoded as unique symbols (e.g., character symbols) so that the entire input CG 542 can be represented as a sequence of symbols. In some embodiments, the triples are mapped to Chinese Unicode characters. Table 1 shows an example of encoding each node of the CG 542 shown in Figures 8A and 8B. For ease of understanding, in Table 1, each node (in the first three columns) is shown in the form of "<operation name> (<reference numerals>)".

[0094] Table 1: Node Encoding Examples

[0095] As can be seen from Table 1, node encoding follows the order of integer labels for nodes 552 to 574, but integer labels are not used when encoding nodes 552 to 574. Furthermore, the reference numerals shown in Table 1 are for ease of identification of nodes in Figures 8A and 8B, and these reference numerals are not used for any aspect of node encoding.

[0096] Therefore, node encoding is usually based on the operation triples of each node, which means that some nodes may be encoded with the same symbol. For example, since nodes 558, 564, and 572 have the same operation triples (i.e., [BN, Conv, ReLU]), nodes 558, 564, and 572 are encoded with the same symbol "liao". Therefore, the CG 542 shown in FIGS. 8A and 8B is encoded as the symbol sequence "de shi bu liao ren bu liao ren wo zai liao you".

[0097] As described above, in step 506 of the first stage 500 of the AutoGO method (see FIG. 7), multiple subsequences of the symbol sequence are identified. In this article, a subsequence (also denoted as "substring") is a unique part of the symbol sequence, indicating that any two subsequences are different. The same subsequence found at multiple positions in the symbol sequence will be regarded as one subsequence that appears multiple times, rather than multiple subsequences. Subsequences correspond to subgraphs, and subsequences and corresponding subgraphs can be converted to each other. In this article, a subgraph or segment is a unique part of the input CG 542, indicating that any two subgraphs are different. The same segment found at multiple positions in the input CG 542 will be regarded as one segment that appears multiple times, rather than multiple segments.

[0098] In step 506, the topological sorting of the input CG 542 obtained in step 504 and the encoded symbol sequence are used as the string corpus for segment extraction, where the n-symbol substring (also denoted as "subsequence") of the symbol sequence corresponds to the n-node segment of the input CG 542 (n > 0 and is an integer).

[0099] In this step, the BPE method is used to find multiple substrings from the string corpus (i.e., the encoded symbol sequence of the input CG 542). Those skilled in the art understand that the BPE method was initially used for data compression and has been used for text tokenization in natural language processing (NLP).

[0100] More specifically, the BPE method collects all single-symbol substrings (even those appearing only once) to cover all possible primitive (i.e., single-node) operations and adds the collected single-symbol substrings to the vocabulary. Then, the BPE method iteratively collects multi-symbol substrings from the string corpus based on the substring length (i.e., the number of symbols) and / or the substring's occurrence in the symbol sequence. For example, the BPE method might collect substrings in ascending order of length (i.e., starting with substrings of two symbols, then three symbols, and so on). When collecting substrings of the same length, the BPE method might collect substrings in descending order of frequency (i.e., starting with the most frequent or most commonly used substring, then the next most frequent, and so on). The collected multi-symbol substrings are added to the vocabulary. After collecting all substrings and adding them to the vocabulary, the vocabulary is stored in a segment database. In some embodiments, the vocabulary has a size limit (equivalently, the total number of collected substrings is limited). In these embodiments, step 506 terminates when the vocabulary (or the total number of collected substrings) reaches its limit.

[0101] Table 2 shows some examples of substrings and corresponding segments extracted from the symbol sequence of CG shown in Figures 8A and 8B. Eleven substrings are listed, including eight (8) single-symbol substrings (each single-symbol substring corresponds to a single-node segment), two (2) double-node substrings (each double-node substring corresponds to a double-node segment), and one (1) triple-node substring (corresponding to a triple-node segment).

[0102] Table 2: Examples of Encoded Node Strings

[0103] In these embodiments, in step 508 shown in Figure 7, the identified substrings in the vocabulary are restored to subgraphs and then stored in a segment database. The segment database can serve two purposes. First, in the second phase of the AutoGO method, the segments in the database can be units used to segment the input neural architecture or the parent neural architecture. Second, the segment database can serve as a repository for replacing subgraphs, used to replace segments of the input CG during mutation.

[0104] Those skilled in the art will understand that, although in the above embodiments all substrings (i.e., until the vocabulary size limit is reached) are identified for storage in the segment database, in some embodiments, any number of substrings may be identified (according to some specific rules) for storage in the segment database.

[0105] In some embodiments, in step 508, the identified substrings in the vocabulary are not restored to subgraphs. Instead, each identified substring, along with the input and output information of the corresponding subgraph, is stored in a segment database.

[0106] Those skilled in the art will understand that traditional FSM methods suffer from high computational costs, long runtimes, and low memory efficiency when extracting and storing subgraphs. By topologically sorting the input CG and enriching its labels, the first stage of the AutoGO method simplifies the subgraph isomorphic computation problem to mining segments from sequences, rather than mining from the graph itself. Regarding sequences, pattern extraction techniques offer high performance in terms of runtime and memory consumption. For example, using an optimized open-source framework, performing FSM on a large string corpus using BPE is computationally simpler.

[0107] B-4. The Second Stage of the AutoGO Method In the second stage, the AutoGO method optimizes the input neural architecture through mutation. Figure 10A illustrates a schematic diagram of the workflow 600 of the second stage of the AutoGO method provided in some embodiments of the present invention. For ease of illustration, Figure 7 shows only one iteration of the processing of a baseline CG 402. Those skilled in the art will understand that the AutoGO method shown in Figure 10A can be executed in one iteration or iteratively in multiple iterations. In each iteration, the input CG (either the target CG to be optimized in the first iteration or the CG output in the previous iteration) can be represented as the “parent CG” of the output CG (correspondingly, the input neural architecture is the parent neural architecture of the output neural architecture), and the output CG can be represented as the “child CG” of the input CG (correspondingly, the output neural architecture is the child neural architecture of the input neural architecture).

[0108] As shown in the figure, the input neural architecture is represented as input CG 542, which is topologically sorted, labeled and encoded as a sequence of symbols or a string (similar to steps 502 and 504 shown in Figure 7).

[0109] Then, the segment database 600 is used to identify one or more segments 606 in the input CG 542. For example, the second stage of the AutoGO method may select segments (or more specifically, substrings corresponding to segments) from the segment database 600 (e.g., by random selection or other suitable selection methods, such as selection according to predefined rules), and then search for the selected segments in the input CG 542. The second stage of the AutoGO method may repeat the selection and search steps to identify multiple segments 606 in the input CG 542.

[0110] At box 608, one or more identified segments are used as computational units for mutation to obtain a modified CG (and a modified neural architecture), wherein each segment in one or more identified segments 606 is replaced with a corresponding replacement segment having the same number of inputs and the same number of outputs.

[0111] At box 610, evaluate the performance of the modified CG (in other words, the modified neural architecture).

[0112] In existing technologies, neural predictors and similar methods can be used to evaluate the performance of neural architectures or neural networks. However, most existing performance evaluation methods rely on expert rules and assumptions, failing to capture the details in CG and therefore cannot be used for segment-level variation.

[0113] In these embodiments, at block 610, a predecessor-segment-successor (PSC) predictor 612 is used to estimate the performance variation between the modified CG and the input CG 542, which is caused by mutation of one or more recognition segments in CG 542. The PSC predictor 612 is sensitive to the position of the recognition segments in the entire neural network and their context, encoding important details such as tensor resolution. If the modified CG improves performance compared to the input CG 612, the modified CG is retained or otherwise selected; otherwise, the modified CG is discarded and the input CG 612 is retained or selected.

[0114] In some embodiments, the AutoGO method can optimize the neural architecture based on multiple objectives. For example, in some embodiments, objective metrics such as FLOP do not require detailed estimation mechanisms for measurement and can be used as constraints to limit the changes that the AutoGO method may make to the architecture within predefined boundaries (box 616).

[0115] In some embodiments, the AutoGO method may also use the Pareto front to construct a neural architecture best suited for multiple objectives (box 618). Those skilled in the art will understand that in multi-objective optimization, the Pareto front is the set of all Pareto-efficient solutions, representing the optimal trade-off among all objective functions (i.e., no operation or assignment can improve one objective without causing another objective to deteriorate).

[0116] The selected CG (either the modified CG or the input CG 542, depending on their performance comparison) can be used as the input CG in another iteration of optimization as described above.

[0117] Figure 10B illustrates a schematic diagram of the second-stage workflow 600' of the AutoGO method provided in some other embodiments of the present invention. The workflow 600' shown in Figure 10B is similar to the workflow 600 shown in Figure 10A, and the blocks with the same reference numerals are identical in both figures. Therefore, the following description focuses only on block 614 in Figure 10B.

[0118] As described above, mutation block 608 generates a modified CG (corresponding to a modified neural architecture). The AutoGO method can then update the modified neural architecture (block 614) as needed to maintain its correctness. This update can include: adjusting the relationship between the output of the preceding part of the modified CG (i.e., the portion of the modified CG before the replaced segment) and the input of the replaced segment, such that the output and input match as needed; and adjusting the relationship between the output of the replaced segment and the input of the following part of the modified CG (i.e., the portion of the modified CG after the mutation block), such that the output and input match as needed. For example, in computer vision applications, the AutoGO method can update the modified neural architecture to propagate resolution from the preceding part of the modified CG through the replaced segment to the subsequent part of the modified CG (described in more detail later).

[0119] B-5. Performance Estimation Using a PSC Predictor In some embodiments, each of the input CG 542 and the modified CG is first divided into a PSC format; then, at box 610, a PSC predictor 612 is used to estimate the performance variation between the input CG 542 and the modified CG, which is caused by mutation of one or more segments in the input CG 542.

[0120] Figure 11A shows a schematic diagram of the partitioning of input CG 542, where rounded rectangles represent individual segments. For clarity, Figure 11A shows only the identified single segment 606 (also referred to as segment) used for replacement. In this example, segment 606 includes two inputs 624, one output 626, and multiple operations 628.

[0121] Figure 11B illustrates the method of replacing segment 652 (also referred to as segment 652) with the replacement segment. The modified CG 642 obtained from the input CG 542 replaces the recognition segment 606, wherein the replacement segment 652 has two inputs 654, one output 656, and multiple operations 658. The replacement segment 652 has the same number of inputs and the same number of outputs as the segment to be replaced 606. However, the replacement segment 652 may have different operations 658 compared to the operations 628 of the recognition segment 606. Furthermore, the replacement segment 652 may be a segment extracted from a neural network that is completely different from the input CG 542.

[0122] As described above, after segment mutation or replacement, the PSC predictor 412 is used to estimate the performance of the modified CG 642 (or more precisely, the neural architecture corresponding to the modified CG 642). For this purpose, each of the input CG 542 and the modified CG 642 is partitioned into a PSC format. More specifically, before segment mutation, the segments of the input CG 542 are divided into three partitions: a pre-partition 632 comprising all segments preceding the identified segment 606, a segment partition 634 comprising the identified segment 606, and a successor partition 636 comprising all segments following the identified segment 606. After segment mutation, the segments of the modified CG 642 are also divided into three partitions: a pre-partition 632 comprising all segments preceding the segment 652, a segment partition 634 comprising the segment 652, and a successor partition 636 comprising all segments following the segment 652.

[0123] Then, the PSC format of the input CG 542 and the modified CG 642 is used to estimate the... replace The resulting performance differences. The PSC predictor 412 estimates the performance of the input CG 542. for:

[0124] in, P represents the preceding partition 632, and C represents the succeeding partition 636; GNN(X) represents taking CG (the whole graph or a subgraph) X as input (X can be P in equation (1)). Or C) and calculate the vector of fixed length d. Graph neural networks, i.e. , MLP(X) represents a multilayer perceptual ANN that receives vector Y and maps vector Y to prediction y; (Those skilled in the art will understand that MLP is a feedforward ANN used to generate a set of outputs from a set of inputs; the characteristic of MLP is that it has several layers of input nodes connected in the form of a directed graph between the input layer and the output layer.) Cascade [Z1, Z2, …] represents a cascade operation on inputs Z1, Z2, …, with outputs Z1, Z2, … being a cascade. In equation (2), the cascade operation combines three… Combine into a longer vector .

[0125] In other words, the PSC predictor 612 uses a GNN with each of the preceding partition 632, segment partition 634, and successor partition 636 of the input CG 542 as input to compute a fixed-length vector for each of these partitions. Then, the PSC predictor 612 concatenates the vectors from the preceding partition 632, segment partition 634, and successor partition 636 to form a concatenated vector, and generates a performance estimate of the input CG 542 based on this concatenated vector (e.g., using an MLP ANN with the concatenated vector as input).

[0126] Similarly, the performance of the modified CG 642 It can be estimated as follows:

[0127] In equation (3), P and C represent the pre-partition 632 and the successor partition 636 of the modified CG 642.

[0128] In other words, the PSC predictor 612 uses a GNN with each of the preceding partition 632, segment partition 634, and succeeding partition 636 of the modified CG 642 as input to compute a fixed-length vector for each of these partitions. Then, the PSC predictor 612 concatenates the vectors from the preceding partition 632, segment partition 634, and succeeding partition 636 of the modified CG 642 to form a concatenated vector, and generates a performance estimate of the modified CG 642 based on this concatenated vector (e.g., using an MLP ANN with the concatenated vector as input).

[0129] Then, compare the estimated performance. and To determine whether the performance of the input CG 542 is affected by this segment variation (i.e., by using...). replace And how much it has improved.

[0130] In some embodiments, one or more segment variations may be tested, the optimal segment variation (i.e., the segment variation that maximizes performance) may be selected, and the optimal segment variation may be output from box 610. When multiple segment variations are tested, at least some of the segment variations may replace the same segment with different replacement segments, and / or at least some of the segment variations may replace different segments with different or the same replacement segments.

[0131] The aforementioned PSC format supports subtle variations, such as altering a small portion of the neural architecture. This PSC format enhances the sensitivity of the PSC predictor to minute changes, enabling it to recognize the location of mutations and their context (precursor and successor parts). Furthermore, the PSC format builds upon the aforementioned CG network representation by encoding important details such as tensor dimensions and computational weight dimensions.

[0132] B-6. Mixed-integer linear programming (MILP) for resolution propagation. As described above, the AutoGO method uses FSM and BPE to mine subgraphs from different architecture families. The mined subgraphs differ in topology and node names (i.e., operations). The mining process does not consider characteristics of the operations, such as tensor height, width, and channel resolution in computer vision applications, which can vary significantly across different search spaces and individual networks.

[0133] When optimizing the input CG 402 by mutating the segment, the AutoGO method uses the constraint that the segment to be replaced and the replacement segment have the same number of inputs and the same number of outputs. However, mismatches may exist in the input / output of the preceding, subsequent, and final parts. In other words, the output of the preceding part of the modified CG may not match the input of the replacement segment (i.e., it may not meet the input requirements of the replacement segment), and the output of the replacement segment may not meet the input requirements of the final part of the modified CG. Therefore, additional steps (box 614 in Figure 10B) may be required to maintain the correctness of the sub-neural network (i.e., the neural network corresponding to the modified CG) after mutation.

[0134] For example, when a neural network is used for computer vision, the set of (height, width, channel) resolutions of the output of the pre-processor of the modified CG may not meet the required set of (height, width, channel) resolutions of the input of the replacement segment, and the set of (height, width, channel) resolutions of the output of the replacement segment may not meet the required set of (height, width, channel) resolutions of the input of the subsequent processor of the modified CG.

[0135] Figure 12 shows a schematic diagram of a portion of the modified CG 642 used to illustrate the resolution mismatch problem. This portion of the modified CG 642 has two paths and includes nodes 702 to 706 belonging to the preceding partition 622, a replacement segment 624 with nodes 708 to 720, and a successor partition 626 with node 722. The output resolution of each operation (i.e., each node) is in the form (h, w, c).

[0136] Replacement segment 652 requires its input dimension size (for the Conv and BN operations on its input nodes 708 and 710, respectively) to be (32, 32, 16), which is the resolution of the output nodes 704 and 706 of the preceding partition 622. Furthermore, the successor partition 626 requires its input dimension size to be (16, 16, 32), which requires replacement segment 624 to output a feature map with this input dimension size at output node 720 (Add operation).

[0137] Therefore, in some embodiments, the AutoGO method can adjust the relationship between the output of the pre-processor portion 632 of the modified CG 642 and the input of the replaced segment 652, such that the output and input can be matched as needed, and adjust the relationship between the output of the replaced segment 652 and the input of the successor portion 636 of the modified CG 642, such that the output and input can be matched as needed. For example, in computer vision applications, the AutoGO method can maintain the correctness of the neural network after mutation by propagating a suitable set of resolutions (height, width, channels) from the pre-processor portion 632 of the modified CG 642 to the replaced segment 652 of the modified CG 642, and then to the successor portion 636 of the modified CG 642.

[0138] In the following description, some variables are used, including: h: Tensor "height", is a natural number.

[0139] w: Tensor "width", is a natural number.

[0140] c: Tensor "channel", which is a natural number.

[0141] hwc: Tensor resolution, a tuple of height, width, and number of channels used for flow control in a convolutional network. For example, an input image in RGB color format might have c=3, meaning the input image has three channels, including one for each of the red, green, and blue color intensities. For example, if h=w=32, then hwc=(32, 32, 3).

[0142] In some embodiments, resolution propagation can be performed by changing the operator nodes in the mutation segment. Operators are either mutable or immutable. Table 3 enumerates several commonly used operator node types and describes their ability to modify the h, w, and c properties of the input tensor.

[0143] Table 3: Exemplary computation nodes and whether they can modify the h, w, and c of the input tensor

[0144] Therefore, the resolution propagation problem can be viewed as a search task for the hwc value of each operation in the mutation segment to achieve correct resolution propagation. In some embodiments, the solution to this problem is based on mixed-integer linear programming (MILP), where MILP is an optimization problem represented by linear objectives, linear constraints, and integer-valued variables.

[0145] In intermediate nodes 708 to 718 with operations Conv2d, BN, and ReLU, the nodes with operations BN and ReLU are immutable (i.e., h, w, and c cannot be changed). Therefore, the MLIP problem can be established and solved at box 614 (see Figure 10B) to adjust the resolution of the two variable Conv2d operation nodes 708 and 716 (each of nodes 708 and 716 is on the corresponding path).

[0146] In this example, since the Add operation of node 720 requires its input tensor to have precise dimensions, adjusting only one of the variable Conv2d operation nodes 708 and 716 or keeping neither node 708 nor 716 adjusted will result in incorrect propagation.

[0147] In some cases, such as if the segment does not include compute nodes that can adjust hwc in the desired manner, resolution propagation may fail. In such cases, the AutoGO method can determine that resolution propagation is not feasible and discard the modified CG. Otherwise, the AutoGO method generates guidelines indicating the compute nodes in the replacement segment that need to be adjusted for resolution propagation.

[0148] By using MILP for resolution propagation, segments collected by the FSM / BPE method can be utilized to pair segments from different search spaces, thereby further enabling segment mutation in different parts of the input CG. Furthermore, MILP can more efficiently determine whether resolution propagation is feasible or infeasible compared to enumerating all possible combinations of hwc values ​​for each operation.

[0149] In some embodiments, instead of changing or otherwise modifying the computation nodes in replacement segment 652, resolution propagation can be performed by changing the computation nodes in the preceding portion 632 and / or the succeeding portion 636 as needed.

[0150] B-7. Performance and Resource Metrics In some embodiments, the AutoGO method considers two objectives when optimizing the input CG: (1) improving performance (accuracy and / or peak signal-to-noise ratio, PSNR, etc.) while (2) reducing resource metrics (FLOPs, parameters, and / or power consumption, etc.). Typically, these objectives are conflicting. For example, optimization for accuracy metrics may lead to an increase in the number of FLOPs. Furthermore, optimization for one resource metric (e.g., FLOPs) may not correspond to optimization for all resource metrics (e.g., on-chip latency) because they may be hardware-dependent. Therefore, in these embodiments, the AutoGO method may further generate an optimized output neural architecture based on the hardware dependence of the relevant performance and resource metrics.

[0151] Table 4 lists some examples of performance and resource metrics that the AutoGO method might consider when optimizing neural architectures. In these examples, some performance metrics are relevant to computer vision tasks (e.g., accuracy might be a key metric in classification), while precision and recall are used for recognition.

[0152] Table 4: Examples of Performance and Resource Metrics

[0153] The ability to consider different performance / resource metrics during the optimization process allows the AutoGO method to be flexibly used in deployment environments, such as optimizing network design for deployment on specific product hardware.

[0154] B-8. Application of the AutoGO Method The AutoGO method disclosed herein can improve the performance of an ANN by optimizing the neural architecture of the corresponding CG. Those skilled in the art will understand that AI such as ANNs has greatly improved the functionality and performance of computing devices and computer systems. By optimizing the neural architecture, the AutoGO method disclosed herein can further improve the functionality and performance of computing devices and computer systems.

[0155] As described above, in some embodiments, the AutoGO method can be used for computer vision applications, such as computer vision applications involving mobile phone cameras.

[0156] For example, scene detection tasks might require a neural network model that classifies the scene of an image to be captured (e.g., portraits, nature, animals, daytime and / or nighttime, etc.) and adjusts the camera settings accordingly. Processing a frame requires latency L and energy consumption E, both of which are largely determined by the network design. Inappropriately minimizing L and E can lead to unintended degradation in the model's performance and accuracy.

[0157] In some embodiments, NAS (also referred to as "the NAS method disclosed herein") in conjunction with the AutoGO method disclosed herein can be used to optimize network design to minimize L and / or E while maximizing its performance. The input neural network is typically an independent, manually constructed design using expert knowledge for a specific task. In these embodiments, the AutoGO method can slightly adjust the input neural network to reduce L and / or E while maintaining its performance. The AutoGO method disclosed herein can also be used for other tasks requiring the design of high-performance, lightweight models, such as image denoising, image deblurring, and / or face recognition.

[0158] In some embodiments, the AutoGO method disclosed herein can be used in various computer vision products in Ascend provided by Huawei Technologies Co., Ltd. in Shenzhen, China, such as image classification, image segmentation, super-resolution, and image denoising.

[0159] When optimizing the CG of a neural architecture, the AutoGO method takes a Directed Acyclic Graph (DAG) as input and outputs one or more DAGs (i.e., one or more modified CGs). For example, the AutoGO method can generate the Pareto front of the neural network that performs best on multiple objectives (e.g., accuracy and FLOPs) and return all of these neural networks. The AutoGO method uses a search process guided by surrogate performance metrics estimated by a PSC predictor or otherwise measured to create new DAGs. Therefore, the AutoGO method can also be used as a general framework for DAG optimization.

[0160] The AutoGO method disclosed herein can be used in various fields of AI. For example, in some embodiments, the AutoGO method can be used to optimize CNNs in computer vision applications, such as face recognition, segmentation, and bounding box detection, which process images. In other embodiments, the AutoGO method can be used to optimize AI models that process text or speech rather than images in NLP-based applications. Those skilled in the art will understand that, compared to typical CNNs, ANNs (denoted as "NLP networks") in NLP-based applications are typically larger, require more resources (e.g., data and computing power) for training, and are more difficult to adapt to end-user mobile devices.

[0161] A popular building block for NLP networks is the attention-driven Transformer block, which has also found widespread application in computer vision tasks in the form of visual Transformer networks. Since Transformer-based models require significant resource investment for adequate training, ensuring their neural architecture is of high quality and suitable for minimizing resource burden is crucial. In some embodiments, the AutoGO method disclosed herein can be used to optimize Transformer-based networks by, for example, changing matrix multiplication operations in different parts of the network (e.g., reducing the weight dimension in less important network regions, ignoring portions of the Transformer block in certain regions to reduce computation). Furthermore, during optimization, the AutoGO method can be performed under strict resource constraints, such as ensuring that the CG of the neural architecture is small enough to adapt to the chosen target hardware (e.g., the Da Vinci platform and Huawei chipsets).

[0162] In various embodiments, the AutoGO method disclosed herein can be used in various technical fields to optimize various DAGs (not limited to DAGs or CGs of neural architectures). For example, in some embodiments, the AutoGO method can be used in the field of communications, where the AutoGO method can use a communication network mesh as input DAG and generate an optimized or best-fit routing graph between two endpoints. In these embodiments, each edge in the DAG can be associated with the communication cost, and the overall goal can be to reduce the total communication cost. In some embodiments, the AutoGO method deployed in one technical field can be transferred to another technical field, but the input / output format needs to be changed as needed, the segment database repopulated, and the PSC predictor retrained.

[0163] In the above embodiments, various examples of operations such as Conv2d, BN, and ReLU are described. Those skilled in the art will understand that the AutoGO method disclosed herein is not limited to processing the above operations. In other words, the AutoGO method in the various embodiments can be used to process neural architectures and their CGs that have other operations such as parametric rectified linear units (PReLU) and / or Gaussian error linear units (GeLU). The AutoGO method in the various embodiments can be used to process neural architectures and their CGs that have segments from other fields such as NLP and / or involve operations such as recurrent neural networks (RNNs), Transformers, and / or self-attention.

[0164] In some embodiments, the segment database 404 used in the AutoGO method may include any suitable CG segments, such as segments extracted from CGs of neural architectures used in various technical fields (rather than segments extracted from CGs of neural architectures used in a specific technical field).

[0165] In various embodiments, the AutoGO method disclosed herein can handle various CG formats such as ".pb", ".onnx", and / or Keras.

[0166] In the above embodiments, the AutoGO method combines the FSM method with the BPE method to extract segments from the input CG. In some other embodiments, other suitable frequent subgraph mining methods can be used to extract segments from the input CG.

[0167] Therefore, the AutoGO method disclosed in this paper yields an improved NAS approach with multiple objectives, namely, finding architectures that combine high performance with hardware friendliness (e.g., fewer FLOPs, parameters, latency, and / or energy). The AutoGO method also utilizes an evolutionary algorithm for the search and employs a predictor to estimate the performance of subnetworks during the search process.

[0168] Although the NAS method disclosed in this paper may be similar to some existing NAS methods (e.g., DARTS, SNAS, NASNet, ENAS, NAO, GDAS, PC-DARTS, GA-NAS, L2NAS, Predictor-NAS, OFA, etc.), the NAS method disclosed in this paper has been improved in various ways and is very different from existing NAS methods.

[0169] More specifically, the following lists some similarities and differences between the NAS method disclosed in this paper and existing NAS methods: Both existing NAS methods and the NAS method disclosed in this paper output architectures with high performance (i.e., similar outputs), as this is the common goal of all NAS methods.

[0170] Some existing NAS methods (e.g., predictor-NAS and NAO) also rely on performance predictors as performance proxies. However, the predictors of these existing NAS methods are only applicable to their predefined search spaces.

[0171] Some existing NAS methods also mine optimal subgraphs from neural networks. However, compared to the NAS method disclosed in this paper, these existing NAS methods use different approaches, employing shallow graph representations of neural networks and being limited to the DARTS search space.

[0172] The NAS method disclosed herein differs significantly from existing NAS methods. For example, in various embodiments, the NAS method disclosed herein uses CG in NAS and creates a searchable space by mining computational cells rather than manually predefining computational cells.

[0173] The NAS method disclosed in this paper progressively improves upon a given input neural architecture, while existing NAS techniques train the network from scratch based on a given dataset and task.

[0174] The NAS method disclosed in this paper solves at least some of the following problems existing in the prior art: Existing NAS search spaces are inflexible because they are built on expert rules and heuristics. A neural network from one search space cannot be used for software understanding of different search spaces. In existing technologies, the granularity of changes that can be made to a neural network in a search space is predefined, which can bias and limit the range of possible performance changes (e.g., accuracy) or resource cost changes (e.g., FLOPs or on-chip latency). By using CG to represent neural architectures, the AutoGO method disclosed in this paper understands the structure of neural networks (or independent neural networks) from different search spaces and modifies the input CG and its granularity using subgraph-level mutations with a variable number of nodes / edges.

[0175] The goal of NAS is to automate the design and optimization of neural networks. This goal is limited by the use of manually designed and artificially constructed search spaces. The AutoGO method disclosed in this paper circumvents this limitation by performing coarse-to-fine CG modifications, allowing for the careful customization of neural architectures to be tailored to specific, dedicated hardware.

[0176] In existing technologies, the search space in NAS requires a predefined set of searchable units at certain locations within the network structure. For NAS to be suitable for execution on CG, these units need to be defined on subgraphs, which is a costly task as it involves traversing several subgraph locations within the CG. FSM can be used to discover patterns of interest or frequent patterns in a graph. Typical FSM requires traversing many possible subgraphs, which can lead to high computational complexity when dealing with graphs like CG with many nodes and edges. By combining FSM with NLP techniques such as BPE, the AutoGO method disclosed in this paper leverages topology search and NLP techniques to perform efficient FSM using a DAG structure.

[0177] In the NAS search process, existing NAS methods rely on performance predictors to predict the performance of the selected architecture. Such predictors depend on the dataset (e.g., CIFAR10 or CIFAR100) and the task (e.g., image classification). Furthermore, existing predictors are often unaware of the location of changes within the network and their context, yet the location of computation can significantly impact the network's performance or latency response. The AutoGO method disclosed in this paper uses a unique PSC predictor that encodes the locations of changes.

[0178] Tensor resolution (e.g., height, width, and number of channels) varies across different search spaces and neural networks, posing a challenge to editing segments from one search space to another. To address this issue, the AutoGO method disclosed in this paper uses resolution propagation MILP to evaluate the feasibility of CG mutation.

[0179] The AutoGO method disclosed in this paper aims to extend the scope of NAS beyond the search spaces and computer vision tasks provided in the literature. The AutoGO method uses a generalizable format to combine aspects of multiple search spaces and can optimize independent architectures for specific performance metrics, such as power consumption.

[0180] The AI ​​system 100 and NAS method disclosed in this paper have various advantages. For example: By using CG to represent and modify neural architectures, the NAS method disclosed in this paper achieves search space generalization, thereby enabling the combination of information from different search spaces and the computation of neural networks outside these search spaces and in different tasks.

[0181] By combining FSM with BPE, the NAS method disclosed in this paper extracts segments with different characteristics in terms of size, topology, and / or granularity. The efficiency of the NAS method disclosed in this paper is improved by applying a combined FSM and BPE to the topologically ordered sequence format of the graph.

[0182] The NAS method disclosed in this paper uses a PSC format predictor that is aware of the location and context of segment variations. The PSC format predictor is trained to be sensitive to variations in the neural network (e.g., computational and resolution variations), thus enabling it to rank neural architectures with high accuracy based on their performance.

[0183] By establishing and solving the MILP problem, the NAS method disclosed in this paper is applicable to processing segments with different resolutions corresponding to different parts of the neural network and with strictly defined search spaces. This enables the combination of segments in different search spaces and the utilization of knowledge between different families of neural architectures. In other words, subgraphs mined from family X can help improve the performance of architectures from family Y.

[0184] The NAS method disclosed in this paper achieves robust optimization and can optimize both performance and resource metrics, with flexibility in the selection of performance and resource metrics.

[0185] C. Abbreviations

[0186] D. Definitions of some technical terms Neural Architecture Search (NAS): An automated architecture search method or process (rather than a manual trial-and-error process) for finding neural network models with improved performance.

[0187] Computational graph (CG): A directed acyclic graph that describes the types of computations (convolution, addition, ReLU, and / or maxpool, etc.) and the flow of computation within a neural network.

[0188] Computer vision (CV): A field of artificial intelligence that trains computers to interpret and understand the visual world.

[0189] Natural language processing (NLP): A branch of artificial intelligence or AI that provides computers with the ability to understand text and spoken language in the same or similar way as humans.

[0190] Byte-pair encoding (BPE): A robust data compression algorithm.

[0191] Floating-point operation (FLOP): The total number of addition and multiplication operations, usually defined for neural networks.

[0192] Mixed-integer linear programming (MILP): A mathematical optimization program in which some or all of the variables are restricted to integers.

[0193] Graph neural network (GNN): A special type of neural network designed for graphical input.

[0194] A directed acyclic graph (DAG) is a graph with directed edges but no cycles.

[0195] Predecessor, segment, successor (PSC): According to some aspects of the present invention, a graph coding technique is used to encode a subgraph of a graph CG while considering the preceding and succeeding subgraphs of the subgraph in the neural network.

[0196] Convolutional Neural Networks (CNNs): A special type of neural network designed for image input. Currently, there are many manually designed CNN families, such as ResNet, Inception, MobileNet, and MobileNext. The main differences between CNN families lie in the number of blocks / operators in the network, the type of convolutional blocks / operators, and the network topology (i.e., operator connection patterns).

[0197] Convolution operator: A commonly used operator in CNNs. It has trainable weights and is therefore responsible for learning generalizable knowledge from the input. Convolution operators have many properties, such as input / output channel size, kernel size, stride, dilation rate, and / or grouping, which can affect the operator's computational cost and learning ability.

[0198] Vision Transformer (ViT): A neural network designed for image input. Its performance surpasses that of many CNNs.

[0199] This document describes various embodiments of an AI system 100 and an AutoGO method for NAS. Those skilled in the art will understand that the various embodiments and / or features thereof described above can be customized and / or combined as needed or required. Furthermore, while embodiments have been described above with reference to the accompanying drawings, those skilled in the art will understand that variations and modifications can be made without departing from the scope defined by the appended claims.

Claims

1. A computerized method, characterized in that, include: Generate an input computational graph (CG) for representing a neural network architecture, wherein the input CG includes one or more nodes, each node corresponding to an operation; perform multiple optimization steps in at least one iteration to obtain a modified neural network architecture represented by a modified CG, thereby obtaining a neural network for one or more computing devices; wherein the multiple optimization steps include: identifying one or more subgraphs in the input CG; obtaining the modified CG by replacing the identified one or more subgraphs with one or more alternative subgraphs respectively; evaluating one or more metrics of the modified neural network architecture represented by the modified CG; and selecting the modified CG or the input CG based on the evaluation; wherein, when the multiple optimization steps are performed in multiple iterations, the selected CG obtained in one of the multiple iterations is used as the input CG in the next iteration of the multiple iterations.

2. The computerized method according to claim 1, characterized in that, Also includes: Adjust a first relationship between one or more inputs of each replacement subgraph and one or more outputs of the modified CG, located in a first portion preceding the replaced segment of the modified CG, such that the one or more inputs match each other; and / or adjust a second relationship between one or more outputs of each replacement subgraph and one or more inputs of the modified CG, located in a second portion following the replaced segment of the modified CG, such that the one or more outputs match each other.

3. The computerized method according to claim 1 or 2, characterized in that, Also includes: Mixed-integer linear programming (MILP) is used to adjust a first relationship between one or more inputs of each replacement subgraph and one or more outputs of the modified CG, located in a first portion preceding the replaced segment of the modified CG, such that the one or more inputs match the one or more outputs; and / or a second relationship is adjusted between one or more outputs of each replacement subgraph and one or more inputs of the modified CG, located in a second portion following the replaced segment of the modified CG, such that the one or more outputs match the one or more inputs.

4. The computerized method according to any one of claims 1 to 3, characterized in that, The evaluation of the one or more metrics of the modified neural network architecture represented by the modified CG includes: evaluating the one or more metrics of the modified neural network architecture represented by the modified CG under one or more constraints.

5. The computerized method according to any one of claims 1 to 4, characterized in that, The selection of the modified CG or the input CG includes: using the Pareto front to select the modified CG or the input CG.

6. The computerized method according to any one of claims 1 to 5, characterized in that, The identification of the one or more subgraphs in the input CG includes: forming a triplet for each of the one or more nodes in the input CG, wherein the triplet includes the operation of the node, the operation of the node's in node, and the operation of the node's out node; encoding each triplet as a symbol according to the node traversal order to obtain a symbol sequence; and identifying one or more substrings by searching the one or more subgraphs in the symbol sequence, wherein each of the one or more substrings represents a corresponding subgraph in the one or more subgraphs.

7. The computerized method according to claim 6, characterized in that, The input CG is a directed acyclic graph (DAG) having multiple nodes and one or more directed edges between a pair of nodes among the multiple nodes; the computerized method further includes: topologically indexing the input CG by assigning a number to each of the multiple nodes, wherein for each directed edge from node i to node j among the multiple nodes, the number of node i is less than the number of node j, and the numbers assigned to the multiple nodes form the node traversal order.

8. The computerized method according to any one of claims 1 to 7, characterized in that, Each subgraph in the identified one or more subgraphs and its corresponding subgraph in the one or more replacement subgraphs have the same number of inputs and the same number of outputs.

9. The computerized method according to any one of claims 1 to 8, characterized in that, The evaluation of the modified neural network architecture represented by the modified CG includes: dividing the modified CG into segment partitions comprising the one or more replacement subgraphs, a preceding partition preceding the segment partition, and a succeeding partition following the segment partition; using a graph neural network (GNN) with each of the preceding partition, the segment partition, and the succeeding partition as input to compute a fixed-length vector for each of the preceding partition, the segment partition, and the succeeding partition; concatenating the vectors of the preceding partition, the segment partition, and the succeeding partition to form a concatenated vector; and generating a performance estimate of the modified CG based on the concatenated vector.

10. The computerized method according to claim 9, characterized in that, The performance estimate for generating the modified CG based on the concatenated vectors includes: generating the performance estimate for the modified CG using a multi-layer perception (MLP) artificial neural network (ANN) with the concatenated vectors as input.

11. An apparatus, characterized in that, include: One or more processors; One or more non-transitory computer-readable storage devices that store computer-executable instructions, which, when executed by the one or more processors, cause the one or more processors to perform a computerized method according to any one of claims 1 to 10.

12. One or more non-transitory computer-readable storage devices, characterized in that, It includes computer-executable instructions, wherein, when executed, the instructions cause one or more processors to perform the computerized method according to any one of claims 1 to 10.

13. A computerized method, characterized in that, include: Generate an input CG for representing a neural network architecture, wherein the input CG includes one or more nodes, each node corresponding to an operation; form a triplet for each of the one or more nodes of the input CG, wherein the triplet includes the operation of the node, the operation of the node's input node, and the operation of the node's output node; encode each triplet into a symbol according to the node traversal order to obtain a symbol sequence; collect one or more unique substrings from the symbol sequence; convert the collected substrings into the plurality of subgraphs; and store the plurality of subgraphs to optimize one or more target neural network architectures.

14. The computerized method according to claim 13, characterized in that, The input CG is a directed acyclic graph (DAG) having multiple nodes and one or more directed edges between a pair of nodes among the multiple nodes; the computerized method further includes: topologically indexing the input CG by assigning a number to each of the multiple nodes, wherein for each directed edge from node i to node j among the multiple nodes, the number of node i is less than the number of node j, and the numbers assigned to the multiple nodes form the node traversal order.

15. An apparatus, characterized in that, include: One or more processors; One or more non-transitory computer-readable storage devices that store computer-executable instructions, which, when executed by the one or more processors, cause the one or more processors to perform the computerized method according to claim 13 or 14.

16. One or more non-transitory computer-readable storage devices, characterized in that, It includes computer-executable instructions, wherein, when executed, the instructions cause one or more processors to perform the computerized method according to claim 13 or 14.

Images