Optimization of executable graphs for artificial intelligence model reasoning

Abstract

The invention relates to optimization of executable graphs for AI model reasoning. The optimization method may include: copying executable graphs to generate a number M of identical executable graphs; determining, from the executable graph, one or more nodes eligible for optimization based on inference throughput related parameters associated with an inference device for performing AI model inference; and generating an optimized executable graph for AI model reasoning by optimizing one or more nodes from each of the same number M of executable graphs. Here, M is an integer in a range from 2 to a maximum number N of permitted executable graphs, and N is an integer estimated or manually configured based on the memory size of the inference device and the size of the executable graphs.

Description

Technical Field

[0001] Embodiments described herein relate generally to artificial intelligence (AI) technology, and more specifically, to methods and apparatus for optimizing executable graphs for AI model reasoning. Background Art

[0002] With the rapid development of AI technology in recent years, more and more excellent trained models are deployed to replace traditional algorithms. Even many cloud companies have provided customers with a variety of AI services. Whether on a local server or on a cloud platform, the general reasoning process can include converting the trained model into an executable graph and running the trained model on a primitive implementation kernel according to the executable graph. Both the executable graph and the primitive implementation kernel play a key role in improving reasoning performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0003] Various advantages of the embodiments will become apparent to those skilled in the art upon reading the following description and appended claims, and by referring to the following drawings, in which:

[0004] Figure 1 The general processing flow of AI model reasoning is shown;

[0005] Figure 2 shows an example node with low resource utilization in an executable graph for AI model inference;

[0006] Figure 3 The optimized processing flow of AI model reasoning according to some embodiments of the present disclosure is shown;

[0007] Figure 4 An example flow chart showing an executable graph optimization process according to some embodiments of the present disclosure is shown;

[0008] Figure 5 An example optimization for improving instruction utilization by merging multiple identical nodes from multiple executable graphs is shown according to some embodiments of the present disclosure;

[0009] Figure 6 shows a comparison of example dumped assembly code corresponding to a conventional executable graph with low register utilization and an optimized executable graph with improved register utilization, respectively, according to some embodiments of the present disclosure;

[0010] Figure 7 An example executable graph optimization process implemented by memory management and executable graph modification according to some embodiments of the present disclosure is shown;

[0011] Figure 8An example optimized executable graph for AI model reasoning according to some embodiments of the present disclosure is shown;

[0012] Fig. 9 An example flow chart showing an executable graph optimization process according to some embodiments of the present disclosure is shown;

[0013] Fig.10 is a block diagram illustrating components according to some example embodiments that are capable of reading instructions from a machine-readable medium or a computer-readable medium and performing any one or more of the methodologies discussed herein;

[0014] Fig.11 is a block diagram of an example processor platform according to some embodiments of the present disclosure. DETAILED DESCRIPTION

[0015] The various aspects of the illustrative embodiments will be described using terms commonly used by those skilled in the art to convey the essence of the present disclosure to other persons skilled in the art. However, it will be apparent to those skilled in the art that many alternative embodiments may be practiced using portions of the described aspects. For purposes of explanation, specific quantities, materials, and configurations are set forth to provide a thorough understanding of the illustrative embodiments. However, it will be apparent to those skilled in the art that alternative embodiments may be practiced without these specific details. In other instances, known features may be omitted or simplified to avoid blurring the illustrative embodiments.

[0016] In addition, various operations will be described as multiple discrete operations in sequence in a manner that is most helpful for understanding the illustrative embodiments; however, the order of description should not be interpreted as implying that these operations are necessarily order-dependent. In particular, these operations do not need to be performed in the order presented.

[0017] Figure 1 The general processing flow of AI model reasoning is shown, which can be implemented on a local server or a cloud platform. Figure 1As shown, the trained model can be converted to a high-level graph after a series of transformations and optimization passes. The trained model can be any of a variety of trained models from different training frameworks (e.g., TensorFlow, PyTorch), and all these transformations and optimization passes are usually device-independent. The high-level graph can then be sent to a device-dependent back-end optimizer for a variety of graph optimizations, such as fusion, layout changes, reordering insertions, primitive kernel selections, partitions, memory allocation checks, and the like. After these optimizations, an executable graph can be generated and output by a device-dependent back-end optimizer. Finally, the reasoning application can call a primitive implementation kernel such as OneDNN to perform AI model reasoning based on the output executable graph. It can be seen that both the executable graph and the primitive implementation kernel play a key role in reasoning performance.

[0018] Since reasoning is computationally intensive, the workload throughput of AI model reasoning is a very important factor affecting the market competitiveness of devices used to perform AI model reasoning (e.g., central processing units (CPUs), graphics processing units (GPUs), tensor processing units (TPUs), vision processing units (VPUs), or field programmable gate arrays (FPGAs)). The theoretical throughput can be obtained from a roofline model or estimated by the processing power of the device (e.g., trillions of operations per second (TOPS)) and the computational complexity of the AI ​​model. However, in actual usage scenarios, the actual performance of AI model reasoning may be much lower than the theoretical value. In other words, the processing power of the device to perform reasoning may not be fully utilized during the actual reasoning process.

[0019] Figure 2 An example node with low resource utilization in an executable graph for AI model inference is shown. Figure 2 As shown in part (a) of , the example node can perform a MatMul operation by using a block matrix multiplication unit (TMUL) instruction (e.g., an AMX TDPBUSD instruction). In this example, since the shape of the input matrix A of the node is 9×64, the MatMul operation cannot maximize the utilization of the TMM registers supported by the AMX instruction set. Figure 2In part (b) of FIG. 1 , a node for performing a convolution operation in an executable graph of an INT8 quantized network security model is shown as an example. In this example, when a vector neural network instruction (VNNI) instruction (e.g., an AVX512-VNNI VPDPBUSD instruction) in the OIhw4i16O4i format is used to perform a convolution operation, since the input channel (IC) of the input data of the node is 1 (less than 4), the convolution operation cannot fully utilize the capability of the VNNI instruction.

[0020] from Figure 2 It can be seen from the example shown that in the actual reasoning process, the processing power of the device used to perform the reasoning may not be fully utilized, so the reasoning performance may be lower than the expected performance. Of course, for the second example, the implementation of the convolution primitive kernel can be manually modified for this particular case. But if the IC of the input data is equal to 2 or 3 instead of 1, the implementation of the convolution primitive kernel may need to be modified again. In other words, the implementation of the primitive kernel may need to be modified for many special cases, and the primitive kernel may be subject to various special treatments. Similarly, for the first example, special modifications are required to store enough data for the TMM registers and manage the partitioning of the data. Open source primitive kernels often do not accept these special processing patches because they are completely different from the general processing code of the kernel, and a large amount of workaround code must be added for each special case. Therefore, how to improve the throughput of AI model reasoning on a device for performing reasoning (hereinafter referred to as "inference device") without making special modifications to the original primitive implementation kernel remains a problem.

[0021] According to the present disclosure, a method and apparatus for optimizing an executable graph for AI model reasoning are proposed to improve the throughput of AI model reasoning by optimizing inference throughput-related parameters associated with an inference device, thereby, for example, improving instruction utilization and / or register utilization, reducing cache miss rates, etc. without changing the general processing code of a primitive implementation kernel.

[0022] As mentioned above, both the executable graph and the primitive implementation kernel play a key role in inference performance. Figure 1 An advanced executable graph optimizer for generating an executable graph from a common graph optimizer is described to optimize the executable graph based on characteristics of multiple identical executable graphs and an inference device, thereby improving inference performance.

[0023] Figure 3 FIG. 2 shows an optimized processing flow of AI model reasoning according to some embodiments of the present disclosure. Figure 3 As shown, the sub-process in the dashed box can be added as Figure 1The general process of AI model reasoning shown in the figure is used to obtain the optimized processing flow of AI model reasoning. Note that the proposed executable graph optimizer is different from the common executable graph optimizer in that the proposed executable graph optimizer can be configured to: copy the executable graph generated by the common graph optimizer to generate M identical executable graphs; determine one or more nodes eligible for optimization from the executable graph based on the inference throughput-related parameters associated with the inference device; and generate an optimized executable graph for AI model reasoning according to M identical executable graphs by optimizing one or more execution nodes in each of the M identical executable graphs. Here, M can be an integer in the range of 2 to the maximum number of executable graphs allowed, N, and N can be an integer estimated or manually configured based on the memory size of the inference device and the size of the executable graph.

[0024] To better understand the proposed solution for executable graph optimization, reference will be made to Figure 4 Describe in detail the operation of the proposed executable graph optimizer, Figure 4 An example flow chart of an executable graph optimization process according to some embodiments of the present disclosure is shown.

[0025] First, at operation 410, the maximum number N of executable graphs allowed can be roughly estimated. For example, N can be estimated based on the memory size of the inference device used to perform AI model inference and the size of the executable graph to be optimized. Alternatively, the user can also manually configure the parameter according to actual conditions. Therefore, the maximum number N of executable graphs allowed can be determined as follows.

[0026]

[0027] At operation 420, multiple (N-times) iterations for executable graph optimization may be started to find the best executable graph. Specifically, for the Mth iteration (1<M≤N), the executable graph may be copied to generate M number of identical executable graphs, and operation 430 may be performed on the M number of identical executable graphs. At operation 430, optimization rounds may be performed on the combination of the M number of identical executable graphs to generate a new executable graph. Figure 4 As shown, the optimization pass performed at operation 430 may include instruction utilization checking, register utilization checking, graph memory management, affinity checking, and new executable graph generation, which will be referred to as Figures 5 to 8 Describe in detail.

[0028] In this way, by increasing the number M from 2 to N, a number (N-1) of optimized executable graphs can be generated. At operation 440, the best optimized executable graph with the highest inference throughput can be selected from the number (N-1) of optimized executable graphs and stored. For example, once a new executable graph is generated in a new iteration, the new executable graph can be compared with the stored executable graph with the current highest inference throughput, and if the inference throughput associated with the new executable graph is better than the current highest inference throughput, the stored executable graph is replaced with the new executable graph, otherwise, the next iteration can be performed until the Nth iteration is performed, and the best optimized executable graph with the highest inference throughput is thus obtained.

[0029] After completing the iteration, the best optimized executable graph and the corresponding number N' (1 < N' ≤ N) of executable graphs that have been combined to obtain the best optimized executable graph may be determined. Since the optimization rounds in operation 430 may affect the memory shape and layer execution when generating a new executable graph from multiple graphs, the latency and memory cost may change in order to improve the inference throughput, a threshold may be set to measure whether it is worthwhile to improve the inference throughput by combining multiple executable graphs.

[0030] Therefore, the highest inference throughput by using the best optimized executable graph may be compared with the reference inference throughput at operation 450. If the improvement of the highest inference throughput compared to the reference inference throughput is greater than a threshold, the best optimized executable graph may be output as an optimized executable graph for AI model inference, otherwise, the previous single executable graph generated by a common executable graph optimizer may still be used for AI model inference.

[0031] Note that the threshold may be different for different numbers N' (1<N'≤N) of executable graphs to be combined, since different N' have different effects on latency and memory cost. For example, the inference throughput of the best optimized executable graph may be compared to the throughput of AI model inference by using a previous single executable graph in batch mode with a batch size of N', which is more challenging than the throughput of AI model inference by using a previous single executable graph in normal mode. For example, for N'=2, the threshold may be set to 8%, and for N'=4, the threshold may be set to 15%. Therefore, if the best optimized executable graph combined from four identical executable graphs is selected as the optimized executable graph for AI model inference, we expect the inference throughput to be at least 15% improved compared to the inference throughput by using a previous single executable graph in batch mode with a batch size of 4.

[0032] Referenced above Figure 4An example process for executable graph optimization is discussed, but it should be noted that not all operations in the example process are required to perform executable graph optimization. In other words, some operations are provided to make the performance of executable graph optimization better, but these operations may not be included in the process, or may be simplified or replaced with other operations. For example, according to some embodiments, the number of identical executable graphs to be combined to obtain an optimized executable graph may be predefined based on an analysis of the processing power of an inference device and the structure of the executable graph to be optimized. In this case, it may not be necessary to perform all (N-1) iterations. For another example, the threshold for measuring the performance of executable graph optimization at operation 450 may be determined in different ways, or even operation 450 for measuring the performance of executable graph optimization may be omitted in specific usage scenarios.

[0033] In fact, as mentioned above, the basic idea of ​​the proposed executable graph optimizer is to generate an optimized executable graph for AI model reasoning from a specific number of identical executable graphs by optimizing one or more nodes eligible for optimization from multiple identical executable graphs, which can be achieved by Figure 4 This is accomplished by the optimization rounds performed at operation 430 as shown.

[0034] The following will refer to Figures 5 to 8 Describes optimization passes, such as instruction utilization checking, register utilization checking, graph memory management, affinity checking, and new executable graph generation.

[0035] First, it is required to determine which nodes are eligible for optimization from the executable graph. In other words, the reasoning performance can be improved by optimizing these nodes. According to some embodiments, the instruction utilization and register utilization at each node of the executable graph can be checked to determine whether the instruction utilization and register utilization at the node reach the expected level according to the processing power of the reasoning device and the operation at the node and the input data structure and output data structure.

[0036] The dimensionality of the input data for each node of the executable graph can directly affect the efficiency of data loaded by registers, thereby affecting the utilization of the instruction set. Figure 2 As shown in part (a) of , for the MatMul node, the shape of the input matrix A is 9×64, where the data type is INT8. For the default executable graph, when using the TDPBUSD / TDPBUUD / TDPBSSD instructions, the MatMul kernel will use a configuration of M=9 for the TMM registers. If an instruction uses a total of 3 TMM registers out of 8 TMM registers, this may be very wasteful. Therefore, the node can be determined as a node eligible for optimization, and multiple identical nodes from multiple identical executable graphs can be merged to improve instruction utilization, which is determined by Figure 5 As shown. Figure 5 As shown, by merging three MatMul nodes, the MatMul core can use a configuration of M'=16 for TMM registers and thus can improve instruction utilization.

[0037] It should be noted that data size checks can be performed from both input and output perspectives, and any node in any layer of the executable graph that satisfies one of the optimization conditions can be eligible for optimization. From the input perspective, if any key dimension size of the input data at the node is smaller than the corresponding dimension size of the input data determined by the SIMD instruction that consumes a lot of calculations and the data type of the input data, the node can be eligible for optimization. For example, those compute-intensive primitive kernels that use VNNI instructions with INT8 data types may need to compare the input channel size with 4 and the output channel size with 16; those primitive kernels that use TMUL instructions with INT8 data types may need to check whether M is less than 16, K is less than 64, and N is less than 16; and those layers that use vfmadd213ps instructions with FP32 data types for quantization may need to check whether the input channel size is less than 16. In other words, the instruction utilization check may include comparing the key dimension sizes of the input data and output data at the node with the corresponding dimension sizes of the input data and output data determined by the SIMD instruction to be executed at the node and the data type of the input data.

[0038] For example, for the input data of the node, the optimization condition can be given by express.

[0039] In addition to instruction utilization checks, register utilization checks can also be performed to determine which nodes are eligible for optimization. In practice, primitive implementation kernels are often expanded in a particular dimension (e.g., input channels, output height, or output width) during implementation. The compiler can automatically generate assembly code for execution. It has been found that the generated assembly code includes repetition of basic code units in the expansion dimension, as shown in Figure 2. Figure 6 For most register allocation strategies, if the expansion width is not enough, it will result in Figure 6 The problem of low register utilization problem shown in part (a) of FIG.

[0040] Due to insufficient unrolling width, a basic code unit may use only a small number of ZMM registers, and the rest of the registers will be free. In batch mode, although multiple input frames can be sent at once, the repetition of the basic code unit remains unchanged, as Figure 6In contrast, for an optimized executable graph obtained by merging one or more nodes from multiple executable graphs, the node shape has changed. As the expansion width becomes larger, the basic code unit can make full use of the ZMM registers to reduce instruction dependencies and improve parallelism, as shown in FIG. Figure 6 as shown in part (b).

[0041] For example, for a convolution node on a CPU using the VNNI instruction set, three element registers can be used. Specifically, one register is used for input data, one register is used for weight data, and one register is used for output data. When weight data is shared, the estimated number of basic code units that can be executed in parallel using available registers (hereinafter referred to as the estimated number of basic code units) is (32-1) / 2. For a quantization node, five basic registers can be used, one for the minimum value, one for the maximum value, one for the scale, one for the bias, and one for the input data. In addition to the input data, other data involved at the quantization node (e.g., minimum value, maximum value, scale, and bias) are shared by the same channel, so the estimated number of basic code units is (32-4) / 1. For a MatMul node using an AMX instruction, one register is used for input data A, one register is used for input data B, and one register is used for output data. If input data B is constant data, the estimated number of basic code units is (8-1) / 2. If the input data B is variable, the estimated number of basic code units is 8 / 3.

[0042] According to some embodiments, a register utilization check may be performed on an execution node to check whether the register utilization of a basic code unit on the execution node is less than the expected register utilization. If the register utilization of a basic code unit on the execution node is less than the expected register utilization, it may be determined that the execution node is eligible for merge optimization to improve register utilization. An example condition for a register utilization check may be represented by the following formula. In the formula, the value on the left represents the value of the expansion width on a key dimension of the input data or output data at the execution node (e.g., the value of the input channel, output channel, output width, or output height of the input data or output data), and the value on the right represents the corresponding estimated number of basic code units as described above.

[0043]

[0044] In fact, this is only a rough estimate of resource allocation. The actual implementation of the convolution operation on the CPU in the OIhw4i16O4i format may not be accurate, but the basic inspection can help find potential opportunities for improving register utilization. Even if there is a small deviation, it is not important because later rounds will verify the actual improvement of this optimization.

[0045] After completing the above-described instruction utilization check and register utilization check, nodes eligible for optimization have been determined, and the executable graph can be optimized by merging these nodes from multiple identical executable graphs. Figure 7 An example executable graph optimization process implemented by memory management and executable graph modification according to some embodiments of the present disclosure is shown. Figure 7 In the example executable graph optimization process in FIG. 1 , it is assumed that nodes A and C are temporarily ineligible for optimization, and node B is eligible for optimization. The input memory and output memory of the new shape of the optimized node B in the new executable graph may be the same multiple of the input memory and output memory of the shape of the original node B. Figure 7 As shown, in order not to affect other nodes, some new "non-execution" layers that do not cost calculation are inserted into the new executable graph before / after the optimized nodes.

[0046] According to some embodiments, these "non-execution" layers may be generated by a memory manager for memory management and will not be executed during runtime. For example, when node A and node C remain separate, a node Concat is added to merge the output memories of multiple nodes A to align with the input memory of the optimized node B. Because the memory manager will allocate physically contiguous memory according to the size of the input memory of node B, and each node A only needs the starting address of the input memory of node B to output data to node B, the Concat node does not need to be executed during runtime. Similarly, a node Slice is added to slice the output memory of the optimized node B to align with the input memories of multiple nodes C, and the Slice node does not need to be executed during runtime. When the optimized executable graph is sent to the inference device to perform inference, all of these "non-execution" nodes or layers will be removed because the primitive kernel only needs the input buffer and output buffer for calculation, and these auxiliary layers are useless during runtime.

[0047] As described above, the executable graph can be optimized by merging nodes that are determined to be eligible for optimization based on instruction utilization checks and register utilization checks, so as to maximize instruction utilization and register utilization of AI model inference on the inference device. Figure 4As shown, the optimization round may include an affinity check, which may be configured to check whether the affinity mode of the node should be modified from the frame affinity mode (ie, the default affinity mode) to the layer affinity mode to reduce the cache miss rate.

[0048] Specifically, since most primitive kernels are implemented from the perspective of input data, if the size of the input data is not large enough or the size of constant data such as weights, scales, or biases is too large, the cache miss introduced by the load or store operation will play a more important role than the calculation in the inference performance. Execution nodes involving operations that utilize constant data (e.g., MatMul and convolution, quantization, dequantization, etc. that utilize small-sized input data) may all face this problem. Based on instruction utilization checks and register utilization checks, it may be determined that some nodes are not suitable for merging, but by modifying the affinity mode of these nodes to the layer affinity mode, these nodes may be eligible for optimization to reduce the cache miss rate.

[0049] For normal situations, the frame affinity mode can be better than the layer affinity mode because the output buffer of the execution node can be used directly by the next execution node. However, for some execution nodes involving constant data, since the constant data is the same for each input frame, the layer affinity mode of the execution nodes involving constant data may be more efficient than the default frame affinity mode due to the low cache miss rate for loading constant data. Therefore, after the above-mentioned instruction utilization check and register utilization check and memory management, an affinity check can be performed on the optimized executable graph to determine a better affinity mode for a specific execution node. Therefore, the optimized executable graph can be further optimized by modifying the affinity mode of these execution nodes from the frame affinity mode to the layer affinity mode so as to reduce the cache miss rate, thereby improving the overall inference throughput.

[0050] For example, the condition for determining a node eligible for optimization by modifying the affinity mode of the node can be expressed as follows: That is, if the size of the constant data involved at the node is greater than a specified multiple γ of the size of the input size, the node can be eligible for optimization by modifying the affinity mode of the node.

[0051] Const size =γ*Input_size

[0052] According to some embodiments of the present disclosure, it is proposed to set the affinity mode of the nodes in the executable graph that meet the above conditions to the layer affinity mode. If the overall performance is reduced, the affinity mode of the node can be reset to the default frame affinity mode. Therefore, according to the affinity mode setting and all the above-mentioned merge optimizations, the shape of the executable graph can be changed, and the memory used for the nodes in the executable graph can be reallocated.

[0053] Figure 8 An example optimized executable graph for AI model reasoning according to some embodiments of the present disclosure is shown. The example optimized executable graph can be obtained after performing all the proposed optimization rounds on the executable graph output from a common executable graph optimizer. For example, Figure 8 As shown, nodes A, B, C and nodes F, G remain in frame affinity mode. All original nodes D from the same executable graphs of number N are merged to generate optimized nodes D, and new continuous memory is allocated for the input and output of the optimized nodes D. Unlike the single optimized node D, more than one optimized node E is generated according to the characteristics of the primitive implementation kernel used to execute node E. That is, for nodes eligible for merging, nodes from each of the same executable graphs of the specified number of same executable graphs can be merged to generate optimized nodes, or nodes from each of the executable graphs in a subset of the same executable graphs of the specified number of same executable graphs can be merged to generate optimized nodes. Node H is optimized in almost the same way as node E, but the affinity mode of node H is configured as a layer affinity node, so node H on the right depends on node H on the left in the network topology, although there is actually no dependency, just to ensure that node H is executed in layer affinity mode. Similar to nodes A, B, C, F, and G, node I is not suitable for merge optimization, but a layer affinity mode can be set for node I to reduce the cache miss rate, thereby achieving better overall inference throughput.

[0054] In order to better understand the overall idea of ​​optimizing the executable graph for AI model reasoning proposed in this disclosure, reference will be made to Fig. 9 The flowchart shown further describes the proposed executable graph optimization process.

[0055] Fig. 9 An example flow chart of an executable graph optimization process according to some embodiments of the present disclosure is shown. The executable graph optimization process may be implemented by a processor circuit system and may include operations 910 to 930.

[0056] At operation 910, the processor circuit system may copy the executable graph to generate M identical executable graphs. Here, M may be an integer in the range of 2 to the maximum number of executable graphs allowed, N, and N may be an integer estimated or manually configured based on the memory size of the inference device and the size of the executable graph.

[0057] At operation 920 , the processor circuitry may determine one or more nodes from the executable graph that are eligible for optimization based on an inference throughput-related parameter associated with an inference device used to perform AI model inference.

[0058] According to some embodiments, the inference throughput related parameters may include at least one of: instruction utilization, register utilization, and cache miss rate.

[0059] At operation 930 , the processor circuit system may generate an optimized executable graph for AI model reasoning by optimizing one or more nodes from each of the M number of identical executable graphs.

[0060] According to some embodiments, a processor circuit system may generate an optimized executable graph by performing the following operations: for a node from one or more nodes determined to be eligible for optimization based on instruction utilization or register utilization, merging nodes from each of M identical executable graphs to generate an optimized node.

[0061] According to some embodiments, a processor circuit system may generate an optimized executable graph by performing the following operations: for a node from one or more nodes determined to be eligible for optimization based on instruction utilization or register utilization, merging nodes from each executable graph in a subset of M identical executable graphs to generate an optimized node.

[0062] According to some embodiments, a processor circuit system may generate an optimized executable graph by performing the following operations: for a node among one or more nodes determined to be eligible for optimization based on a cache miss rate, modifying the affinity mode of the node from a frame affinity mode to a layer affinity mode to generate an optimized node for reducing the cache miss rate.

[0063] According to some embodiments, the processor circuit system may perform the following operations: execute a number N-1 iterations by increasing the number M from 2 to N to generate a number N-1 optimized executable graphs, each iteration including: copying the executable graph, determining one or more nodes eligible for optimization, and generating an optimized executable graph; and selecting the best optimized executable graph with the highest inference throughput from the number N-1 optimized executable graphs as the optimized executable graph.

[0064] According to some embodiments, the processor circuit system may perform the following operations: determine whether the improvement in the inference throughput associated with the optimized executable graph compared to the reference inference throughput is greater than a threshold; and when it is determined that the improvement in the inference throughput associated with the optimized executable graph is greater than the threshold, send the optimized executable graph to the inference device.

[0065] According to some embodiments, the processor circuit system may perform the following operations: determining a number N' of identical executable graphs, the optimized executable graphs being generated based on a number N' of identical executable graphs, N' being an integer in the range of 2 to N; and calculating the inference throughput of AI model inference using the executable graphs in a batch mode with a batch size of N' as a reference inference throughput.

[0066] According to some embodiments, the processor circuit system may perform the following operations: insert one or more nodes for performing memory management associated with the optimized node before or after the optimized node; and mark the one or more nodes for performing memory management as non-execution nodes, which are to be removed from the optimized executable graph during runtime of the optimized executable graph on the inference device.

[0067] Fig.10 is a block diagram illustrating components according to some example embodiments that are capable of reading instructions from a machine-readable medium or a computer-readable medium (e.g., a non-transitory machine-readable storage medium) and performing any one or more of the methods discussed herein. Specifically, Fig.10 A diagrammatic representation of hardware resources 1000 is shown including one or more processors (or processor cores) 1010, one or more memory / storage devices 1020, and one or more communication resources 1030, each of which may be communicatively coupled via a bus 1040. For embodiments in which node virtualization (e.g., NFV) is utilized, a hypervisor 1002 may be executed to provide an execution environment for one or more network slices / sub-slices to utilize the hardware resources 1000.

[0068] Processor 1010 may include, for example, processor 1012 and processor 1014, which may be, for example, a central processing unit (CPU), a graphics processing unit (GPU), a tensor processing unit (TPU), a vision processing unit (VPU), a field programmable gate array (FPGA), or any suitable combination thereof.

[0069] The memory / storage device 1020 may include a main memory, a disk storage device, or any suitable combination thereof. The memory / storage device 1020 may include, but is not limited to, any type of volatile or non-volatile memory, such as a dynamic random access memory (DRAM), a static random access memory (SRAM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a flash memory, a solid-state storage device, etc.

[0070] The communication resources 1030 may include an interconnect or network interface component or other suitable device to communicate with one or more peripheral devices 1004 or one or more databases 1006 via the network 1008. For example, the communication resources 1030 may include a wired communication component (e.g., for coupling via a universal serial bus (USB)), a cellular communication component, an NFC component, Components (e.g. Low power consumption), components and other communication components.

[0071] The instructions 1050 may include software, programs, applications, applet, apps, or other executable code for causing at least any one of the processors 1010 to perform any one or more of the methods discussed herein. The instructions 1050 may reside in whole or in part within the processor 1010 (e.g., within a cache memory of the processor), at least one of the memory / storage device 1020, or any suitable combination thereof. In addition, any portion of the instructions 1050 may be transmitted to the hardware resources 1000 from any combination of the peripheral device 1004 or the database 1006. Therefore, the memory of the processor 1010, the memory / storage device 1020, the peripheral device 1004, and the database 1006 are examples of computer-readable media and machine-readable media.

[0072] Fig.11 1 is a block diagram of an example processor platform according to some embodiments of the present disclosure. The processor platform 1100 may be, for example, a server, a personal computer, a workstation, a self-learning machine (e.g., a neural network), a mobile device (e.g., a cellular phone, a smart phone, a tablet computer (e.g., an iPad TM ), a personal digital assistant (PDA), an Internet appliance, a DVD player, a CD player, a digital video recorder, a Blu-ray player, a game console, a personal video recorder, a set-top box, headphones or other wearable device, or any other type of computing device.

[0073] The processor platform 1100 of the illustrated example includes a processor 1112. The processor 1112 of the illustrated example is hardware. For example, the processor 1112 can be implemented by one or more integrated circuits, logic circuits, microprocessors, GPUs, DSPs, or controllers from any desired series or manufacturer. The hardware processor can be a semiconductor-based (e.g., silicon-based) device. In some embodiments, the processor implements one or more of the above methods or processes.

[0074] The processor 1112 of the illustrated example includes a local memory 1113 (e.g., a cache). The processor 1112 of the illustrated example communicates with a main memory including a volatile memory 1114 and a non-volatile memory 1116 via a bus 1118. The volatile memory 1114 may be comprised of a synchronous dynamic random access memory (SDRAM), a dynamic random access memory (DRAM), Dynamic Random Access Memory The non-volatile memory 1116 may be implemented by flash memory and / or any other desired type of memory device. Access to the main memory 1114, 1116 is controlled by a memory controller.

[0075] The processor platform 1100 of the illustrated example also includes an interface circuit system 1120. The interface circuit system 1120 may be implemented by any type of interface standard (e.g., Ethernet interface, Universal Serial Bus (USB), interface, near field communication (NFC) interface and / or PCI Express interface).

[0076] In the example shown, one or more input devices 1122 are connected to the interface circuitry 1120. The input devices 1122 allow a user to enter data and / or commands into the processor 1112. The input devices may be implemented by, for example, an audio sensor, a microphone, a camera (still or video), a keyboard, buttons, a mouse, a touch screen, a trackpad, a trackball, and / or a voice recognition system.

[0077] One or more output devices 1124 are also connected to the interface circuit system 1120 of the illustrated example. The output device 1124 can be implemented, for example, by a display device (e.g., a light emitting diode (LED), an organic light emitting diode (OLED), a liquid crystal display (LCD), a cathode ray tube display (CRT), an in-place switching (IPS) display, a touch screen, etc.), a tactile output device, a printer, and / or a speaker. Therefore, the interface circuit system 1120 of the illustrated example typically includes a graphics driver card, a graphics driver chip, and / or a graphics driver processor.

[0078] The interface circuitry 1120 of the illustrated example also includes communication devices (e.g., transmitters, receivers, transceivers, modems, residential gateways, wireless access points, and / or network interfaces) to facilitate data exchange with external machines (e.g., any kind of computing device) via the network 1126. Communications may be via, for example, an Ethernet connection, a digital subscriber line (DSL) connection, a telephone line connection, a coaxial cable system, a satellite system, a line-of-site wireless system, a cellular telephone system, etc.

[0079] For example, interface circuitry 1120 may include a training data set input via input device 1122 or retrieved from network 1126 .

[0080] The processor platform 1100 of the illustrated example also includes one or more mass storage devices 1128 for storing software and / or data. Examples of such mass storage devices 1128 include floppy disk drives, hard disk drives, compact disk drives, Blu-ray disk drives, redundant array of independent disks (RAID) systems, and digital versatile disk (DVD) drives.

[0081] The machine-executable instructions 1132 may be stored in the mass storage device 1128, in the volatile memory 1114, in the non-volatile memory 1116, and / or on a removable, non-transitory computer-readable storage medium such as a CD or DVD.

[0082] Additional notes and examples:

[0083] Example 1 includes an apparatus for optimizing an executable graph for artificial intelligence (AI) model reasoning, comprising: an interface circuit system; and a processor circuit system coupled to the interface circuit system and configured to: copy the executable graph received via the interface circuit system to generate M identical executable graphs; determine one or more nodes eligible for optimization from the executable graph based on an inference throughput-related parameter associated with an inference device used to perform the AI ​​model reasoning; and generate an optimized executable graph for the AI ​​model reasoning by optimizing the one or more nodes from each of the M identical executable graphs, wherein M is an integer in the range of 2 to a maximum number N of executable graphs allowed, and N is an integer estimated or manually configured based on a memory size of the inference device and a size of the executable graph.

[0084] Example 2 includes the apparatus of Example 1, wherein the processor circuit system is further configured to: execute a number N-1 iterations by increasing the number M from 2 to N to generate a number N-1 optimized executable graphs, each iteration comprising: copying the executable graph, determining the one or more nodes eligible for optimization, and generating the optimized executable graph; and selecting the best optimized executable graph with the highest inference throughput from the number N-1 optimized executable graphs as the optimized executable graph.

[0085] Example 3 includes the apparatus of Example 1 or 2, wherein the processor circuit system is further configured to: determine whether the improvement in the inference throughput associated with the optimized executable graph compared to a reference inference throughput is greater than a threshold; and when it is determined that the improvement in the inference throughput associated with the optimized executable graph is greater than the threshold, provide the optimized executable graph to the interface circuit system for transmission to the inference device.

[0086] Example 4 includes the apparatus of Example 3, wherein the processor circuit system is further configured to: determine a number N' of identical executable graphs, the optimized executable graphs being generated based on the number N' of identical executable graphs, N' being an integer in the range of 2 to N; and calculate the inference throughput of the AI ​​model inference using the executable graphs in a batch mode with a batch size of N' as the reference inference throughput.

[0087] Example 5 includes the apparatus of any one of Examples 1 to 4, wherein the inference throughput-related parameter comprises at least one of: instruction utilization, register utilization, and cache miss rate.

[0088] Example 6 includes the apparatus of Example 5, wherein the processor circuit system is configured to generate the optimized executable graph for the AI ​​model reasoning by performing the following operations: for a node among the one or more nodes determined to be eligible for optimization based on the instruction utilization or the register utilization, merging the nodes from each of the M number of identical executable graphs to generate an optimized node.

[0089] Example 7 includes the apparatus of Example 5, wherein the processor circuit system is configured to generate the optimized executable graph for the AI ​​model reasoning by: for a node among the one or more nodes determined to be eligible for optimization based on the instruction utilization or the register utilization, merging the nodes from each executable graph in the subset of the M number of identical executable graphs to generate an optimized node.

[0090] Example 8 includes the apparatus of Example 5, wherein the processor circuit system is configured to generate the optimized executable graph for the AI ​​model reasoning by performing the following operations: for a node among the one or more nodes determined to be eligible for optimization based on the cache miss rate, modifying the affinity mode of the node from a frame affinity mode to a layer affinity mode to generate an optimized node for reducing the cache miss rate.

[0091] Example 9 includes the apparatus of Example 6 or 7, wherein the processor circuit system is further configured to: insert one or more nodes for performing memory management associated with the optimized nodes; and mark the one or more nodes for performing memory management as non-execution nodes, and the non-execution nodes are to be removed from the optimized executable graph during runtime of the optimized executable graph on the inference device.

[0092] Example 10 includes a method for optimizing an executable graph for artificial intelligence (AI) model reasoning, comprising: copying the executable graph to generate M identical executable graphs; determining one or more nodes eligible for optimization from the executable graph based on an inference throughput-related parameter associated with an inference device used to perform the AI ​​model reasoning; and generating an optimized executable graph for the AI ​​model reasoning by optimizing the one or more nodes from each of the M identical executable graphs, wherein M is an integer in the range of 2 to a maximum number N of executable graphs allowed, and N is an integer estimated or manually configured based on a memory size of the inference device and a size of the executable graph.

[0093] Example 11 includes the method of Example 10, further including: executing a number N-1 iterations by increasing the number M from 2 to N to generate a number N-1 optimized executable graphs, each iteration including: copying the executable graph, determining the one or more nodes eligible for optimization, and generating the optimized executable graph; and selecting the best optimized executable graph with the highest inference throughput from the number N-1 optimized executable graphs as the optimized executable graph.

[0094] Example 12 includes the apparatus of Example 10 or 11, further comprising: determining whether an improvement in the inference throughput associated with the optimized executable graph compared to a reference inference throughput is greater than a threshold; and when it is determined that the improvement in the inference throughput associated with the optimized executable graph is greater than the threshold, sending the optimized executable graph to the inference device.

[0095] Example 13 includes the apparatus of Example 12, further comprising: determining a number N' of identical executable graphs, the optimized executable graphs being generated based on the number N' of identical executable graphs, N' being an integer in the range of 2 to N; and calculating the inference throughput of the AI ​​model inference using the executable graphs in a batch mode with a batch size of N' as the reference inference throughput.

[0096] Example 14 includes the method of any one of Examples 10 to 13, wherein the inference throughput-related parameter comprises at least one of: instruction utilization, register utilization, and cache miss rate.

[0097] Example 15 includes the method of Example 14, wherein generating the optimized executable graph for reasoning of the AI ​​model includes: for nodes among the one or more nodes determined to be eligible for optimization based on the instruction utilization or the register utilization, merging the nodes from each of the M number of identical executable graphs to generate optimized nodes.

[0098] Example 16 includes the method of Example 14, wherein generating the optimized executable graph for reasoning of the AI ​​model includes: for nodes among the one or more nodes determined to be eligible for optimization based on the instruction utilization or the register utilization, merging the nodes from each executable graph in the subset of the same number M of executable graphs to generate optimized nodes.

[0099] Example 17 includes the method of Example 14, wherein generating the optimized executable graph for reasoning of the AI ​​model includes: for a node among the one or more nodes determined to be eligible for optimization based on the cache miss rate, modifying the affinity mode of the node from a frame affinity mode to a layer affinity mode to generate an optimized node for reducing the cache miss rate.

[0100] Example 18 includes the method of Example 15 or 16, further comprising: inserting one or more nodes for performing memory management associated with the optimized nodes; and marking the one or more nodes for performing memory management as non-execution nodes, wherein the non-execution nodes are to be removed from the optimized executable graph during runtime of the optimized executable graph on the inference device.

[0101] Example 19 includes a computer-readable medium having instructions stored thereon, wherein the instructions, when executed by a processor circuit system, cause the processor circuit system to perform a method according to any one of Examples 10 to 18.

[0102] Example 20 includes an apparatus comprising a unit for performing the method according to any one of Examples 10 to 18.

[0103] Various technologies or specific aspects or parts thereof may take the form of program codes (i.e., instructions) embodied in tangible media (e.g., floppy disks, CD-ROMs, hard disk drives, non-transitory computer-readable storage media, or any other machine-readable storage media), wherein when the program code is loaded into a machine (e.g., a computer) and executed by the machine, the machine becomes a device for practicing various technologies. Non-transitory computer-readable storage media may be computer-readable storage media that do not include signals. In the case of executing program codes on a programmable computer, a computing system may include a processor, a processor-readable storage medium (including volatile and non-volatile memory and / or storage elements), at least one input device, and at least one output device. Volatile and non-volatile memory and / or storage elements may be RAM, EPROM, flash drives, optical drives, magnetic hard disk drives, solid-state drives, or other media for storing electronic data. One or more programs that may implement or utilize the various technologies described herein may use application programming interfaces (APIs), reusable controls, etc. Such programs may be implemented in high-level procedural or object-oriented programming languages ​​to communicate with a computer system. However, if necessary, the program may be implemented in assembly language or machine language. In any case, the language may be a compiled language or an interpreted language, and combined with a hardware implementation. Exemplary systems or devices may include, but are not limited to, laptop computers, tablet computers, desktop computers, smart phones, computer terminals and servers, storage databases, and other electronic devices utilizing circuit systems and programmable memory, such as household appliances, smart televisions, digital video disk (DVD) players, heating, ventilation and air conditioning (HVAC) controllers, light switches, etc.

[0104] The above detailed description includes reference to the accompanying drawings, which form a part of the detailed description. The accompanying drawings illustrate specific embodiments that can be put into practice by way of illustration. These embodiments are also referred to as "examples" in this article. Such examples may include elements other than those shown or described. However, the inventors also contemplate examples in which only those elements shown or described are provided. In addition, the inventors also contemplate examples using any combination or arrangement of those elements (or one or more aspects thereof) shown or described with respect to a specific example (or one or more aspects thereof) or with respect to other examples (or one or more aspects thereof) shown or described herein.

[0105] All publications, patents, and patent documents cited in this document are incorporated herein by reference in their entirety, as if individually incorporated by reference. In the event of inconsistencies in usage between this document and those documents incorporated by reference, the usage in the incorporated references should be considered supplementary to that in this document; for irreconcilable inconsistencies, the usage in this document controls.

[0106] In this document, as is common in patent documents, the terms "a" or "an" are used to include one or more than one, regardless of any other instances or uses of "at least one" or "one or more." In this document, unless otherwise noted, the term "or" is used to refer to a non-exclusive or, such that "A or B" includes "A but not B," "B but not A," and "A and B." In the appended claims, the terms "including" and "in which" are used as the plain English equivalents of the respective terms "comprising" and "wherein." Moreover, in the appended claims, the terms "including" and "comprising" are open-ended, that is, systems, devices, articles, or processes that include elements in a claim in addition to those listed after such a term are still considered to fall within the scope of the claim. Furthermore, in the appended claims, the terms "first," "second," and "third," etc. are used merely as labels and are not intended to impose numerical requirements on their objects.

[0107] The above description is intended to be illustrative and not restrictive. For example, the above examples (or one or more aspects thereof) may be used in combination with each other. For example, other embodiments may be used by a person of ordinary skill in the art when reading the above description. The abstract is to allow the reader to quickly determine the nature of the technical disclosure and is submitted with the understanding that it will not be used to interpret or limit the scope or meaning of the claims. Moreover, in the above detailed description, various features may be grouped together to simplify the disclosure. This should not be interpreted as an intention that the disclosed features that are not claimed for protection are necessary for any claim. Instead, the subject matter of the invention may be less than all the features of a particular disclosed embodiment. Therefore, the following claims are incorporated herein into the detailed description, with each claim itself as a separate embodiment. The scope of the embodiment should be determined with reference to the appended claims and the full scope of equivalents given by these claims.

Claims

1. A device for optimizing an executable graph for artificial intelligence (AI) model reasoning, comprising: Interface circuit system; as well as processor circuitry coupled to the interface circuitry and configured to: duplicating the executable graph received via the interface circuitry to generate a number M of identical executable graphs; Determining one or more nodes from the executable graph that are eligible for optimization based on an inference throughput-related parameter associated with an inference device used to perform inference of the AI ​​model; as well as Generating an optimized executable graph for the AI ​​model reasoning by optimizing the one or more nodes from each of the M number of identical executable graphs, Wherein, M is an integer in the range of 2 to the maximum number of executable graphs allowed, N, and N is an integer estimated or manually configured based on the memory size of the inference device and the size of the executable graph.

2. The device according to claim 1, wherein: The processor circuitry is further configured to: performing N-1 number of iterations by increasing the number M from 2 to N to generate N-1 number of optimized executable graphs, each iteration comprising: copying the executable graph, determining the one or more nodes eligible for optimization, and generating the optimized executable graph; and A best optimized executable graph with the highest inference throughput is selected from the N-1 optimized executable graphs as the optimized executable graph.

3. The device according to claim 1 or 2, wherein: The processor circuitry is further configured to: determining whether an improvement in inference throughput associated with the optimized executable graph compared to a reference inference throughput is greater than a threshold; as well as When it is determined that the improvement in the inference throughput associated with the optimized executable graph is greater than the threshold, the optimized executable graph is provided to the interface circuitry for transmission to the inference device.

4. The device according to claim 3, wherein: The processor circuitry is further configured to: Determining a number N' of identical executable graphs, the optimized executable graph being generated based on the number N' of identical executable graphs, N' being an integer in the range of 2 to N; as well as The inference throughput of the AI ​​model inference using the executable graph in a batch mode with a batch size of N′ is calculated as the reference inference throughput.

5. The device according to claim 1 or 2, wherein: The inference throughput related parameters include at least one of: instruction utilization, register utilization, and cache miss rate.

6. The device according to claim 5, wherein: The processor circuit system is configured to generate the optimized executable graph for the AI ​​model reasoning by: For a node among the one or more nodes determined to be eligible for optimization based on the instruction utilization or the register utilization, the node from each of the M number of identical executable graphs is merged to generate an optimized node.

7. The device according to claim 5, wherein: The processor circuit system is configured to generate the optimized executable graph for the AI ​​model reasoning by: For a node of the one or more nodes determined to be eligible for optimization based on the instruction utilization or the register utilization, the node from each executable graph in the subset of the number M of identical executable graphs is merged to generate an optimized node.

8. The device according to claim 5, wherein: The processor circuit system is configured to generate the optimized executable graph for the AI ​​model reasoning by: For a node among the one or more nodes determined to be eligible for optimization based on the cache miss rate, an affinity mode of the node is modified from a frame affinity mode to a layer affinity mode to generate an optimized node for reducing the cache miss rate.

9. The device according to claim 6, wherein: The processor circuitry is further configured to: inserting one or more nodes for performing memory management associated with the optimized nodes; and The one or more nodes for performing memory management are marked as non-executable nodes to be removed from the optimized executable graph during runtime of the optimized executable graph on the inference device.

10. A method for optimizing an executable graph for artificial intelligence (AI) model reasoning, comprising: Copying the executable graph to generate M identical executable graphs; Determining one or more nodes from the executable graph that are eligible for optimization based on an inference throughput-related parameter associated with an inference device used to perform inference of the AI ​​model; as well as Generating an optimized executable graph for the AI ​​model reasoning by optimizing the one or more nodes from each of the M number of identical executable graphs, Wherein, M is an integer in the range of 2 to the maximum number of executable graphs allowed, N, and N is an integer estimated or manually configured based on the memory size of the inference device and the size of the executable graph.

11. The method according to claim 10, further comprising: performing N-1 number of iterations by increasing the number M from 2 to N to generate N-1 number of optimized executable graphs, each iteration comprising: copying the executable graph, determining the one or more nodes eligible for optimization, and generating the optimized executable graph; and A best optimized executable graph with the highest inference throughput is selected from the N-1 optimized executable graphs as the optimized executable graph.

12. The device according to claim 10 or 11, further comprising: determining whether an improvement in inference throughput associated with the optimized executable graph compared to a reference inference throughput is greater than a threshold; as well as When it is determined that the improvement in the inference throughput associated with the optimized executable graph is greater than the threshold, the optimized executable graph is sent to the inference device.

13. The apparatus according to claim 12, further comprising: Determining a number N' of identical executable graphs, the optimized executable graph being generated based on the number N' of identical executable graphs, N' being an integer in the range of 2 to N; as well as The inference throughput of the AI ​​model inference using the executable graph in a batch mode with a batch size of N′ is calculated as the reference inference throughput.

14. The method according to claim 10 or 11, wherein: The inference throughput related parameters include at least one of: instruction utilization, register utilization, and cache miss rate.

15. The method according to claim 14, wherein: Generating the optimized executable graph for the AI ​​model reasoning includes: For a node among the one or more nodes determined to be eligible for optimization based on the instruction utilization or the register utilization, the node from each of the M number of identical executable graphs is merged to generate an optimized node.

16. The method according to claim 14, wherein: Generating the optimized executable graph for the AI ​​model reasoning includes: For a node of the one or more nodes determined to be eligible for optimization based on the instruction utilization or the register utilization, the node from each executable graph in the subset of the number M of identical executable graphs is merged to generate an optimized node.

17. The method according to claim 14, wherein: Generating the optimized executable graph for the AI ​​model reasoning includes: For a node among the one or more nodes determined to be eligible for optimization based on the cache miss rate, an affinity mode of the node is modified from a frame affinity mode to a layer affinity mode to generate an optimized node for reducing the cache miss rate.

18. The method according to claim 15, further comprising: inserting one or more nodes for performing memory management associated with the optimized nodes; as well as The one or more nodes for performing memory management are marked as non-executable nodes to be removed from the optimized executable graph during runtime of the optimized executable graph on the inference device.

19. A computer readable medium having instructions stored thereon, wherein: The instructions, when executed by a processor circuit system, cause the processor circuit system to perform a method according to any one of claims 10 to 18.

20. An apparatus comprising means for performing the method according to any one of claims 10 to 18.

Images