A numa-aware dnn compilation system optimization method and system

Abstract

The application discloses a NUMA-aware DNN compiling system optimization method and system, the method comprises the following steps: initializing thread-node mapping; initializing available memory; obtaining a calculation node in a calculation graph of a DNN, constructing a parameter node for each calculation node, preallocating model parameter memory, and updating the available memory; allocating priority to each calculation node; constructing a copy parameter node for the calculation node according to the allocated priority and updating the available memory until the available memory is exhausted; traversing the calculation node to allocate the memory of the copy parameter node to a NUMA node to minimize the maximum distance to data among all threads; and performing operator optimization on each calculation node corresponding operator until the optimization target is completed. The application aims to evenly allocate the memory of a neural network inference job based on the characteristics of data involved in the neural network and the memory access characteristics of a NUMA architecture server, and improve the local memory access rate.

Description

Technical Field

[0001] This invention relates to DNN compiler system optimization technology in the field of operating systems, specifically to a NUMA-aware DNN compiler system optimization method and system. Background Technology

[0002] A neural network (NN) is a computational model that simulates the human nervous system, consisting of multiple neurons connected in a hierarchical structure. A typical neural network consists of an input layer, hidden layers, and an output layer. Each neuron is connected to neurons in the previous layer or to the input data, and information is transmitted and computed through weights and biases. The memory space allocated during the execution of a neural network inference task is primarily used for two types of data (Tensors): neural network parameters and intermediate results. Specifically, each layer of a neural network typically accesses three memory locations: input, output, and neural network parameter memory, where the input and output represent intermediate result data.

[0003] Current DNN inference frameworks, due to the need to deploy static neural networks on specific devices, mostly require graph optimization and operator optimization of the neural network. Simplifying the computation graph through equivalent transformations and improving data reuse in memory close to the processor through loop optimization of the constituent operators are both aimed at reducing the overhead of data transfer between memory and cache, thus improving data locality. Although cache reuse has been sufficiently optimized, the overhead of accessing main memory is still not negligible. To improve main memory access bandwidth, NUMA (Non-Uniform Memory Access) architecture is commonly used. This is a multiprocessor computer system architecture where main memory is divided into multiple nodes, with each processor directly connected to one of these nodes, and accessing memory on the local node is faster than accessing memory on non-local nodes. This architecture can improve the memory access performance and scalability of multiprocessor systems, but it also introduces memory access locality and balance problems. Specifically, the locality problem refers to the slowdown when a processor needs to access remote memory; while the balance problem refers to the slowdown when multiple processors access the same memory simultaneously due to insufficient bandwidth.

[0004] Currently, parallel computing in neural networks is generally based on the fork-join model, where memory initialization and access occur on different threads. According to the default memory allocation strategy of operating systems (such as Linux), physical memory is allocated to the NUMA node where the thread that first triggers access resides. As a result, neural network jobs typically allocate memory to the node where the main thread resides. When performing parallel computing on a NUMA architecture server, the memory access pressure is concentrated on a single memory controller, and threads not on the same node as the main thread need to perform remote access. This not only fails to leverage the advantages of the NUMA architecture but can also lead to performance degradation.

[0005] The performance of DNN programs running on NUMA architectures is significantly affected by memory allocation strategies. Several solutions exist: 1. Static memory allocation strategies provided by the operating system, such as FirstTouch, are generally used as the default. As mentioned earlier, in the fork-join model, balance and locality are poor. Interleave allocation strategy: This strategy allocates data blocks across different memory nodes in an interleaved manner. This strategy can evenly distribute memory load and improve the parallelism of memory access, but it does not improve memory access locality, and the performance optimization effect is unstable and may even decrease. 2. Operating system-level technologies such as NUMA Balancing target the overall performance improvement of the operating system, and do not significantly improve the execution efficiency of individual applications. Furthermore, dynamic memory allocation strategies are often accompanied by historical memory access behavior statistics or thread / data migration operations, which incur additional performance overhead. The two types of data required for neural network inference jobs—model parameters (accessed by all threads, regardless of which thread they are migrated to, causing non-local access by other threads) and intermediate results (with short memory lifecycles, migration may not necessarily yield positive benefits)—are also problematic. In comparison, the data involved in neural network inference tasks is more suitable for static allocation strategies. 3. Due to the characteristics of the fork-join model, memory allocation and access do not occur simultaneously, which causes some techniques that determine the memory allocation location based on the real-time state of the system to fail. Summary of the Invention

[0006] The technical problem to be solved by this invention is to provide a NUMA-aware DNN compilation system optimization method and system to address the above-mentioned problems in the prior art. This invention aims to improve the local memory access ratio by uniformly allocating memory for neural network inference jobs based on the characteristics of data involved in neural networks and adapting to the memory access characteristics of NUMA architecture servers.

[0007] To solve the above-mentioned technical problems, the technical solution adopted by the present invention is as follows:

[0008] A NUMA-aware DNN compilation system optimization method includes the following steps:

[0009] S1, Initialize the thread-node mapping to map the threads required for DNN inference to all NUMA nodes;

[0010] S2, initialize the available memory Avail based on the current available memory size of the system;

[0011] S3, obtain the computation nodes in the computation graph of the DNN, construct parameter nodes for each computation node to provide the neural network parameters required for computation for that computation node, pre-allocate model parameter memory, and update the available memory Avail;

[0012] S4, the computational complexity of each computing node is used as the allocation priority and sorted according to the allocation priority;

[0013] S5, construct replica parameter nodes for the compute nodes according to the allocation priority, and update the available memory Avail until the available memory Avail is exhausted;

[0014] S6, traverse the compute nodes to allocate memory for the NUMA nodes corresponding to the replica parameter nodes of the compute nodes so that the maximum distance from the data is minimized among all threads;

[0015] S7 performs operator optimization for the operators corresponding to each computing node: including constructing operator optimization templates, parallelizing the outermost loop, allocating parameter nodes as inputs within the loop body, allocating local memory as outputs, and evaluating the optimization effect.

[0016] S8, determine whether the optimization goal has been completed. If it has not been completed, jump to step S7; otherwise, end and exit.

[0017] Optionally, in step S2, when initializing the available memory Avail based on the current available memory size of the system, the available memory Avail is the difference between the current available memory size of the system and the reserved intermediate result memory size, where the reserved intermediate result memory size refers to the memory size required by the intermediate input variables of the DNN.

[0018] Optionally, before obtaining the computational nodes in the computational graph of the DNN in step S3, the method further includes a step of generating the computational graph of the DNN using graph optimization techniques. The graph optimization techniques include some or all of the following: expression simplification, common expression elimination, constant propagation, matrix merging, and operator fusion, so that each operator serves as a computational node in the computational graph.

[0019] Optionally, updating the available memory Avail in step S3 includes: obtaining the construction parameter node P for each compute node i.i,0 The size of the memory pre-allocated model parameters is sizeof(P) i,0 ), the sizeof(P) of the pre-allocated model parameter memory for all computing nodes i. i,0 The total size of the pre-allocated model parameter memory, sizeof(P), is obtained by summing the values. The available memory, Avail, is then subtracted from the total size of sizeof(P) to obtain the updated available memory, Avail.

[0020] Optionally, step S5 includes: allocating priority according to the priority of each computing node i. i The parameter nodes P of each computation node i are ordered from highest to lowest. i,0 Construct replica parameter node P i,j ,in , Let P be the number of NUMA nodes in the system, and let P be the replica parameter node. i,j At that time, if the available memory size Avail is reduced by the replica parameter node P i,j If the memory usage result is less than 0, it is determined that the available memory is exhausted, and the process jumps to step S6; otherwise, the available memory size Avail is reduced by the replica parameter node P. i,j The result of the memory usage is used as the new available memory size Avail, and continues for the next parameter node P. i,0 Construct replica parameter node P i,j Until all parameter nodes P are completed i,0 Construct replica parameter node P i,j Then proceed to step S6.

[0021] Optionally, step S6 includes:

[0022] S6.1, Traverse the computation nodes in the computation graph of the DNN. If the traversal is complete, jump to step S7; otherwise, jump to step S6.2.

[0023] S6.2, determine whether it is the replica parameter node P constructed by the current computing node. i,j The number of replicas, if the replica parameter node P i,j If the quantity is 1, then proceed to step S6.3; if the replica parameter node P i,j The number is greater than 1 and less than the number of NUMA nodes in the system. If so, proceed to step S6.4; if the replica parameter node P i,j The number is equal to the number of NUMA nodes in the system. If so, proceed to step S6.5;

[0024] S6.3, set the replica parameter node P of the current compute node to... i,jThe memory is mapped to all NUMA nodes using a round-robin strategy. Proceed to step S6.1.

[0025] S6.4, transfer all replica parameter nodes P of the current compute node. i,j The memory is evenly distributed among NUMA nodes to obtain a node set NodeSubset. ij Then, set all replica parameter nodes P of the current compute node as replicas. i,j Mapped to the node set NodeSubset using a round-robin strategy ij On the NUMA node, proceed to step S6.1;

[0026] S6.5, transfer all replica parameter nodes P of the current compute node. i,j The memory allocated to the NodeSet, which consists of NUMA nodes in the system. j On the NUMA node, proceed to step S6.1.

[0027] Optionally, step S7 includes:

[0028] S7.1, Traverse the computation nodes in the computation graph of the DNN. If the traversal is complete, jump to step S8; otherwise, jump to step S7.2.

[0029] S7.2, construct a loop optimization template for the current computing node n using one or more of the following methods: loop rearrangement, loop partitioning, and loop expansion. n Among them, loop rearrangement refers to swapping the positions of the inner and outer loops of a computing node, loop splitting refers to splitting the loop of a computing node into multiple loops, and loop unrolling refers to removing the loop and directly writing the operation of each loop.

[0030] S7.3, Optimization of Templates for Loops n The outermost loop is parallelized, and within the loop body, model parameters are obtained from the nearest NUMA node, and the address of the input feature map is set. i,t The nearest replica node P of the NUMA node where the thread resides. i,j The starting address of memory:

[0031] Input i,t =Addr(Nearest(P i,j )),

[0032] Where Nearest is the nearest operation; within the loop, it is calculated based on the loop count Num(Roll). i Calculate the memory size required for intermediate results in each loop iteration and allocate memory accordingly:

[0033] Output i,t =Alloc(sizeof(Output i ) / Num(Roll i )),

[0034] Among them, Output i This represents the total memory required for intermediate results by the current computing node. `Alloc` is the memory allocation operation, and `sizeof(Output)` is the memory allocation value. i ) represents the total memory size required for intermediate results by the current computing node, Num(Roll) i () represents the number of iterations;

[0035] S7.4, evaluate the DNN inference latency of the template. If the DNN inference latency meets the optimization objective, the process ends; otherwise, return to step S7.1.

[0036] Furthermore, the present invention also provides a NUMA-aware DNN compiler optimization system, including an interconnected microprocessor and a memory, wherein the microprocessor is programmed or configured to execute the NUMA-aware DNN compiler optimization method.

[0037] Furthermore, the present invention also provides a computer-readable storage medium storing a computer program or instructions that are programmed or configured to execute the NUMA-aware DNN compiler optimization method by a processor.

[0038] Furthermore, the present invention also provides a computer program product, including a computer program or instructions that are programmed or configured to execute the NUMA-aware DNN compiler optimization method via a processor.

[0039] Compared with existing technologies, this invention mainly achieves the following beneficial effects: To achieve uniform memory allocation for neural network inference jobs and improve the local memory access ratio by adapting to the memory access characteristics of NUMA architecture servers based on the characteristics of data involved in neural networks, this invention focuses on data involved in the parallel domain. This is because data in the serial domain is allocated and accessed by the main thread, and there are no issues of balance or locality of reference. However, in the parallel domain, the memory for input, output, and model parameters is accessed by multiple threads, and the memory allocation strategy for this part has a significant impact on the performance of neural network programs running on NUMA architecture systems. This invention proposes a memory allocation strategy based on the characteristics of data involved in neural networks. In DNN inference jobs, this invention's memory allocation strategy has the following advantages: 1. This invention, by adapting to the memory access characteristics of NUMA architecture servers and targeting the data access characteristics of DNNs, can stably improve the execution efficiency of neural network inference jobs. 2. This invention can improve memory load balancing while heuristically allocating memory to various nodes based on the access relationship between threads and data, thereby improving the overall local memory access ratio (the ratio of the number of times all threads access local node data to the total number of data accesses). Attached Figure Description

[0040] Figure 1 This is a schematic diagram of the basic process of the method in an embodiment of the present invention.

[0041] Figure 2 This is a schematic diagram of the NUMA topology in an embodiment of the present invention.

[0042] Figure 3 This is an example of a computation graph for a DNN in an embodiment of the present invention. Detailed Implementation

[0043] To enable those skilled in the art to better understand the technical solutions of the present invention, the technical solutions of the present invention will be further described in detail below with reference to the accompanying drawings in the embodiments of the present invention.

[0044] This embodiment is implemented on TVM 0.17. The optimized DNN is a convolutional neural network with 25 million parameters, unquantized, with 400MB of available system memory and 100MB reserved for intermediate results. Additionally, the parameters are set as follows: 8 threads, 4 NUMA nodes, and the main thread is assumed to be T0. Figure 1 As shown, the NUMA-aware DNN compilation system optimization method in this embodiment includes the following steps:

[0045] S1, Initialize the thread-node mapping to map the threads required for DNN inference to all NUMA nodes;

[0046] S2, initialize the available memory Avail based on the current available memory size of the system;

[0047] S3, obtain the computation nodes in the computation graph of the DNN, construct parameter nodes for each computation node to provide the neural network parameters required for computation for that computation node, pre-allocate model parameter memory, and update the available memory Avail;

[0048] S4 prioritizes the computational complexity of each node and sorts them accordingly. Computational complexity can be determined by the amount of computation required; all operators in a DNN can be converted to multiply-accumulate operations, so computational complexity can also be understood as the number of multiply-accumulate operations required by the operator. In non-extreme cases, the order is convolution > fully connected > batch normalization > pooling > activation. For operators of the same type, the complexity can be simply compared using the number of parameters.

[0049] S5, construct replica parameter nodes for the compute nodes according to the allocation priority, and update the available memory Avail until the available memory Avail is exhausted;

[0050] S6, traverse the compute nodes to allocate memory for the NUMA nodes corresponding to the replica parameter nodes of the compute nodes so that the maximum distance from the data is minimized among all threads;

[0051] S7 performs operator optimization for the operators corresponding to each computing node: including constructing operator optimization templates, parallelizing the outermost loop, allocating parameter nodes as inputs within the loop body, allocating local memory as outputs, and evaluating the optimization effect.

[0052] S8, determine whether the optimization goal has been completed. If it has not been completed, jump to step S7; otherwise, end and exit.

[0053] In step S1 of this embodiment, when initializing the thread-node mapping to map the threads required for DNN inference to all NUMA nodes, record... The number of NUMA nodes in the system. If the number of threads T_num is less than the number of NUMA nodes. ,but =T_num, and add the NUMA node mapped to the thread to the NUMA node set NodeSet that participates in parallel computing. Figure 2 This is a schematic diagram of the NUMA topology in this embodiment. The NUMA nodes include nodes 0 (NUMA0) to 3 (NUMA3), where C1 and C2 represent CPU processor cores, L3 Cache represents the level 3 cache, and each NUMA node includes an independent memory node. For example, the memory node corresponding to node 0 (NUMA0) is denoted as memory node 0. For example, in this embodiment, the number of parallel computing threads is set to 8, then thread T... 2n ~T 2n+1Mapping to Node n (n=0,1,2,3) =4.

[0054] In step S2 of this embodiment, when initializing the available memory Avail based on the current available memory size of the system, the available memory Avail is the difference between the current available memory size and the reserved intermediate result memory size, where the reserved intermediate result memory size refers to the memory size required by the intermediate input variables of the DNN. For example, in this embodiment, after subtracting the reserved intermediate result memory, the available memory for the model parameters is 400-100=300MB.

[0055] Before obtaining the computational nodes in the computational graph of the DNN in step S3 of this embodiment, the method further includes a step of generating the computational graph of the DNN using graph optimization techniques. The graph optimization techniques include some or all of the following: expression simplification, common expression elimination, constant propagation, matrix merging, and operator fusion, so that each operator serves as a computational node in the computational graph. Figure 3 This is an example of the computation graph of the DNN in this embodiment, where each dashed box is a computation node, and each computation node corresponds to an operator, which is assigned an index i in sequence (due to graph optimization such as operator fusion, this operator can also represent a fusion operator). Figure 3 In this context, Call represents a call, nn.conv2d represents a 2D convolution for DNN inference, nn.bias_add represents a summation of biases for DNN inference, nn.relu represents ReLU activation for DNN inference, nn.max_pool2d represents max pooling for DNN inference, and add represents a summation operation for DNN inference.

[0056] In this embodiment, step S3, updating the available memory Avail, includes: obtaining the construction parameters of each compute node i and node P respectively. i,0 The size of the memory pre-allocated model parameters is sizeof(P) i,0 ), the sizeof(P) of the pre-allocated model parameter memory for all computing nodes i. i,0 The total size of the pre-allocated model parameter memory, sizeof(P), is obtained by summing the values. The available memory, Avail, is then subtracted from the total sizeof(P) to obtain the updated available memory, Avail. Each parameter node represents the neural network parameters required by the corresponding computation node, and its pre-allocated parameter memory is sizeof(P). i,0Finally, the available memory Avail is subtracted from the total size sizeof(P) to obtain the updated available memory Avail, which can be expressed as: Avail = Avail - sizeof(P). For example, in this embodiment, 3) constructs the computation graph and parameter nodes. The neural network has 25 million parameters, stored in fp32 format, occupying 100MB of memory. Then, the available memory Avail = Avail - 100 = 200MB.

[0057] Step S5 in this embodiment includes: allocating priority according to the priority of each computing node i. i The parameter nodes P of each computation node i are ordered from highest to lowest. i,0 Construct replica parameter node P i,j ,in , Let P be the number of NUMA nodes in the system, and let P be the replica parameter node. i,j At that time, if the available memory size Avail is reduced by the replica parameter node P i,j If the memory usage result is less than 0, it is determined that the available memory is exhausted, and the process jumps to step S6; otherwise, the available memory size Avail is reduced by the replica parameter node P. i,j The result of the memory usage is used as the new available memory size Avail, and continues for the next parameter node P. i,0 Construct replica parameter node P i,j Until all parameter nodes P are completed i,0 Construct replica parameter node P i,j Then proceed to step S6. In this embodiment, when the available memory Avail for the model parameters is 200MB, all computing nodes have... Each parameter node also requires 3 × 100 = 300 MB, therefore high-priority compute nodes have There are 1 parameter nodes, and low-priority computation nodes have 1~ Each parameter node.

[0058] In step S6 of this embodiment, when traversing the compute nodes to allocate the memory of the corresponding replica parameter nodes to NUMA nodes to minimize the maximum distance between all threads and the data, the maximum distance between all threads and the data can be expressed as: minimize max(t∈threads){distance(t,data)}, where minimize means minimize, max means maximize, t∈threads represents thread t in the thread set threads, and distance(t,data) represents the distance between thread t and data data. Assuming there are two NUMA nodes and one parameter node, and threads 1 and 2 are both on NUMA 0, then to minimize the distance between the threads and the data, all memory of the parameter node needs to be allocated to NUMA 0. Assuming thread 1 is on NUMA 0 and thread 2 is on NUMA 1, if there are two parameter nodes, they are allocated to the two NUMA nodes respectively; however, if there is only one parameter node, it needs to be evenly divided and allocated to the two NUMA nodes. In this embodiment, step S6 includes:

[0059] S6.1, Traverse the computation nodes in the computation graph of the DNN. If the traversal is complete, jump to step S7; otherwise, jump to step S6.2.

[0060] S6.2, determine whether it is the replica parameter node P constructed by the current computing node. i,j The number of replicas, if the replica parameter node P i,j If the quantity is 1, then proceed to step S6.3; if the replica parameter node P i,j The number is greater than 1 and less than the number of NUMA nodes in the system. If so, proceed to step S6.4; if the replica parameter node P i,j The number is equal to the number of NUMA nodes in the system. If so, proceed to step S6.5;

[0061] S6.3, set the replica parameter node P of the current compute node to... i,j The memory is mapped to all NUMA nodes using a round-robin strategy, and then proceed to step S6.1; where the round-robin strategy refers to a round-robin scheduling strategy, and in this embodiment, it means using a round-robin method to bind memory pages among all NUMA nodes;

[0062] S6.4, transfer all replica parameter nodes P of the current compute node. i,j The memory is evenly distributed among NUMA nodes to obtain a node set NodeSubset. i,j Then, set all replica parameter nodes P of the current compute node as replicas. i,jMapped to the node set NodeSubset using a round-robin strategy i,j On the NUMA node, proceed to step S6.1;

[0063] S6.5, transfer all replica parameter nodes P of the current compute node. i,j The memory allocated to the NodeSet, which consists of NUMA nodes in the system. j On the NUMA node, proceed to step S6.1.

[0064] Specifically, in this embodiment, for any computation node i, if there are 4 parameter nodes, then P i,n Mapping to Node n If there are 3 parameter nodes, then P i,0 Mapped to Node0, P i,1 Mapped to Node1, P i,2 Map to Node2 and Node3 in a round-robin fashion; if there are two parameter nodes, then P... i,0 Mapped to Node0 and Node1 in a round-robin manner, P i,1 It is also mapped to Node2 and Node3 in a round-robin manner; if there is only one parameter node, then P is... i,0 Map to Node0~Node3 in a round-robin manner.

[0065] In this embodiment, step S7 includes:

[0066] S7.1, Traverse the computation nodes in the computation graph of the DNN. If the traversal is complete, jump to step S8; otherwise, jump to step S7.2.

[0067] S7.2, for the current computing node n, construct a loop optimization template using one or more of the following methods: reordering, splitting, and unrolling. n The loop rearrangement refers to swapping the inner and outer loops of a computation node; loop splitting refers to breaking down a computation node's loop into multiple loops; and loop unrolling refers to removing the loop and directly writing the operation for each loop. The operators corresponding to the computation nodes in the computation graph of a DNN are usually multi-layered nested loops. Swapping the inner and outer loops (loop rearrangement), splitting a loop into multiple loops (loop splitting), and removing the loop and directly writing the operation for each loop (loop unrolling) are all methods used to construct an optimization template. Different optimization templates can be understood as using different methods to execute the same operator. These methods are all well-known.

[0068] S7.3, Optimization of Templates for Loops n Parallelization of the outermost loop, loop optimization template n The outermost loop is:

[0069] #pragma omp parallel for / / OpenMP parallel declaration

[0070] for(){

[0071] do something

[0072] }

[0073] Here, "do something" represents the execution operation of the outermost loop;

[0074] Within the loop, model parameters are retrieved from the nearest NUMA node, and the address of the input feature map (Input) is set. i,t The nearest replica node P of the NUMA node where the thread resides. i,j The starting address of memory:

[0075] Input i,t =Addr(Nearest(P i,j )),

[0076] Here, Nearest represents the nearest operation. Previously, multiple replica parameter nodes were constructed and mapped to different NUMA nodes. This section determines which replica parameter node a thread accesses. Clearly, the thread should access the parameter node on its own NUMA node for the fastest memory access. Since the parameter node on the thread's NUMA node may not be complete, the nearest operation (Nearest) is used here. Within the loop, based on the loop count Num(Roll)... i Calculate the memory size required for intermediate results in each loop iteration and allocate memory accordingly:

[0077] Output i,t =Alloc(sizeof(Output i ) / Num(Roll i )),

[0078] Among them, Output i This represents the total memory required for intermediate results by the current computing node. `Alloc` is the memory allocation operation, and `sizeof(Output)` is the memory allocation value. i ) represents the total memory size required for intermediate results by the current computing node, Num(Roll) i ) represents the number of iterations; with Figure 2For example, assuming parameter node 0 is assigned to NUMA0 and NUMA1, and parameter node 1 is assigned to NUMA2 and NUMA3, then thread 0 running on NUMA0 should access the memory of parameter node 0; the required memory size for intermediate results is Output. i Unrelated to the parameter nodes mentioned earlier, this node is used to store computation results. Since the main thread and child threads may not be on the same NUMA node, and threads allocate memory on their own NUMA node by default, the child thread is allowed to allocate the necessary memory to store computation results to avoid cross-NUMA node memory access. An optimized template for the fusion operator is constructed through free combinations of methods such as loop reordering, loop splitting, and loop unrolling. Assuming the output size is 4MB, the outermost loop has 8 iterations, and there are 4 parameter nodes, then the Output... i,t =Alloc(4M / 8*sizeof(fp32));

[0079] S7.4, evaluate the DNN inference latency of the template (the time for one forward propagation of DNN inference). If the DNN inference latency meets the optimization objective (it can be less than a set value, or the minimum value selected from multiple solutions, etc.), the process ends; otherwise, return to step S7.1. For example, in this embodiment, TVM evaluates the execution efficiency of the current template using the xgboost model. If the number of evaluated templates reaches the optimization objective, the compilation optimization ends and the optimal template is returned.

[0080] In summary, the NUMA-aware DNN compilation system optimization method in this embodiment targets neural network inference scenarios under NUMA architecture. It allocates memory for specific data required by neural network inference jobs in a NUMA-aware manner, thereby ensuring that neural network jobs are evenly allocated memory on the NUMA architecture and improving the local memory access ratio, which can improve the execution efficiency of DNN inference jobs.

[0081] Furthermore, this embodiment also provides a NUMA-aware DNN compiler optimization system, including a microprocessor and a memory interconnected, wherein the microprocessor is programmed or configured to execute the NUMA-aware DNN compiler optimization method. This embodiment also provides a computer-readable storage medium storing a computer program or instructions programmed or configured to execute the NUMA-aware DNN compiler optimization method via a processor. This embodiment also provides a computer program product, including a computer program or instructions programmed or configured to execute the NUMA-aware DNN compiler optimization method via a processor.

[0082] Those skilled in the art will understand that the technical solutions provided by this invention may take the form of a method, system, or computer program product. Therefore, this invention may take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this invention may take the form of a computer program product embodied on one or more computer-readable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code. This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, produce an implementation of the flowchart... Figure 1 One or more processes and / or boxes Figure 1 The computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to operate in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The functions specified in one or more boxes. These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable apparatus for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0083] The above description is merely a preferred embodiment of the present invention. The scope of protection of the present invention is not limited to the above embodiments. All technical solutions falling within the scope of the present invention's concept are within the scope of protection of the present invention. It should be noted that for those skilled in the art, any improvements and modifications made without departing from the principles of the present invention should also be considered within the scope of protection of the present invention.

Claims

1. A NUMA-aware DNN compilation system optimization method, characterized in that, The method comprises the following steps: S1, initializing a thread-node mapping to map threads required for DNN inference onto all NUMA nodes; S2, initializing available memory Avail according to the current available memory size of the system; S3, obtaining computing nodes in a computation graph of the DNN, constructing parameter nodes for providing neural network parameters required for computation for each computing node, pre-allocating model parameter memory, and updating available memory Avail; S4, taking the computation complexity of each computing node as an allocation priority and sorting according to the allocation priority; S5, constructing replica parameter nodes required for mapping to different NUMA nodes for the computing nodes according to the allocation priority and updating available memory Avail until available memory Avail is exhausted; S6, traversing the computing nodes to allocate the memory of the replica parameter nodes corresponding to the computing nodes to the NUMA nodes to minimize the maximum data distance among all threads; S7, performing operator optimization for each operator corresponding to the computing nodes, including constructing an operator optimization template, parallelizing the outermost loop, allocating parameter nodes as inputs in the loop body, allocating memory locally as outputs, and evaluating the optimization effect; S8, determining whether the optimization target is completed, and if not, jumping to step S7; otherwise, ending and exiting. 2.The NUMA-aware DNN compilation system optimization method of claim 1, wherein, In step S2, when initializing available memory Avail according to the current available memory size of the system, available memory Avail is the difference between the current available memory size of the system and the reserved intermediate result memory size, wherein the reserved intermediate result memory size refers to the memory size required by the intermediate input variables of the DNN. 3.The NUMA-aware DNN compilation system optimization method of claim 1, wherein, Before step S3, the method further comprises the step of generating a computation graph of the DNN by using a graph optimization technique, wherein the graph optimization technique includes some or all of expression simplification, common expression elimination, constant propagation, matrix merging, and operator fusion, so that each operator is a computing node in the computation graph.

4. The NUMA-aware DNN compilation system optimization method of claim 1, wherein, Step S3, updating available memory Avail, includes: obtaining the construction parameters for each compute node i and node P respectively. i,0 The size of the memory pre-allocated model parameters is sizeof(P) i,0 ), the sizeof(P) of the pre-allocated model parameter memory for all computing nodes i. i,0 The total size of the pre-allocated model parameter memory, sizeof(P), is obtained by summing the values. The available memory, Avail, is then subtracted from the total size of sizeof(P) to obtain the updated available memory, Avail.

5. The NUMA-aware DNN compilation system optimization method of claim 1, wherein, Step S5 includes: allocating priority according to each computing node i. i The parameter nodes P of each computation node i are ordered from highest to lowest. i,0 Construct replica parameter node P i,j ,in , Let P be the number of NUMA nodes in the system, and let P be the replica parameter node. i,j At that time, if the available memory size Avail is reduced by the replica parameter node P i,j If the memory usage result is less than 0, it is determined that the available memory is exhausted, and the process jumps to step S6; otherwise, the available memory size Avail is reduced by the replica parameter node P. i,j The result of the memory usage is used as the new available memory size Avail, and continues for the next parameter node P. i,0 Construct replica parameter node P i,j Until all parameter nodes P are completed i,0 Construct replica parameter node P i,j Then proceed to step S6.

6. The NUMA-aware DNN compilation system optimization method of claim 1, wherein, Step S6 comprises: S6.1, traversing the computing nodes in the computation graph of the DNN, and if the traversal is complete, jumping to step S7; otherwise, jumping to step S6.2; S6.2, determine whether it is the replica parameter node P constructed by the current computing node. i,j The number of replicas, if the replica parameter node P i,j If the quantity is 1, then proceed to step S6.3; if the replica parameter node P i,j The number is greater than 1 and less than the number of NUMA nodes in the system. If so, proceed to step S6.4; if the replica parameter node P i,j The number is equal to the number of NUMA nodes in the system. If so, proceed to step S6.5; S6.3, set the replica parameter node P of the current compute node to... i,j The memory is mapped to all NUMA nodes using a round-robin strategy. Proceed to step S6.

1. S6.4, evenly distribute the memory of all replica parameter nodes P i,j of the current computing node to the NUMA nodes of the node set NodeSubset i,j , and then map all replica parameter nodes P i,j of the current computing node to the NUMA nodes in the node set NodeSubset i,j in a round-robin strategy, and jump to step S6.1; S6.5, transfer all replica parameter nodes P of the current compute node. i,j The memory allocated to the NodeSet, which consists of NUMA nodes in the system. j On the NUMA node, proceed to step S6.

1.

7. The NUMA-aware DNN compilation system optimization method of claim 1, wherein, Step S7 comprises: S7.1, traversing the computing nodes in the computation graph of the DNN, and if the traversal is complete, jumping to step S8; otherwise, jumping to step S7.2; S7.2, constructing a loop optimization template Template for the current computing node n by using one or more of the following methods: loop rearrangement, loop splitting, loop unrolling n ; wherein the loop rearrangement refers to exchanging the positions of the inner loop and the outer loop of the computing node, the loop splitting refers to splitting the loop of the computing node into multiple loops, and the loop unrolling refers to removing the loop and directly writing the operation of each loop. S7.3, for the loop optimization template Template n the outermost loop parallelization, the model parameters are fetched from the nearest NUMA node in the loop body, and the address of the input feature map Input i,t the parameter node P of the nearest copy for the thread's NUMA node i,j the starting address of the memory of the parameter node P Input i,t =Addr(Nearest(P i,j )), Input i,t =Addr(Nearest(P i,j )), Input where Nearest is a nearest operation; and within the loop, the memory size of the intermediate result needed for each loop is calculated according to the loop number Num(Roll i ) and memory is allocated: Output i,t =Alloc(sizeof(Output i ) / Num(Roll i )), Output i,t =Alloc(sizeof(Output i ) / Num(Roll i )), wherein Output i is the total memory needed for the intermediate result of the current computing node, Alloc is the operation of allocating memory, sizeof(Output i ) is the total memory size needed for the intermediate result of the current computing node, Num(Roll i ) is the number of loops; S7.4, evaluating the DNN inference delay of the template, and if the DNN inference delay meets the optimization target, ending; otherwise, returning to step S7.

1.

8. A NUMA-aware DNN compilation system optimization system comprising interconnected microprocessors and memory, characterized by, The microprocessor is programmed or configured to perform the NUMA-aware DNN compilation system optimization method of any one of claims 1-7.

9. A computer-readable storage medium having stored therein a computer program or instructions, characterized in that, The computer program or instructions are programmed or configured to perform the NUMA-aware DNN compilation system optimization method of any one of claims 1-7 by the processor.

10. A computer program product comprising computer programs or instructions, characterized in that, The computer program or instructions are programmed or configured to perform the NUMA-aware DNN compilation system optimization method of any one of claims 1-7 by the processor.

Images