Method for distributing neural network computational operations between components of a heterogeneous computing platform

Abstract

FIELD: computing and information technology.SUBSTANCE: invention relates to methods for increasing the efficiency of performing computing tasks on multicomponent computing systems. A method is implemented for distributing computing operations between components of a heterogeneous computing platform where digital models are formed; determining, based on the models, candidate plans for placing operations on computers of the heterogeneous platform that have the best performance indicator, using a multi-level algorithm; verifying the selected candidate plans by testing them on the platform with measuring the actual execution time; selecting the most productive placement plan based on the test results or adjusting the performance model to improve the accuracy of forecasting; distributing computing operations using the most productive plan for placing operations on computers of the heterogeneous platform.EFFECT: increasing the efficiency of using the computing resources of a heterogeneous platform when performing computing operations, which is expressed in a reduction in the overall time of performing calculations, a reduction in the load on communication channels between the platform's computers, and ensuring adaptability to different equipment characteristics.11 cl, 2 dwg, 1 tbl

Description

[0001] Field of invention

[0002] The invention relates to computing and information technology, specifically to methods for improving the efficiency of computing tasks on multi-component computing systems. More specifically, the invention relates to a method for distributing computing operations between components of a heterogeneous computing platform to improve its performance. The invention can be used to perform neural network operations in areas such as computer vision systems, natural language processing, robotics, and in on-board computing systems of autonomous (unmanned) vehicles, where achieving minimal response time and maximum throughput in data processing is critical.

[0003] Background of the invention

[0004] Modern neural networks are characterized by high computational complexity and require significant hardware resources to run. To achieve the required performance when solving such networks, heterogeneous computing platforms have become widespread.

[0005] A heterogeneous computing platform integrates computing devices (computers) with different architectures and characteristics, such as central processing units (CPUs), graphics processing units (GPUs), specialized neural network accelerators (NPUs), field-programmable gate arrays (FPGAs), image processing units (ISPs), digital signal processors (DSPs), microcontrollers (MCUs), and others. Each of these components has a unique performance profile, power consumption, and supported set of operations, making it more or less efficient for executing different parts of the neural network computation graph.

[0006] A key challenge when using such platforms is the optimal distribution (also known as mapping, placement, or allocation) of neural network computational operations across available computing devices. Inefficient distribution leads to idle time for expensive accelerators, overloaded communication channels, and, consequently, increased overall execution time (latency) of the neural network. Solving this problem is a combinatorial one, since the space of possible distribution options grows exponentially with the number of network operations and the number of computing devices on the platform.

[0007] The method for distributing neural network computing operations between components of a heterogeneous computing platform, disclosed in the patent application [US 2020 / 0249998 A1, publication date: 06.08.2020], was selected as a prototype. The method includes the following main steps:

[0008] – formation of a computation graph, during which the neural network is represented as a directed acyclic graph, where the vertices (nodes) correspond to operations or individual nodes implementing independent algorithms, and the edges correspond to data dependencies;

[0009] – reduction of the computation graph, during which the subgraphs corresponding to known architectural blocks are replaced by a single composite vertex;

[0010] – graph partitioning, which involves cutting a simplified computation graph into several independent subgraphs;

[0011] – search for the optimal placement plan, during which optimization is performed for each subgraph using a reinforcement learning algorithm. During this stage, a sequence of trial placement plans is generated. Each trial plan is evaluated using a theoretical performance model, which, based on previously collected data on the characteristics of the computing devices and communication channels, calculates a predicted performance indicator (total latency). Based on this assessment, the algorithm adjusts the next generated trial plan to maximize the performance indicator;

[0012] – combining the optimal plans found for each subgraph and forming the final plan for placing operations of the entire neural network.

[0013] The main disadvantage of the prototype is the low efficiency of using the computing resources of the heterogeneous platform when performing operations, which increases the overall execution time of calculations.

[0014] This flaw is related to the prototype's use of a theoretical performance model based solely on predictions derived from static, pre-collected data. The prototype does not include a mechanism for verifying these predictions or selecting a final solution based on actual, rather than simulated, performance.

[0015] This approach leads to the problem of a gap between the model and reality, as no analytical or simulation model can accurately account for the entire complex, nonlinear, and dynamic factors that influence the performance of a real computing platform. Factors that cannot be modeled by a prototype include:

[0016] – competition for system resources, since the prototype model estimates data transfer costs based on static channel bandwidth. In reality, the effective bandwidth of, for example, the PCIe bus, dynamically changes due to its simultaneous use by multiple computers and other system devices, leading to unpredictable latencies not accounted for in the model;

[0017] – microarchitectural features, as the operation execution time taken from the profiling database is an average value. It does not take into account the actual state of the processor cache at the time of execution, which can lead to significant deviations in node performance;

[0018] – overhead costs of the software environment, since the prototype does not model the delays introduced by the operating system scheduler, device driver calls when initiating calculations or transferring data, as well as the overhead costs of the neural network framework runtime environment itself;

[0019] – dynamic changes in the platform state, since the actual performance of the processors may vary over time due to the effects of thermal throttling, dynamic changes in clock frequencies, or the influence of background processes of the operating system.

[0020] The consequence of this discrepancy between forecast and actual performance is that the layout plan the prototype identifies as optimal is only so within the framework of its simplified and inaccurate model. In practice, such a plan may prove significantly less productive than other, alternative plans that the model mistakenly rejected during the optimization phase. Having found its theoretical optimum, the prototype terminates its operation without a mechanism for validating this choice and comparing it with other candidates under real-world operating conditions.

[0021] Furthermore, the optimization approach itself, in which the original computation graph is divided into several independent subgraphs, each of which is individually and once optimized, while reducing complexity, breaks dependencies in the graph and reduces the accuracy of local plan determination. Plans that are locally optimal for each individual subgraph do not, taken together, guarantee an optimal overall operation placement plan.

[0022] This leads to suboptimal utilization of the heterogeneous platform's computing resources, as the selected plan can lead to computer downtime and overloaded communication channels. Furthermore, the prototype lacks an adaptive feedback mechanism, preventing it from adjusting and improving the performance model's accuracy based on data obtained during real runs, thereby adapting it to the specific hardware configuration.

[0023] Thus, the technical problem that the invention is aimed at solving is to eliminate the existing shortcomings of the prototype and to increase the efficiency of the known method of distributing computing operations.

[0024] Disclosure of the essence of the invention

[0025] The technical result that the invention is aimed at achieving consists in increasing the efficiency of using the computing resources of a heterogeneous platform when performing computing operations, expressed in a reduction in the overall time of execution of calculations (decreased latency), a reduction in the load on communication channels between the platform's computers and ensuring adaptability to the actual characteristics of the equipment by distributing operations between different types of computers, taking into account their actual characteristics and the intensity of data exchange between operations or individual nodes.

[0026] An additional technical result that the invention is aimed at achieving is a reduction in energy consumption during operations due to more efficient use of platform computers and minimization of redundant data transmissions.

[0027] An additional technical result, which the invention is aimed at achieving, consists in increasing the predictability of the execution time of neural network operations in real-time systems, including autopilot systems for vehicles, where a guaranteed response within specified time frames is required.

[0028] An additional technical result, which the invention is aimed at achieving, consists in increasing the throughput in processing sensory data in vehicle autopilot systems due to the optimal distribution of software units (ROS2 nodes) that perform a specific function in the overall system of autonomous (unmanned) vehicles, between the components of the on-board computing platform.

[0029] An additional technical result, which the invention is aimed at achieving, is to reduce the response time of the vehicle control system by minimizing delays in the processing and analysis of data from sensors and thereby increasing the actual metrics of the accuracy of the operation of vehicle control systems.

[0030] The essence of the invention lies in a method for distributing computing operations between components of a heterogeneous computing platform, where the computing operations correspond to neural networks, individual operations of neural networks or individual algorithms, and the method includes steps during the execution of which the following is carried out:

[0031] – formation of digital models, including:

[0032] – a computation graph where the nodes correspond to computational operations and the edges correspond to data dependencies between them;

[0033] – a graph of a computing platform, in which the vertices correspond to the platform’s computers with an indication of their characteristics, and the edges correspond to the communication channels with an indication of their characteristics;

[0034] – a system performance model that predicts the total execution time of a computation graph in the form of computational costs for performing operations on computers of a heterogeneous platform and communication costs for transferring data between computers of a heterogeneous platform;

[0035] – determination, based on the specified models, of candidate plans for placing operations on computers of a heterogeneous platform that have the best performance indicator, using a multi-level algorithm that includes:

[0036] – reduction of the computation graph into its simplified representation by aggregating groups of related operations;

[0037] – definition of the initial layout plan for the specified simplified representation;

[0038] – restoration of the detailed placement plan for the original computation graph with its subsequent local optimization;

[0039] – verification of selected candidate placement plans by testing them on a heterogeneous computing platform with measurement of actual execution time;

[0040] - Select the most productive deployment plan based on test results or adjust the performance model to improve the prediction accuracy;

[0041] – distribution of computing operations between components of a heterogeneous computing platform using the selected most productive plan for placing operations on the computers of the heterogeneous platform.

[0042] At the initial stage of the developed method, digital models are generated that, taken together, form a digital twin of the system being optimized, including a heterogeneous computing platform and the neural network running on it. To achieve this, a computational graph is formed, where vertices correspond to operations or individual nodes implementing independent algorithms, and edges correspond to data dependencies. Formation of this graph is a prerequisite for subsequent communication cost analysis. Formation of the computational graph can be accomplished through syntactic and semantic analysis of the neural network architecture, represented in one of the standard formats, such as ONNX, TensorFlow GraphDef, or TorchScript. The vertices in such a graph represent specific computational operations or nodes, while the edges represent data flows, i.e., tensors, transferred between these operations or nodes.

[0043] Each vertex is associated with the computational complexity of the operation, and each edge is associated with the amount of data transferred in bytes. It should be noted that operations can be not only mathematical operations but also larger software units, such as ROS2 nodes, and the computation graph, accordingly, can be a ROS2 graph. This application is particularly relevant for autonomous vehicle systems, where software is often built on the ROS2 framework, and multiple interacting software modules (nodes) must be efficiently distributed across the components of the onboard computing system. This allows for increased throughput in processing sensor data and reduced response time of the vehicle control system.

[0044] Simultaneously with the computation graph generation stage, or before its execution, a computing platform graph is generated, where vertices correspond to physical computers and edges correspond to communication channels. The computing platform graph provides all the necessary information about the hardware capabilities and limitations of the system, which is a prerequisite for finding a technically feasible and high-performance placement plan.

[0045] Graph formation can be achieved by profiling the platform by collecting and organizing data on hardware resources. Vertices can represent various types of computing units (processors) in a heterogeneous computing platform, including central processing units (CPUs), graphics processing units (GPUs), neural network accelerators (NPUs), field-programmable gate arrays (FPGAs), as well as specialized image processing accelerators (ISPs), digital signal processors (DSPs), and microcontrollers (MCUs).

[0046] Each vertex is associated with a set of characteristics that describe its computational capabilities. These characteristics can be obtained from technical documentation or, preferably for improved model accuracy, by running a set of specialized tests (microbenchmarks) on real hardware. The key characteristics specified for vertices include, at a minimum, a performance profile defining the execution speed of various types of operations (e.g., in FLOPS or TOPS) for different data formats (e.g., FP32, FP16, INT8), the amount of available memory (e.g., VRAM for GPUs or RAM for CPUs), and support for specialized hardware units or instructions, such as tensor cores in GPUs.

[0047] The edges of a computing platform graph correspond to physical or logical communication channels connecting pairs of computing devices. Each edge represents a path along which data can be transmitted between two different computing devices. Each edge is associated with a set of characteristics that define its throughput capabilities. The key characteristics include, at a minimum, effective throughput, latency, and others. These characteristics are also determined either from specifications or through direct measurements on the platform.

[0048] Based on the resulting models, a system performance model is created. This model is a mathematical expression, typically in the form of an objective function, that predicts the total network execution time as the sum of two main components: computational cost, which is the estimated execution time of each operation on its assigned computer, and communication cost, which is the estimated time required to transfer data between operations or nodes if they are located on different physical computers. This predictive model enables the selection of candidate placement plans for computing operations, and its accuracy directly impacts the quality of the selected candidate placement plans.

[0049] In the context of the present invention, a placement plan (also known as a mapping or allocation) is a machine-readable instruction that uniquely specifies on which physical computer on a heterogeneous platform each computational operation should be executed. One or more candidate placement plans for computational operations are determined using a multi-level algorithm that solves a combinatorial optimization problem, thereby finding the most high-performance placement plans for large-scale computational graphs in minimal time.

[0050] The specified multi-level optimization algorithm is implemented by: reducing the computation graph, determining the initial placement plan for the specified simplified representation, and restoring the detailed placement plan for the original computation graph with its subsequent local further optimization.

[0051] Computation graph reduction is the process of iteratively simplifying a graph by aggregating groups of related operations into composite nodes. This step is key to optimizing communication costs, as it transforms the original high-dimensional graph into an equivalent graph with a smaller number of coarse-grained elements. Reduction may involve identifying a highly connected subgraph within the computation graph. Instead of merging random or simply adjacent nodes, the algorithm deliberately analyzes the graph topology and data flows to find groups (clusters) of operations that intensively communicate primarily with each other. Such a group of operations is a logical candidate for colocation on a single physical computer. Once identified, such a subgraph is considered as a single entity and aggregated into a single composite node for subsequent steps.

[0052] The identification procedure can be performed by first transforming the original directed computational multigraph into a weighted undirected graph, which further reduces the load on communication channels between platform computers and, consequently, reduces the overall computational time. The original multigraph is the most accurate representation of a system where several parallel and multidirectional data channels can exist between a pair of operations. The transformation aggregates all these connections into a single metric—the total data exchange intensity, which becomes the edge weight in the new, undirected graph. This simplified but informative representation is used to search for a fully connected group of operations (clique), that is, a subgraph where each operation is connected to all others, with the maximum total weight of internal connections.

[0053] A heuristic algorithm can be applied to the resulting fully connected group of operations. This algorithm uses the specified group as the cluster core, i.e., the initial crystallization center. An iterative cluster creation process then occurs, in which peripheral operations (adjacent to those already included in the cluster) are sequentially added to this core, one at a time. The criterion for adding operations at each step is maximizing internal traffic; that is, the operation most closely related to the already formed group is selected. This process continues until all related operations are exhausted or the computing resource limits of the computer are reached (e.g., RAM or CPU load limit).In this case, a separate cluster of operations can be formed for each computer in the computing platform graph, and already allocated operations can be excluded from consideration when forming subsequent clusters. Thus, after the cluster for the first computer is fully formed, all operations included in it are excluded from further consideration, and the process can be repeated for the remaining operations and the next available computer.

[0054] Determining an initial placement plan for the resulting simplified graph enables the application of computationally intensive heuristic or exact optimization methods, such as simulated annealing or genetic algorithms. At this stage, for each constituent node, representing a coarse-grained block of operations, the most suitable physical computer is determined based on a performance model. The result is a high-level placement plan that determines the strategic distribution of the main functional blocks. This initial plan serves as the basis for the subsequent reconstruction and local optimization stage.

[0055] During detailed placement plan reconstruction, the plan obtained for the simplified graph is iteratively transferred to the original, detailed graph and locally optimized. Local optimization is a procedure for iteratively improving the current placement plan. After a composite node mapped to a specific compute node is disaggregated into its constituent initial operations, the local optimization algorithm searches for modifications to the placement of individual boundary operations that lead to an improvement in the predicted performance metric.

[0056] Local optimization can be achieved by pairwise reordering of operations in opposite directions, which simultaneously exchanges operations between two different computers. Pairwise reordering involves trial moves of operations / operations from one cluster (and, consequently, from one computer) to another. Each trial move is evaluated using a performance model. If the move improves (decreases) the predicted performance, it is accepted; otherwise, it is rejected. This mechanism allows for modification of the placement plan while strictly adhering to hardware resource constraints, such as available memory.

[0057] Verification of selected candidate placement plans eliminates the discrepancy between the performance model's prediction results and the hardware capabilities of a heterogeneous computing platform. This is accomplished by testing each candidate plan on a real platform. The testing process may include automatically generating executable code or a configuration file for the target framework (e.g., TensorFlow, PyTorch, TensorRT), which forces operations to be placed on the compute devices according to the plan being tested. Operations, algorithms, or the entire neural network are then run multiple times, with the actual execution time measured for each cycle, allowing for statistically reliable performance data.Initial runs can be discarded, ensuring statistical reliability and stability of measurements, as the first runs of a computing task on a platform are often unrepresentative due to overhead costs such as just-in-time compilation or library initialization. Removing these artifacts from the sample allows for measuring system performance in a steady-state, steady-state mode.

[0058] Based on the remaining runs, the end-to-end execution time can be averaged, eliminating the impact of stochastic factors such as short-term background operating system activity or clock rate fluctuations, and obtaining a single performance metric for each candidate plan. This combination of techniques allows for a highly accurate comparison of candidate plans and an informed choice in favor of the one that performs best under real, rather than simulated, conditions.

[0059] Based on the results obtained during testing, the most efficient placement plan that demonstrates the best actual performance indicator, such as the shortest measured average execution time, is selected.

[0060] Measurement results can be used to adjust the performance model, creating a feedback mechanism. During this process, the model's predicted values ​​are compared with actual measurement results. Systematic discrepancies identified (e.g., consistent underestimation of PCIe bus data transfer times) are used to refine the parameters in the numerical models, particularly in the compute platform graph. This adaptive mechanism allows the model to learn from real data, improving the accuracy of its predictions for subsequent allocation tasks. For example, if the actual execution time is systematically higher than the predicted time for all or most candidate plans that involve intensive PCIe bus transfers, this indicates that the performance model is underestimating the communication costs for this link.Based on the identified discrepancies, subsequent modifications (adjustments) of the computing platform graph parameters are performed. In the example shown, this could result in a reduction in the throughput value or an increase in the latency value for the edge corresponding to the PCIe bus. Similarly, node parameters can be adjusted, for example, the performance factors of a specific computer for certain types of operations, if the model systematically misestimates computational costs. This iterative self-calibration process allows the performance model to adapt to the specifics of a particular hardware configuration and software environment, improving the accuracy of its predictions for subsequent operation assignments and enhancing its adaptability to real-world hardware characteristics.

[0061] The distribution of computing operations is carried out in accordance with the most productive placement plan selected during the verification stage. At this stage, the configuration generated for the selected plan is deployed to the target platform for productive operation, which is the ultimate goal of the entire method.

[0062] The invention is characterized by a previously unknown set of essential features from the prior art, distinguished in that:

[0063] - Candidate allocation plans are determined using a multi-level algorithm, including: computation graph reduction, initial plan generation for a simplified representation, and reconstruction of a detailed plan with local optimization. This approach (multi-level reduction and multi-level reconstruction), unlike graph partitioning and single-step optimization, allows for finding optimal solutions for individual operations, algorithms, or the neural network as a whole, and more accurately adapting the final allocation to the specifics of the computing platform.

[0064] This directly reduces the load on communication channels, as the reduction stage creates logically linked computational blocks designed for co-location on a single physical computer, minimizing the amount of data transferred over external buses. At the same time, local optimization further refines the placement plan. This process, which includes, for example, pairwise permutations of individual operations between clusters, enables more precise tuning of the distribution to the microarchitectural features of the platform, identifying small but important improvements that further contribute to reducing the overall computation time.

[0065] - Verify the selected candidate plans by testing them on a real heterogeneous computing platform and measuring actual execution times, which helps address the issue of the practical applicability of the model. This reduces the overall computation time, since the final choice is made not based on theoretical predictions, but on actual measurements that take into account all factors not accounted for in the model.

[0066] - selects a final plan based on test results and subsequently adjusts the performance model, which not only ensures the best solution is selected but also enables the creation of an adaptive self-learning system. Thanks to the resulting feedback mechanism, in which discrepancies between the forecast and actual measurements are used to refine the platform graph parameters, the system is trained on real data. This affects adaptability to real hardware characteristics, as the model becomes more accurate over time, adapting to the specific hardware configuration, which is impossible with a static model.

[0067] The combination of features of the developed method, namely the combination of a multi-level algorithm with subsequent verification and adaptive feedback, creates a synergistic effect that allows solving the technical problem at a qualitatively different level, unattainable using each of these approaches separately.

[0068] Each approach individually has fundamental limitations:

[0069] – using only theoretical modeling and optimization for individual parts of the graph (as in the prototype) allows for the rapid exploration of a large space of options, but inevitably leads to suboptimal solutions due to the inability of the model to adequately reflect all the complexities of a real computing system, thus obtaining a quick but inaccurate result;

[0070] – Using an empirical enumeration would allow us to find the best plan, since it is based on real measurements. However, the space of possible placement plans for modern neural networks is excessively large, making an exhaustive enumeration computationally infeasible in any reasonable time.

[0071] The synergistic effect in the developed method arises from the fact that these two approaches are not used sequentially, but within the framework of a single closed cycle, where they compensate for the shortcomings and repeatedly enhance each other's advantages.

[0072] Specifically, a multi-level algorithm addresses the computational complexity of the empirical approach. It eliminates many obviously poor options and narrows the vast search space to a small, manageable set of several strong candidate plans. Verification addresses the inaccuracy of theoretical modeling, allowing one to be selected from the set of strong candidates proposed by the model, the one most effective in real-world conditions. Adaptive feedback closes this loop, creating a self-reinforcing effect. It uses accurate data obtained during the verification stage to adjust and improve the model. As a result, in subsequent runs, the model becomes more accurate and begins to suggest even more efficient placement plans, increasing the overall efficiency of the entire process.

[0073] This ensures the achievement of a technical result consisting in increasing the efficiency of using the computing resources of a heterogeneous platform when performing computing operations, expressed in a reduction in the overall time of execution of calculations (decreased latency), a reduction in the load on communication channels between the platform's computers and ensuring adaptability to the actual characteristics of the equipment by distributing operations between different types of computers, taking into account their actual characteristics and the intensity of data exchange between operations or nodes.

[0074] The invention has a set of essential features previously unknown in the prior art, which indicates its compliance with the patentability criterion of “novelty”.

[0075] The invention is unknown in the prior art and does not clearly follow from it. Therefore, the invention meets the patentability criterion of "inventive step."

[0076] The invention is explained by the following figures.

[0077] Fig. 1 – Block diagram of a system for implementing a method for distributing computing operations between components of a heterogeneous computing platform.

[0078] Fig. 2 – Block diagram of the algorithm for distributing computing operations between components of a heterogeneous computing platform.

[0079] Implementation of the invention

[0080] The developed method for distributing computing operations between components of a heterogeneous computing platform is implemented by means of a system 100 (Fig. 1) including a computing unit 110 designed to perform the stages of forming models and determining placement plans; a heterogeneous computing platform 120 on which a neural network, individual operations of neural networks or individual algorithms are executed and tests are carried out; communication means between unit 110 and heterogeneous platform 120.

[0081] The control computing unit 110 includes: a processor; a storage device; a data transmission interface; software located in the storage device and executed by the processor. The software of the computing unit 110 includes the following functional modules: an analysis module 111 designed to convert a neural network model into a computation graph; a platform profiling module 112 designed to determine the characteristics of the components of a heterogeneous platform; an optimization module 113 implementing a multi-level optimization algorithm for determining candidate placement plans; a testing module 114 for performing measurements on the target platform; a feedback module 115 for adjusting the performance model.

[0082] The heterogeneous computing platform 120 includes various types of computing devices: central processing units; graphic processing units; specialized neural network accelerators and other computing devices.

[0083] The method for distributing computing operations between components of a heterogeneous computing platform is implemented as follows.

[0084] Digital Model Generation Stage. This preparatory stage is aimed at significantly reducing the search space for the optimal placement plan. During this stage, a set of digital models (twins) is created, allowing for the simulation of the computational graph execution on the computing platform.

[0085] The input object is a neural network represented in one of the standard formats, such as ONNX, TensorFlow GraphDef, or TorchScript. The analysis module 111, represented as a specialized parser, analyzes this model and transforms it into a directed acyclic computation graph.

[0086] .

[0087] This stage is implemented in the following steps.

[0088] Step 200. Building the computation graph. In this step, each vertex represents a separate computational operation. Examples of such operations include:

[0089] – Basic operations of neural networks, such as 2D convolution (Conv2D), matrix multiplication (MatMul), activation function (e.g. ReLU, Sigmoid, SoftMax), pooling operation (MaxPool, AvgPool), normalization (BatchNorm, LayerNorm);

[0090] – arithmetic and mathematical operations, such as addition (Add), subtraction (Sub), multiplication (Mul), division (Div), exponentiation (Pow), square root (Sqrt), exponential (Exp), logarithm (Log), averaging (ReduceMean), summation (ReduceSum);

[0091] – matrix calculation and tensor transformation operations, such as generalized matrix multiplication (Gemm), tensor transposition (Transpose), tensor reshape (Reshape), tensor concatenation (Concat), tensor division (Split), tensor selection (Slice);

[0092] – specialized activation functions such as GELU activation (Gelu / Erf), hyperbolic tangent (Tanh), used in modern transformer and language model architectures;

[0093] – positional coding and masking operations, such as trigonometric functions (Sin, Cos) for positional coding, conditional replacement (Where), logical operations (Equal, Not, Less), tensor expansion (Expand);

[0094] – auxiliary operations such as selecting elements by indexes (Gather), adding / removing dimensions (Unsqueeze / Squeeze), changing data type (Cast), inserting constant values ​​(Constant);

[0095] – other operations defined by ONNX, TensorFlow, PyTorch and other machine learning frameworks.

[0096] To ​​every peak a vector of attributes is compared, including:

[0097] – operation type;

[0098] – operation parameters;

[0099] – types and sizes of input and output tensors;

[0100] – estimated computational complexity, for example in the number of floating point operations.

[0101] Each rib represents a data dependency between operations , where is the output tensor of the operation is an input to the operation . To each edge a weight equal to the size of the transferred tensor in bytes is associated, denoted by , .

[0102] Thus, the computational complexity of the nodes of the computation graph determines the computational cost, and the weight of the edges of the computation graph determines the communication cost.

[0103] Step 210. Construction of the graph of the computing platform 120. The heterogeneous computing platform 120, on which the neural network model is supposed to be executed, is also represented as a graph This task is performed by the platform profiling module 112.

[0104] Every peak corresponds to a physical computing device (computer) available on platform 120. For example, . Each peak A detailed performance profile is compared, which can be obtained from the manufacturer's specifications or, preferably, by running a set of microbenchmarks on real hardware.

[0105] Profile includes:

[0106] – peak performance for various data types (FP32, FP16, INT8);

[0107] – the volume and bandwidth of local memory (e.g. VRAM for GPU, L3 cache for CPU);

[0108] – support for specialized instructions (for example, tensor cores on GPUs).

[0109] Every rib is a communication channel between two computers . To each edge channel characteristics are compared:

[0110] – effective throughput in GB / s;

[0111] – delay in microseconds, that is, the minimum time to forward a zero-length packet.

[0112] Steps 200 and 210 can be performed in any order or simultaneously.

[0113] Step 220. Formation of the system performance model. The system performance model represented by the objective function defines the predicted performance indicator for a particular plan for placing operations on the computers of platform 120. It allows one to determine such a plan for placing , which minimizes the value of the objective function .

[0114] Function is a model of the total execution time taking into account the computation graph, as well as computational and communication costs, and consists of two components:

[0115]

[0116]

[0117]

[0118] ,

[0119] where:

[0120] – – model moment of the beginning and end of the execution of the vertex ,

[0121] - calculator from , onto which the vertex is mapped from ;

[0122] – — predicted computational costs (operation execution time on the computer This value is calculated based on the estimated complexity of the operation. (in FLOPS) and performance profile (in TFLOPS), with possible correction factors for different types of operations;

[0123] – — predicted communication costs (data transmission time along the edge , if the generating operation is performed on , and the consuming one is on ).

[0124] For example, for a linear graph .

[0125] In this case, if (operations are performed on one computer), is taken to be equal to zero or a small constant that models the access delay to local memory. If (intercomponent transmission), the time is calculated using the standard model: ,

[0126] where:

[0127] – DataSize(e) – the size of the data that needs to be transferred from the computer to the computer along edge (e);– Bandwith( ) – the throughput of the data transmission channel between computers And .

[0128] Candidate Layout Plan Determination Stage. At this stage, one or more potentially optimal operation layout plans are searched for based on previously generated digital models. To efficiently solve this computationally complex problem, a multi-level algorithm is used. It first reduces (simplifies) the computational graph, then determines an initial layout plan for the simplified representation, and reconstructs the detailed plan with subsequent local optimization.

[0129] This stage is implemented in more detail in the following steps.

[0130] Step 230. Determining candidate operation layout plans using a multi-level optimization algorithm. The determination of layout plans is performed by the optimization module 113, for which the following procedure is carried out:

[0131] 1. Graph reduction, in which the original computation graph is simplified by aggregating groups of related operations to obtain a more compact representation. This process can be iterative, with the graph elements gradually becoming larger. .

[0132] 2. Determining the initial layout plan for the specified simplified representation (simplified graph).

[0133] 3. Recovering the detailed placement plan, in which the placement plan obtained for the simplified graph is transferred to the original detailed graph with the possibility of local further optimization to improve the quality of the final distribution.

[0134] In one embodiment (variant 1), the multi-level optimization algorithm is implemented as follows:

[0135] 1. Graph reduction via sequential aggregation.

[0136] At this step, the original graph Iteratively simplifies. At each iteration, pairs of nodes are found and replaced with a single composite node. The criterion for merging is usually the edge weight between nodes: pairs connected by edges with a large amount of data transfer are prime candidates. For example, a convolution operation and the subsequent activation function are almost always efficiently executed on a single computer, so they are combined into a single composite node. This process is repeated, producing a sequence of increasingly generalized graphs. , Where contains several composite vertices.

[0137] 2. Determining the initial placement plan.

[0138] Distribution is carried out on a simplified graph . Since the number of vertices and edges in is of little importance for finding an optimal or close to it solution Exact (such as integer linear programming (ILP), branch and bound) or powerful heuristic methods (such as genetic algorithms), etc. can be applied.

[0139] 3. Restoring and refining the initial placement plan.

[0140] This phase is the reverse of the aggregation phase. Starting with the placement plan , the system iteratively returns to more detailed graphs. When a composite vertex is mapped to the computer , is revealed back into its components, all of which are initially placed on the same computer After this, the local refinement algorithm is launched. It checks whether moving one of the newly expanded operations to a neighboring computation will lead to a decrease in the overall value of the objective function. For such a local search, the Kernighan-Lin heuristic can be used, for example. This process is repeated at all levels of detail until a fully detailed and locally optimized plan is obtained. for the original graph

[0141] In another embodiment of the invention (variant 2), a heuristic multi-level optimization algorithm can be used, based on finding cluster cores (crystallization centers) and iterative cluster formation. This algorithm is a special case of a general optimization algorithm, where the graph reduction step is implemented through clustering into cliques representing a fully connected group of operations, or dense subgraphs, where each operation intensively exchanges data with all others.

[0142] 1. Graph reduction based on fully connected groups of operations (clicks), where each one intensively exchanges data with all the others.

[0143] Initial Directed Computational Multigraph , where vertices correspond to operations (for example, ROS2 nodes) and edges to data flows, is transformed into a weighted undirected graph In this graph, the edge weight reflects the total intensity of data exchange between operations in both directions. In the resulting graph A search is performed for a maximally weighted, fully connected group of operations (a clique)—a complete subgraph where each vertex is connected to all other vertices in the subgraph. The resulting clique is considered a composite vertex in the subsequent distribution of operations.

[0144] 2. Iterative cluster placement.

[0145] Placement of operations on platform calculators occurs iteratively, for each calculator in turn. For the next free calculator the following steps are performed:

[0146] 2.1. Determination of the cluster core (crystallization center) in which in the current graph The most closely connected group of operations is located among the unplaced operations. This group could be, for example, a maximally weighted, fully connected group of operations, where each of them intensively exchanges data with all the others (a clique). This fully connected group of operations becomes the core of the future cluster, which will be placed on the computer. .

[0147] 2.2. Formation of a cluster in which the cluster core is located on the computer The algorithm then iteratively analyzes peripheral operations around the already allocated operations (external, not yet allocated, operations associated with the core). At each step, it selects and adds to the same computer an adjacent operation that maximizes internal cluster traffic (i.e., its connection to already deployed operations or nodes is maximized). This process continues until one of two conditions is met: either all peripheral operations are exhausted, or adding another operation would exceed the compute limits. by resources (for example, CPU load or RAM usage limits taken from the platform profile).

[0148] 3. Update the graph and move to the next evaluator. All operations placed on evaluator u in the previous phase are removed from the graph. unplaced operations. The algorithm then moves to the next free processor and repeats step 2 for it. This process continues until all operations have been placed or until there are no more available processors with sufficient resources.

[0149] 4. Local refinement. After operations are placed, a local search procedure is run to further refine the resulting placement plan. This procedure may include, for example, iterating through permutations of individual operations between already formed clusters. For example, the algorithm may move an operation from the computer to the computer , checking whether this will lead to a decrease in the total inter-cluster traffic (the value of the objective function) and whether it will violate the resource constraints on To free up resources on a pairwise permutation can be performed when another operation moves simultaneously with on The deployment plan that showed the best predicted performance is retained.

[0150] Step 240. Generate a set of candidate operation placement plans. To obtain multiple candidate placement plans, the multi-level algorithm (in both variants 1 and 2) can be run by module 113 multiple times with different initial parameters. For example, with different random states for optimization methods, different computational traversal orders, or different cluster core (crystallization center) selection criteria. As a result, a set of the best placement plans is generated. , which are then passed on to the verification stage.

[0151] Verification of Selected Candidate Placement Plans. This stage is designed to validate and select the best candidate placement plan under production conditions. To do this, each selected plan is sequentially executed on a heterogeneous computing platform, followed by testing with actual performance metrics, such as total execution time. This stage helps eliminate model inaccuracies and select the most performant plan.

[0152] This stage is implemented in more detail as follows.

[0153] Step 250. Performing tests on the computing platform. For each candidate deployment plan, the testing module 114 performs the following procedure:

[0154] 1. Generation of an executable configuration, during which, based on the deployment plan, An executable code or configuration file is generated for the target neural network framework (e.g., TensorFlow, PyTorch, TensorRT). This configuration explicitly specifies on which processor each operation or group of operations should be executed.

[0155] 2. Performance measurement by running the configured model on a real heterogeneous platform. To obtain reliable results, the run is performed multiple times (100 to 1000 times) on the same input data. The first few runs are discarded to eliminate the impact of just-in-time compilation and other overhead. The end-to-end execution time is measured and averaged.

[0156] After this, the stage of selecting the final placement plan and adjusting the performance model (feedback) is carried out.

[0157] Final Plan Selection and Implementation Stage. This stage analyzes the verification results. Based on direct measurements, the final deployment plan that demonstrated the best performance (e.g., shortest execution time) is selected. Optionally, the measurement data can be used to adjust digital models in a feedback loop to improve the accuracy of future forecasts. The selected plan is then used for the final deployment of neural network operations on platform 120.

[0158] In more detail, the following steps are carried out.

[0159] Step 260. Selecting the most efficient operation placement plan. To do this, among all tested candidate plans The plan that demonstrated the shortest average execution time in real-world testing is selected. This plan is considered the final one and is used to deploy the neural network model on the heterogeneous platform 120.

[0160] Step 270. Adjusting the system performance model. In this step, the measurement results are used by module 115 to create a feedback loop. For each candidate the predicted time is compared and measured If a systematic discrepancy is observed (for example, the model underestimates the communication time on the PCIe bus by 30%), adjustments are made to the calculation formula. This self-correction mechanism allows the model to learn from real data, making its predictions more accurate for future allocation tasks.

[0161] Step 280. Distribute Operations According to the Selected Plan. The final step is to distribute computing operations among the components of the heterogeneous computing platform 120 using the most efficient placement plan selected in step 260.

[0162] Thus, the developed method allows us to determine and apply the most efficient computation distribution plans with high accuracy.

[0163] Specific example of implementation of the invention

[0164] To demonstrate the practical implementation of the developed method, a specific example of its application in a vehicle autopilot system is presented below. This example was chosen because it clearly illustrates the advantages of the proposed method under conditions of severe constraints on system response time and limited computing resources.

[0165] Initial data:

[0166] The autopilot system uses a neural network consisting of 120 ROS2 nodes (operations) interacting with each other via 636 links. ROS2 nodes perform various functions such as sensor data processing, route planning, vehicle control, and other autopilot operations.

[0167] Computation graph contains:

[0168] – 120 vertices (ROS2 nodes), each with characteristics: node ID (e.g. “perception_camera_node”), CPU load (from 0.01 to 3.5 units), RAM usage (from 10 to 2000 MB);

[0169] – 636 edges, each with a characteristic: data transfer intensity (from 0.5 to 150 Mbps).

[0170] When converted to a generalized weighted directed graph with parallel edge merging, the number of edges is reduced to 412, and the total volume of data circulating in the network is 4115.646 Mbps.

[0171] The heterogeneous computing platform 120 is an on-board computing complex of a vehicle, which includes 10 computers of three different types:

[0172] 4 high-performance processors:

[0173] CPU: 9.6 units each, performance ratio 5.39;

[0174] RAM: 8192 MB each.

[0175] 4 mid-range processors:

[0176] CPU: 7.0 units each, performance ratio 3.19;

[0177] RAM: 8192 MB each.

[0178] 2 additional processors:

[0179] CPU: 16.8 units each, performance ratio 1.1;

[0180] RAM: 8192 MB each.

[0181] The computers are connected by a high-speed network with a bandwidth of 1000 Mbps.

[0182] Platform graph contains:

[0183] - 10 calculators with the above characteristics;

[0184] - Edges between vertices with parameters: throughput: 1000 Mbps, delay: in the range of 0.1-1.0 ms.

[0185] Implementation of the method.

[0186] Step 200. Building the neural network computation graph. Building the computation graph. Based on an analysis of the ROS2 node system, the following are extracted from the configuration files:

[0187] - list of nodes (from nodes.json file);

[0188] - information about publications and subscriptions (from the publications.json and subscriptions.json files);

[0189] - CPU and RAM load metrics for each node.

[0190] Step 210. Building the computing platform graph. The platform graph is being built in parallel. based on information from files:

[0191] - hw_architecture.json file - to define limits on computing nodes;

[0192] - devices.json file - for building the topology of the computing network.

[0193] Step 220. Generate a system performance model. To assess the quality of the distribution, a system performance model is generated, represented by an objective function. In this example, it is expressed in terms of predicted traffic and resource load: ,

[0194] where:

[0195] – the predicted maximum data traffic passing through the network infrastructure;

[0196] – predicted average CPU load on computers;

[0197] - weighting coefficients reflecting optimization priorities.

[0198] In this example, the coefficients selected are =0.8 and =0.2, which reflects the priority of minimizing network traffic over uniform loading of processors.

[0199] Step 230. Determining candidate layout plans using a multi-level optimization algorithm. To determine the layout plan, a multi-level algorithm according to the second embodiment of the invention is used, implemented as follows:

[0200] 1. Reduction stage:

[0201] – Directed Multigraph Transformation into an undirected graph ;

[0202] – search in the graph fully connected groups of operations (clicks) with maximum total weight for internal links;

[0203] – for the current example, 7 fully connected K5 groups (of 5 vertices) and 40 fully connected K4 groups were found;

[0204] – internal traffic in each click is about 190 Mbps.

[0205] 2. Iterative distribution:

[0206] – a fully connected group of operations is found for the next calculator;

[0207] – the specified fully connected group of operations is placed on the computer, becoming the core of the cluster (crystallization center);

[0208] – peripheral (adjacent) operations are sequentially added to the cluster core, maximizing internal traffic;

[0209] – the process continues until the computer resources or peripheral operations are exhausted;

[0210] – posted transactions are excluded from the graph ;

[0211] – the process is repeated for the next calculator.

[0212] 3. Local optimization:

[0213] – After the initial placement of all operations, the local search algorithm is launched;

[0214] – permutations of operations between calculators are considered;

[0215] – special attention is paid to pairwise permutations, when operations are swapped;

[0216] – The placement plan with the best value predicted by the system performance model is saved.

[0217] As a result, 5 candidate plans are formed with various options for placing fully connected groups of operations and strategies for creating clusters.

[0218] Step 250. Conduct tests on the computing platform. For each of the 5 candidate layouts { }:

[0219] – a configuration file is generated for ROS2, which determines the placement of operations on the computers;

[0220] – the configuration is deployed on a test bench similar to the target platform;

[0221] – the full cycle of the autopilot system operation is launched with a standard set of input data.

[0222] Measured:

[0223] – end-to-end processing time of data from sensors to control commands;

[0224] – loading of each computer by CPU and RAM;

[0225] – the volume of traffic passing through the network infrastructure.

[0226] 10 runs are performed for each configuration, the first 2 runs are excluded from statistics.

[0227] Step 260. Selecting the highest-performance layout. The test results yielded the average values ​​presented in Table 1.

[0228] Table 1

[0229] Plan Max external traffic Average CPU load Processing time M1 950 Mbps 68% 120 ms M2 875 Mbps 72% 110 ms M3 820 Mbps 75% 105 ms M4 790 Mbps 78% 98 ms M5 805 Mbps 79% 102 ms

[0230] Based on the measured processing time, the M4 plan is selected as the one that provides the shortest time (98 ms), which is a critical parameter for the autopilot system.

[0231] Step 270. Adjusting the system performance model. Comparison of the performance model's predicted values ​​with the actual measured ones showed that:

[0232] – the model underestimated the influence of inter-vertex communications by 15-20%;

[0233] – The estimated performance of some computers for certain types of operations was overestimated by 10%.

[0234] Based on this data, adjustments are made to the platform graph parameters:

[0235] – the latency of communication channels increases by 18%;

[0236] – Adjusts performance factors for image processing related operation types.

[0237] Results of applying the highest performing deployment plan.

[0238] The optimal plan M4 obtained using the developed method provided:

[0239] 1. Reduce network traffic:

[0240] – initial total traffic: 4115.646 Mbps;

[0241] – internal traffic (not passing through the network): 3325.646 Mbps (79.3%);

[0242] – external traffic (passing through the network): 850 Mbps (20.7%);

[0243] – The load on the network infrastructure has been reduced by 15% compared to the original distribution.

[0244] 2. Efficient use of computing resources:

[0245] – Average CPU load: 82% (improved by 10%);

[0246] – maximum load of a separate calculator: 95% (versus 98%);

[0247] – More uniform load distribution: the standard deviation of the load has decreased from 20% to 14%.

[0248] 3. Reducing data processing time:

[0249] –original time: 160ms;

[0250] – time with optimal plan: 115 ms;

[0251] – improvement: 28.1%.

[0252] Thus, the application of the developed method made it possible to significantly increase the efficiency of using the computing resources of a heterogeneous platform, which resulted in a significant reduction in the data processing time in the autopilot system.

[0253] Thanks to this, a technical result was achieved consisting in increasing the efficiency of using the computing resources of a heterogeneous platform when performing computing operations, expressed in a reduction in the overall execution time of calculations (decreased latency), a reduction in the load on communication channels between the platform's computers and ensuring adaptability to the actual characteristics of the equipment by distributing operations between different types of computers, taking into account their actual characteristics and the intensity of data exchange between operations or nodes.

Claims

1. A method for distributing computing operations between components of a heterogeneous computing platform, where the computing operations correspond to neural networks, individual operations of neural networks, or individual algorithms, wherein the method includes steps during the execution of which the following is carried out: – formation of digital models, including: – a computation graph where the vertices correspond to computational operations and the edges correspond to data dependencies between them; – a graph of a computing platform, in which the vertices correspond to the platform’s computers with an indication of their characteristics, and the edges correspond to the communication channels with an indication of their characteristics; – a system performance model that predicts the total execution time of a computation graph in the form of computational costs for performing operations on computers of a heterogeneous platform and communication costs for transferring data between computers of a heterogeneous platform; – determination, based on the specified models, of candidate plans for placing operations on computers of a heterogeneous platform that have the best performance indicator, using a multi-level algorithm that includes: – reduction of the computation graph into its simplified representation by aggregating groups of related operations; – determination of the initial layout plan for the specified simplified representation; – restoration of a detailed placement plan for the original computation graph with its subsequent local optimization; – verification of selected candidate placement plans by testing them on a heterogeneous computing platform with measurement of actual execution time; – selecting the most productive deployment plan based on test results or adjusting the performance model to improve forecasting accuracy; – distribution of computing operations between components of a heterogeneous computing platform using the selected most productive plan for placing operations on the computers of the heterogeneous platform.

2. The method according to claim 1, in which the reduction includes identifying in the computation graph a subgraph with high internal connectivity for the subsequent formation of a single composite vertex from this subgraph.

3. The method according to paragraph 2, wherein identifying a subgraph with high internal connectivity comprises: – transformation of the original directed computational multigraph into a weighted undirected graph, where the edge weight reflects the total intensity of data exchange between operations in both directions; – search in the resulting undirected graph for a fully connected group of operations with the maximum total weight of internal connections.

4. The method according to paragraph 3, in the course of which, after the identification of the said fully connected group of operations, it is used as the core of the cluster, to which peripheral operations are iteratively added, maximizing the internal traffic of the cluster, until all connected operations are exhausted or the maximum characteristics of the computer in terms of computing resources are reached.

5. The method according to paragraph 4, during the execution of which a separate cluster of operations is formed for each computer from the graph of the computing platform, excluding already placed operations from consideration when forming subsequent clusters.

6. The method according to paragraph 5, in which the stage of local re-optimization during the restoration of the detailed placement plan includes enumerating the permutations of individual operations between already formed clusters to find a distribution that provides the best predicted performance indicator.

7. The method according to paragraph 6, in which local additional optimization includes pairwise permutations of operations between computers, in which pairs of operations are simultaneously moved in opposite directions to comply with resource constraints.

8. The method according to paragraph 1, in which the testing of the selected candidate mappings includes: – automatic generation of executable code or a configuration file for the target framework, explicitly defining the physical device for performing each operation; – multiple launches of the configured model on a real platform with discarding the initial launches to stabilize the measurements; – averaging the measured end-to-end execution time.

9. The method according to claim 1, wherein adjusting the performance model includes comparing the predicted execution time with the actually measured one and adjusting the parameters of the computing platform graph accordingly.

10. The method according to claim 1, wherein various types of processors are used as calculators in the heterogeneous computing platform, including central processing units (CPUs), graphic processing units (GPUs), neural processing units (NPUs), field programmable gate arrays (FPGAs), as well as specialized image processing accelerators (ISPs), digital signal processors (DSPs), and microcontrollers (MCUs).

11. The method according to claim 1, wherein the computational operations are ROS2 nodes and the computational graph is a ROS2 graph used in vehicle autopilot systems.

Images