A data processing method and related apparatus
By analyzing and evaluating the program code and selectively inlining functions, the problem of graph splitting in the compilation process of dynamic language computation graphs was solved, improving the efficiency of computation graph generation and overall performance.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- HUAWEI TECH CO LTD
- Filing Date
- 2025-01-07
- Publication Date
- 2026-07-07
Smart Images
Figure CN122346313A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of computers, and more particularly to a data processing method and related apparatus. Background Technology
[0002] In the field of artificial intelligence (AI), a computation graph is a data structure used to represent the computational tasks and data flow processes of an application. By optimizing execution paths and reducing redundant computations, the execution efficiency of a program can be significantly improved. Just-In-Time (JIT) compilation technology can compile program code from dynamic languages such as Python into computation graphs, accelerating the operation of AI models and improving overall efficiency.
[0003] However, dynamic languages contain computational operations that cannot be directly expressed as computation graphs, such as nondeterministic operations (e.g., random number generation), dynamically generated functions, and constants bound to functions. These factors can lead to graph splitting during compilation, a phenomenon known as graph splitting. Optimizing the performance degradation caused by graph splitting has become a pressing issue. Summary of the Invention
[0004] In a first aspect, embodiments of this application provide a data processing method. This method is executed by a data processing device, or by some components (e.g., a processor, chip, or chip system) within the data processing device, or it can be implemented by a logic module or software capable of performing all or part of the functions of the data processing device. In this first aspect and its possible implementations, the method is described as being executed by a data processing device. The method includes:
[0005] The program code is analyzed to obtain the first analysis result, which is used to indicate the performance acceleration effect of the program code. The program code includes a first function body and a second function body, wherein the second function body is the function called in the first function body;
[0006] If the performance acceleration effect meets the first condition, then the first function body and the second function body are inlined.
[0007] In this application, the first function body is the function body of the main function, and the second function body is the function body of the sub-functions. A main function can call multiple sub-functions. By analyzing the performance acceleration effect (or performance improvement effect) in the program code, and under the condition that a preset first condition is met, inlining the main function and sub-functions can expand the coverage of the program code during the computation graph transformation process. By compiling a larger computation graph, the number of switching operations during computation graph execution is reduced, the network training time is shortened, and the overall execution efficiency is improved.
[0008] In one possible implementation, the performance acceleration effect is the performance acceleration effect after inlining the first function body and the second function body.
[0009] In one possible implementation, the performance acceleration effect satisfies the first condition, including:
[0010] The performance acceleration effect is greater than the first threshold.
[0011] This application aims to compare the performance acceleration effects of function inlining versus not inlining through code analysis. Specifically, if the performance acceleration effect of function inlining exceeds a preset first threshold (i.e., exceeds the performance acceleration effect of not inlining), then function inlining is selected.
[0012] In one possible implementation, the difference in performance loss between the two is compared, and the solution that results in lower loss is selected as the final optimization strategy.
[0013] In one possible implementation, the performance acceleration effect includes the performance acceleration effect of the first computation graph, which is the computation graph obtained by transforming the program code after function inlining.
[0014] In this application, the performance acceleration mainly considers the performance acceleration effect generated after the program code is compiled and transformed into a computation graph through JIT technology.
[0015] In one possible implementation, the first analysis result is determined based on code snippets in the program code that cause performance loss.
[0016] In one possible implementation, the code segments that cause performance loss include: split graph points and guard nodes. Split graph points are code segments in the program code that cannot be converted into a computation graph, and guard nodes are code segments in the program code used to verify the correctness of the program.
[0017] In one possible implementation, the performance loss caused by the split points is determined based on the number of split points.
[0018] Specifically, when a function contains a small number of split graph points, without inlining, each function will be broken into multiple independent fragments during the compilation and transformation into a computation graph. These fragments are compiled and transformed separately, resulting in significant gaps between the final generated computation graphs, which negatively impacts overall performance. Using function inlining effectively reduces the generation of split graphs, thereby reducing these gaps and improving overall performance.
[0019] In one possible implementation, the performance overhead caused by the guard node is determined based on one or more of the following:
[0020] The number of guard nodes;
[0021] The type of the guard node, or;
[0022] The probability of failure to defend a node.
[0023] In this application, the probability of guard failure is determined based on runtime data statistics (profiling information), assesses the compilation overhead caused by guard failure, and predicts the probability of guard failure again.
[0024] Optionally, different coefficients can be assigned to the number, type, and failure probability of guard nodes to calculate the potential performance overhead of inlining. This takes into account both the overhead of recompiling the computation graph and the potential impact of inlining subfunctions.
[0025] In one possible implementation, the method further includes:
[0026] If the performance acceleration effect does not meet the first condition, then no function inlining will be performed on the first function body and the second function body.
[0027] In one possible implementation, the performance acceleration effect does not satisfy the first condition, including:
[0028] The performance acceleration effect is less than or equal to the first threshold.
[0029] In this application, if the predicted performance acceleration effect does not meet the first condition, no function inlining is performed, and the first function body and the second function body are directly compiled and converted into computation graphs respectively.
[0030] Secondly, embodiments of this application provide a data processing apparatus, the apparatus comprising:
[0031] The analysis module is used to perform code analysis on the program code and obtain the first analysis result. The first analysis result is used to indicate the performance acceleration effect of the program code. The program code includes a first function body and a second function body, wherein the second function body is the function called in the first function body.
[0032] The processing module is used to perform function inlining on the first function body and the second function body when the performance acceleration effect meets the first condition.
[0033] In one possible implementation, the performance acceleration effect is the performance acceleration effect after inlining the first function body and the second function body.
[0034] In one possible implementation, the performance acceleration effect satisfies the first condition, including:
[0035] The performance acceleration effect is greater than the first threshold.
[0036] In one possible implementation, the performance acceleration effect includes the performance acceleration effect of the first computation graph, which is the computation graph obtained by transforming the program code after function inlining.
[0037] In one possible implementation, the first analysis result is determined based on code snippets in the program code that cause performance loss.
[0038] In one possible implementation, the code segments that cause performance loss include: split graph points and guard nodes. Split graph points are code segments in the program code that cannot be converted into a computation graph, and guard nodes are code segments in the program code used to verify the correctness of the program.
[0039] In one possible implementation, the performance loss caused by the split points is determined based on the number of split points.
[0040] In one possible implementation, the performance overhead caused by the guard node is determined based on one or more of the following:
[0041] The number of guard nodes;
[0042] The type of the guard node, or;
[0043] The probability of failure to defend a node.
[0044] In one possible implementation, the processing module is also used for:
[0045] If the performance acceleration effect does not meet the first condition, no function inlining is performed on the first function body and the second function body.
[0046] In one possible implementation, the performance acceleration effect does not meet the first condition, including: the performance acceleration effect is less than or equal to the first threshold.
[0047] Thirdly, embodiments of this application provide a computing device, including: a processor, a memory, and a transceiver. The memory stores computer programs or computer instructions, and the processor is used to call and run the computer programs or computer instructions stored in the memory, causing the processor to perform processing operations as described in the first aspect and any implementation thereof. The transceiver is used to transmit and receive signals, such as implementing the receiving and sending operations as described in the first aspect and any implementation thereof.
[0048] Fourthly, embodiments of this application provide a computing device cluster, including at least one computing device, each computing device including a processor and a memory, the memory storing computer programs or computer instructions, the processor being used to call and run the computer programs or computer instructions stored in the memory, so that the computing device cluster performs the first aspect and any of its optional methods described above.
[0049] Fifthly, embodiments of this application provide a computer-readable storage medium storing a computer program that, when run on a computer, causes the computer to perform the methods described in the first aspect and any of its optional methods.
[0050] Sixthly, embodiments of this application provide a computer program that, when run on a computer, causes the computer to perform the first aspect and any of its optional methods described above.
[0051] In a seventh aspect, this application provides a chip system including a processor for supporting an execution device or training device in implementing the functions involved in the foregoing aspects, such as transmitting or processing data involved in the foregoing methods; or, information. In one possible design, the chip system further includes a memory for storing program instructions and data necessary for the execution device or training device. This chip system may be composed of chips or may include chips and other discrete devices.
[0052] The technical effects of the second, third, fourth, fifth, and sixth aspects of this application can be understood in conjunction with the technical effects of the first aspect and any implementation thereof. Attached Figure Description
[0053] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0054] Figure 1A structural diagram illustrating the main framework of artificial intelligence;
[0055] Figure 2 A schematic diagram of a system processing flow provided in an embodiment of this application;
[0056] Figure 3 A schematic flowchart of a data processing method provided in an embodiment of this application;
[0057] Figure 4 A schematic diagram of function inlining provided in an embodiment of this application;
[0058] Figure 5 Another schematic diagram of function inlining provided in the embodiments of this application;
[0059] Figure 6 Another schematic diagram of the data processing method provided in the embodiments of this application;
[0060] Figure 7 This is a schematic diagram of the structure of a data processing device provided in an embodiment of this application;
[0061] Figure 8 A schematic diagram of the structure of a computing device provided in an embodiment of this application;
[0062] Figure 9 This is a schematic diagram of the structure of a computing device cluster provided in an embodiment of this application;
[0063] Figure 10 This is another schematic diagram of the structure of a computing device cluster provided in an embodiment of this application;
[0064] Figure 11 This is a schematic diagram of the structure of a chip provided in an embodiment of this application. Detailed Implementation
[0065] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.
[0066] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a particular order or sequence. It should be understood that such data can be interchanged where appropriate so that embodiments of the application described herein can be implemented, for example, in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.
[0067] First, the overall workflow of the artificial intelligence system is described; please refer to [link / reference]. Figure 1 , Figure 1 This is a structural diagram illustrating the main framework of artificial intelligence. The following explanation of the AI framework is based on two dimensions: the "Intelligent Information Chain" (horizontal axis) and the "IT Value Chain" (vertical axis). The "Intelligent Information Chain" reflects a series of processes from data acquisition to processing. For example, it could be the general process of intelligent information perception, intelligent information representation and formation, intelligent reasoning, intelligent decision-making, and intelligent execution and output. In this process, data undergoes a condensation process of "data—information—knowledge—wisdom." The "IT Value Chain" reflects the value that artificial intelligence brings to the information technology industry, from the underlying infrastructure of human intelligence and information (provided and processed by technology) to the industrial ecosystem of the system.
[0068] (1) Infrastructure
[0069] Infrastructure provides computing power to support artificial intelligence systems, enabling communication with the external world and providing support through a basic platform. This communication occurs through sensors; computing power is provided by intelligent chips (hardware acceleration chips such as CPUs, NPUs, GPUs, ASICs, and FPGAs); and the basic platform includes distributed computing frameworks and related platform guarantees and support, which may include cloud storage and computing, interconnected networks, etc. For example, sensors communicate with the outside world to acquire data, and this data is provided to intelligent chips in the distributed computing system provided by the basic platform for computation.
[0070] (2) Data
[0071] The data at the next layer of infrastructure is used to represent the data sources in the field of artificial intelligence. The data involves graphics, images, voice, text, and IoT data from traditional devices, including business data from existing systems and sensor data such as force, displacement, liquid level, temperature, and humidity.
[0072] (3) Data processing
[0073] Data processing typically includes methods such as data training, machine learning, deep learning, search, reasoning, and decision-making.
[0074] Among them, machine learning and deep learning can perform intelligent information modeling, extraction, preprocessing, and training on data, including symbolization and formalization.
[0075] Reasoning refers to the process in which, in a computer or intelligent system, the machine thinks and solves problems by simulating human intelligent reasoning, based on reasoning control strategies and using formalized information. Typical functions include search and matching.
[0076] Decision-making refers to the process of making decisions based on intelligent information after reasoning, and it typically provides functions such as classification, sorting, and prediction.
[0077] (4) General ability
[0078] After the data processing mentioned above, the results of the data processing can be used to form some general capabilities, such as algorithms or a general system, for example, translation, text analysis, computer vision processing, speech recognition, image recognition, etc.
[0079] (5) Smart Products and Industry Applications
[0080] Intelligent products and industry applications refer to products and applications of artificial intelligence systems in various fields. They are the encapsulation of overall artificial intelligence solutions, productizing intelligent information decision-making and realizing practical applications. Their application areas mainly include: intelligent terminals, intelligent transportation, intelligent healthcare, autonomous driving, smart cities, etc.
[0081] This application can be applied, but is not limited to, scenarios involving the generation of computational graphs in the field of artificial intelligence, specifically in areas such as compiling dynamic languages into computational graphs. Please refer to [link / reference]. Figure 2 , Figure 2 This is a schematic diagram of a process for generating a computational graph using a data processing device, such as... Figure 2 As shown, the data processing device 100 includes an analysis module 101 and a processing module 102. The analysis module 101 is responsible for analyzing and predicting the input program code, while the processing module 102 processes the program code based on the analysis results to generate optimized program code. Then, the processed program code is compiled into a computation graph using JIT technology, which can provide acceleration services for AI networks such as intelligent driving and scientific computing, reducing training time.
[0082] Optionally, the data processing device 100 may also provide a client to the outside world, so as to interact with users (such as developers). The client may be, for example, a web browser provided by the data processing device 100; or, the client may be an application running on a user terminal.
[0083] Optionally, the data processing device 100 also includes an acquisition module (not shown in the figure) for acquiring program code input by the user, as well as various parameters set by the user, such as threshold values and calculation coefficients.
[0084] As examples, the data processing device 100 can be deployed in the cloud to provide users with cloud services for generating computational graphs. In this case, the data processing device 100 may be implemented by computing devices or clusters of computing devices in the cloud. Alternatively, the data processing device 100 can be deployed locally to provide users with local computational graph generation services.
[0085] In practical applications, the aforementioned data processing device 100 can be implemented through software or through hardware.
[0086] Data processing device 100, as an example of a software functional unit, may include code running on a computing instance. The computing instance may include at least one of a host, a virtual machine, and a container. Further, the aforementioned computing instance may be one or more. For example, data processing device 100 may include code running on multiple hosts / virtual machines / containers. It should be noted that the multiple hosts / virtual machines / containers used to run the code may be distributed in the same region or in different regions. Further, the multiple hosts / virtual machines / containers used to run the code may be distributed in the same availability zone (AZ) or in different AZs, each AZ including one or more geographically proximate data centers. Typically, a region may include multiple AZs.
[0087] Similarly, multiple hosts / virtual machines / containers used to run this code can be distributed within the same Virtual Private Cloud (VPC) or across multiple VPCs. Typically, a VPC is set up within a region. Communication between two VPCs within the same region, as well as between VPCs in different regions, requires a communication gateway to be set up within each VPC to enable interconnection between VPCs.
[0088] As an example of a hardware functional unit, the data processing device 100 may include at least one computing device, such as a server. Alternatively, the data processing device 100 may be implemented using a central processing unit (CPU), an application-specific integrated circuit (ASIC), or a programmable logic device (PLD). The PLD may be a complex programmable logical device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), a data processing unit (DPU), a neural network processing unit (NPU), a system-on-chip (SoC), an offload card, an accelerator card, or any combination thereof.
[0089] The data processing device 100 includes multiple computing devices that can be distributed within the same region or in different regions. Similarly, the multiple computing devices included in the data processing device 100 can be distributed within the same Availability Zone (AZ) or in different AZs. Likewise, the multiple computing devices included in the data processing device 100 can be distributed within the same Virtual Private Cloud (VPC) or in multiple VPCs. These multiple computing devices can be any combination of computing devices such as servers, ASICs, PLDs, CPLDs, FPGAs, GALs, DPUs, NPUs, SoCs, offloading cards, and accelerator cards.
[0090] To facilitate understanding, the relevant terms and concepts involved in the embodiments of this application will be introduced below.
[0091] (1) Computation graph
[0092] A computation graph is a structure used to represent and compute mathematical expressions. It constructs a directed acyclic graph by breaking down complex computational processes into a series of basic computational steps and explicitly defining the dependencies between these steps. This structured representation makes the computation process clearer and more efficient. In the field of AI, computation graphs are often used to represent the forward and backward propagation processes of neural networks. By explicitly representing each layer, each node, and the connections between them in the computation graph, computational tasks can be performed more efficiently.
[0093] (2) Just-In-Time (JIT) compilation
[0094] Just-in-Time (JIT) compilation, also known as real-time compilation, is a method for executing computer code. It involves converting bytecode (or intermediate code) into efficient machine code on the fly during program execution. In deep learning and high-performance computing, the technique of compiling dynamic languages like Python into computational graphs is often referred to as "dynamic-to-static" conversion. JIT technology staticizes the execution flow of dynamic languages, transforming the code into a computational graph representation, thereby improving performance.
[0095] (3) Guard
[0096] A guard, also known as a guard node, is a code segment specifically designed to verify the correctness of a program. In programming, guards are typically used to perform condition checks before code execution to ensure the program runs under safe conditions. If the conditions are not met, the guard will prevent code execution or take other measures to prevent errors from occurring.
[0097] The types of guards include variable type checking (such as checking whether a value is an integer or a boolean) and value checking (such as checking whether the value of a variable is empty).
[0098] (4) Crack diagram
[0099] A split graph refers to a scenario where, during the compilation or conversion of high-level programming language (such as Python or TensorFlow code) into a computation graph, compilation cannot continue or the graph cannot be fully and accurately represented for various reasons. These reasons may include complex control flow structures (such as loops, conditional statements, etc.), nondeterministic operations (such as random number generation), dynamically generated functions, and constants bound to functions. These structures or characteristics are difficult to represent in a static computation graph, thus leading to a split graph.
[0100] Specifically, the code snippet in the program code that causes the cracked graph is called the crack point.
[0101] (5) Function inline
[0102] Function inlining is a compiler optimization technique that replaces function calls with the function body itself, thereby eliminating the overhead of function calls (such as parameter passing, stack frame creation and destruction, etc.) and improving program execution efficiency.
[0103] In the current AI field, Python, a dynamic language, has become the mainstream AI development language due to its ease of use. Existing technologies typically compile Python code into computational graphs specifically designed for AI (such as torch.fx.graph) to improve the performance of AI networks. In practical applications, such as autonomous driving and scientific computing, compiling the entire network into a computational graph can significantly accelerate computation. However, not all dynamic language scenarios can be successfully converted to native code, especially those involving frequently changing control flow and unoptimizable code segments. These scenarios may lead to split graphs and gaps.
[0104] Excessive gaps can lead to frequent switching between dynamic and static execution modes, thus impacting overall performance. To optimize performance, dynamic languages typically employ inlining techniques, merging small functions or code blocks into larger, optimized blocks. However, if gaps exist during inlining, these gaps will not only not be eliminated but may actually increase due to inlining, even being copied multiple times within loops, resulting in unnecessary performance overhead. Therefore, inlining requires certain information to aid in decision-making.
[0105] Based on this, this application provides a data processing method, such as... Figure 3 As shown, the data processing method provided in this application embodiment includes the following steps 301-304. This method can be applied to the above... Figure 2 The data processing apparatus 100 shown may also be applied to other suitable apparatuses. The following describes an application... Figure 2 The data processing device 100 shown is used as an example for explanation.
[0106] 301. Perform code analysis on the program code to obtain the first analysis result. The first analysis result is used to indicate the performance acceleration effect of the program code. The program code includes a first function body and a second function body, wherein the second function body is the function called in the first function body.
[0107] The program code involves sub-function calls, with the second function body being a sub-function called by the first function body. The first analysis result is a predictive conclusion derived from analyzing code segments within the program code that may cause performance degradation. This first analysis result indicates the expected final performance acceleration (or performance improvement) after converting the program code into a computational graph and considering these performance-consuming code segments.
[0108] It is understandable that the first function body is the function body of the main function, and the second function body is the function body of the sub-functions. A main function can call multiple sub-functions. For example, in the "image classification prediction" method in the field of image processing, it may contain multiple sub-functions such as "loading preprocessed images," "model prediction," and "obtaining category labels." This application does not impose specific restrictions on the number of main functions and the number of sub-functions they call.
[0109] In this application, the performance acceleration mainly considers the performance acceleration effect generated after the program code is compiled and transformed into a computation graph through JIT technology.
[0110] The core focus of this application is to compare the performance acceleration effects of the following two processing schemes:
[0111] First, the main function (first function body) and the sub-functions called within it (second function body) are inlined, and then the whole thing is transformed into a computation graph.
[0112] Second, instead of inlining the functions, the first and second function bodies are directly converted into computation graphs.
[0113] The objective of this application is to evaluate these two approaches to determine which offers a better performance acceleration and to make a selection accordingly. Specifically, code snippets that may lead to performance degradation include split graph nodes and guard nodes, which will be fully considered during the evaluation process.
[0114] Please see Figure 4 , Figure 4 This scenario involves functions containing split points. The program code to be processed contains three function bodies: the main function Fun A, and two sub-functions Fun B and Fun C. Fun A calls Fun B, and Fun B further calls Fun C.
[0115] A computation graph is a graph structure used to represent computational processes and data dependencies in a program. When constructing a computation graph independently, such as constructing the computation graph of Fun A, even if Fun A itself does not contain any splitting points, the computation graph of Fun A will still be split due to the call relationship between Fun A and Fun B. In other words, the integrity of the computation graph of Fun A is affected by the splitting characteristic of the called function Fun B.
[0116] Specifically, the performance loss caused by split points is determined based on the number of split points. When there are fewer split points in the function (e.g., ... Figure 4In the example, Fun B and Fun C each contain only one split point. Without inlining, each function cannot directly compile the code segment containing the split point when it is compiled into a computation graph; instead, each function will be divided into multiple segments (e.g., ...). Figure 4 The three functions in the code will ultimately generate seven computation graphs: Fun A is divided into A1 and A2, Fun B into B1, B2 and B3, and Fun C into C1 and C2. These segmented fragments need to be compiled and converted into computation graphs independently, resulting in a large number of discontinuous regions between the final generated computation graphs. These gaps will affect the overall performance of the system.
[0117] In comparison, if for Figure 4 If the program code in the code is inlined, the inlined function body will significantly reduce the number of fragments. Specifically, the inlined code may be divided into only three main parts (such as...). Figure 4 The processed program code includes fragment 1 (merging parts of A1 and B1), fragment 2 (merging parts of B2 and C1), and fragment 3 (merging parts of C2, B3, and A2). This way, only three computation graphs need to be generated in total, reducing gaps compared to solutions that do not perform function inlining, thus helping to improve overall performance.
[0118] It is important to emphasize that when a called subfunction contains a single split graph point, the speedup effect of compiling into a computational graph begins to decrease. When there are many split graph points in the subfunction (e.g., three or more), the overall performance deteriorates excessively, and the graph construction of the subfunction should be abandoned. For example, if Fun A calls Fun B, which contains three split graph points, the graph construction of Fun B should be abandoned. Only Fun A should be compiled into computational graphs A1 and A2 (because Fun A contains the split graph point that calls Fun B), while Fun B should only be compiled and executed using dynamic language without computational graph acceleration.
[0119] Furthermore, when the main function repeatedly calls sub-functions containing split map points, inlining may actually increase the gaps, leading to performance degradation. Therefore, before inlining functions, it is necessary to carefully analyze the gaps between before and after inlining to choose the more efficient approach.
[0120] Please see Figure 5 , Figure 5 This scenario involves a function containing a guard node. The program code to be processed contains two function bodies: the main function Fun D, and the sub-function Fun E. Fun D contains a loop function that calls Fun E a total of N times. Fun E contains a guard node. If... Figure 5If the program code in the code is inlined, the inlined function body will contain N guard nodes.
[0121] Specifically, the performance loss caused by guard nodes is determined based on the number and type of guard nodes (such as "variable type checking" and "value checking") and the probability of guard failure. The average impact of each guard node on program performance is estimated, and the performance acceleration effect after conversion into a computation graph is predicted.
[0122] The role of a guard node in a program is to check conditions and decide whether to execute a specific code block based on those conditions. If a guard node fails (i.e., the condition is not met), the relevant code block will not be executed. This means that if the code block is the target of AI accelerator acceleration, this part of the code will not be accelerated due to the guard's failure.
[0123] The probability of guard failure is determined based on runtime data statistics (profiling information). This assesses the compilation overhead caused by guard failure and predicts the probability of subsequent guard failures (e.g., for guard nodes that repeatedly check if a variable value is 1, 2, or 3). The overhead of recompiling the computation graph and the impact of inlining subfunctions are considered to determine whether they should participate in graph construction. For example, if dynamically generated functions or constants bound to functions are encountered, abandoning inlining may be more appropriate.
[0124] It should be noted that "variable type checking" guard nodes typically have higher performance overhead than "value checking" guard nodes because they require accessing and processing type information, which involves additional memory access and computation.
[0125] The performance loss caused by the number, type, and failure probability of guard nodes varies. In specific analysis, different coefficients can be assigned to the number, type, and failure probability of guard nodes to calculate the performance loss that may be caused by function inlining. This performance loss is used to evaluate the performance acceleration effect obtained by compiling the program code after function inlining into a computation graph.
[0126] In real-world applications, program code often contains both split points and guard nodes, which need to be considered and optimized as a whole to ensure optimal program performance.
[0127] Specifically, the estimation of performance acceleration includes the following three calculation methods:
[0128] (1) Cumulative evaluation of guarded nodes:
[0129] An estimated performance value for the guard is obtained by summing the number of guard nodes multiplied by a type coefficient. Whether to perform function inlining is determined based on a fixed threshold (e.g., a performance estimation threshold of 1) and a running percentage threshold (e.g., 5%).
[0130] (2) Assessment of the probability of defense failure:
[0131] By using the ratio of failures to executions, the percentage of guard failures is derived from runtime data statistics. When a guard fails, whether to perform function inlining is determined based on a fixed value (e.g., setting the number of function object guard failures to 1) or a percentage threshold (e.g., setting the overall guard failure percentage to 50%).
[0132] (3) Evaluation of the number of split points:
[0133] When the number of split points is less than three, consider inlining the function; otherwise, abandon inlining the function with the most split points, and may even abandon graph construction processing for it (i.e. abandon compiling the function into a computation graph).
[0134] For the program code of AI networks, the above decisions can be enumerated in a finite number. In practical applications, the thresholds are dynamically adjusted based on the specific AI network, and the final performance acceleration is calculated by combining these factors.
[0135] For example, if a subfunction has a relatively small number of split graph points (only one), but contains multiple guard nodes, or if the failure probability of the guard nodes is high, these factors will increase the performance penalty after inlining. Quantitative calculations are used to determine whether inlining leads to performance degradation. If the calculation results show that the performance penalty after inlining is higher than the penalty without inlining (or a preset threshold), then function inlining is not performed. The calculated performance penalty may be used to evaluate the performance acceleration effect obtained after compiling the program code into a computational graph.
[0136] 302. Determine whether the performance acceleration effect meets the first condition.
[0137] Specifically, the performance acceleration effect indicated by the first analysis result satisfies a first condition, including that the performance acceleration effect is greater than a first threshold, i.e., whether the performance acceleration effect after inlining is higher than the performance acceleration effect without inlining; or whether the performance loss caused by inlining is lower than the performance loss caused by not inlining. Here, the performance acceleration effect indicated by the first analysis result is the calculated performance acceleration effect produced by "compiling and converting the functions into a computation graph after inlining the functions"; the first threshold can be the calculated performance acceleration effect produced by "converting the functions into computation graphs separately without inlining the functions", or the first threshold can be a preset empirical value.
[0138] If the first condition is met, proceed to step 303; otherwise, proceed to step 304.
[0139] Please see Figure 6 , Figure 6 This is a flowchart illustrating the data processing method provided in this application embodiment. After obtaining the program code to be processed, split graph point evaluation and guard node evaluation are performed respectively. If both evaluations pass, the main function and sub-functions in the program code are inlined, and the inlined program code is compiled and converted into a computation graph.
[0140] For example, when the number of split points in the program code is less than three, evaluation is performed using split points.
[0141] For example, by quantization, if the performance loss caused by the guard node is less than the target threshold (e.g., the performance loss is smaller after function inlining), the performance is evaluated through the guard node.
[0142] Once both evaluations pass, the main function and sub-functions in the program code are inlined, and the inlined program code is compiled into a computational graph. If the evaluation of the split point or the guard node fails, the inlining process is skipped, and the main function and sub-functions are directly compiled into computational graphs respectively.
[0143] In one possible implementation, if the subfunction contains a large number of split graph points (e.g., three or more), the subfunction abandons graph construction and only performs dynamic language compilation and execution, without accelerating the computation graph.
[0144] 303. Perform function inlining on the first function body and the second function body.
[0145] When the performance acceleration effect indicated by the first analysis result meets the first condition, function inlining is performed.
[0146] 304. No inlining is performed between the first function body and the second function body.
[0147] If the performance acceleration effect indicated by the first analysis result does not meet the first condition, function inlining will not be performed.
[0148] In this application, a comprehensive evaluation of potential graph splitting points and guard nodes that may be caused by function inlining is used to determine whether to perform function inlining. This selective inlining strategy avoids repeated graph compilation, expands the coverage of the computation graph, reduces the number of switching operations during computation graph execution, and shortens network training time. The generated guard nodes and their runtime overhead are evaluated, and runtime guard information is collected and analyzed to determine whether a function is suitable for inlining. This reduces the graph switching overhead caused by dynamic function inlining, improving overall performance acceleration.
[0149] The methods provided in the embodiments of this application have been described in detail above. Next, the device for performing the above methods provided in the embodiments of this application will be described.
[0150] Please see Figure 7 , Figure 7 This is a schematic diagram of the structure of a data processing apparatus 700 provided in an embodiment of this application. Figure 7 As shown, the device includes:
[0151] Analysis module 701 is used to perform code analysis on program code and obtain a first analysis result. The first analysis result is used to indicate the performance acceleration effect of program code. Program code includes a first function body and a second function body, wherein the second function body is the function called in the first function body.
[0152] The processing module 702 is used to perform function inlining on the first function body and the second function body when the performance acceleration effect meets the first condition.
[0153] In one possible implementation, the performance acceleration effect is the performance acceleration effect after inlining the first function body and the second function body.
[0154] In one possible implementation, the performance acceleration effect satisfies the first condition, including:
[0155] The performance acceleration effect is greater than the first threshold.
[0156] In one possible implementation, the performance acceleration effect includes the performance acceleration effect of the first computation graph, which is the computation graph obtained by transforming the program code after function inlining.
[0157] In one possible implementation, the first analysis result is determined based on code snippets in the program code that cause performance loss.
[0158] In one possible implementation, the code segments that cause performance loss include: split graph points and guard nodes. Split graph points are code segments in the program code that cannot be converted into a computation graph, and guard nodes are code segments in the program code used to verify the correctness of the program.
[0159] In one possible implementation, the performance loss caused by the split points is determined based on the number of split points.
[0160] In one possible implementation, the performance overhead caused by the guard node is determined based on one or more of the following:
[0161] The number of guard nodes;
[0162] The type of the guard node, or;
[0163] The probability of failure to defend a node.
[0164] In one possible implementation, the processing module 702 is further configured to:
[0165] If the performance acceleration effect does not meet the first condition, no function inlining is performed on the first function body and the second function body.
[0166] In one possible implementation, the performance acceleration effect does not meet the first condition, including: the performance acceleration effect is less than or equal to the first threshold.
[0167] This application also provides a computing device 100. For example... Figure 8 As shown, the computing device 100 includes a bus 102, a processor 104, a memory 106, and a communication interface 108. The processor 104, the memory 106, and the communication interface 108 communicate with each other via the bus 102. The computing device 100 can be a server or a terminal device. It should be understood that this application does not limit the number of processors and memories in the computing device 100.
[0168] Bus 102 can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of representation, Figure 8 The bus 104 may be represented by a single line, but this does not mean that there is only one bus or one type of bus. The bus 104 may include a path for transmitting information between various components of the computing device 100 (e.g., memory 106, processor 104, communication interface 108).
[0169] The processor 104 may include any one or more processors such as a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP).
[0170] Memory 106 may include volatile memory, such as random access memory (RAM). Processor 104 may also include non-volatile memory, such as read-only memory (ROM), flash memory, hard disk drive (HDD), or solid state drive (SSD).
[0171] The memory 106 stores executable program code, which the processor 104 executes to implement the functions of the aforementioned analysis module and processing module, thereby realizing the data processing method. In other words, the memory 106 stores instructions for executing the data processing method.
[0172] Alternatively, the memory 106 stores executable code, which the processor 104 executes to implement the functions of the aforementioned path lookup device, thereby implementing the data processing method. That is, the memory 106 stores instructions for executing the data processing method.
[0173] The communication interface 108 uses transceiver modules such as, but not limited to, network interface cards and transceivers to enable communication between the computing device 100 and other devices or communication networks.
[0174] This application also provides a computing device cluster. The computing device cluster includes at least one computing device. The computing device can be a server, such as a central server, an edge server, or a local server in a local data center. In some embodiments, the computing device can also be a terminal device such as a desktop computer, a laptop computer, or a smartphone.
[0175] like Figure 9 As shown, the computing device cluster includes at least one computing device 100. The memory 106 of one or more computing devices 100 in the computing device cluster may store the same instructions for executing data processing methods.
[0176] In some possible implementations, the memory 106 of one or more computing devices 100 in the computing device cluster may also store partial instructions for executing data processing methods. In other words, a combination of one or more computing devices 100 can jointly execute instructions for executing data processing methods.
[0177] It should be noted that the memory 106 in different computing devices 100 within the computing device cluster can store different instructions, each used to execute a portion of the functions of the path query device. That is, the instructions stored in the memory 106 of different computing devices 100 can implement the functions of one or more devices in the analysis module and processing module.
[0178] In some possible implementations, one or more computing devices in a computing device cluster can be connected via a network. This network can be a wide area network (WAN) or a local area network (LAN), etc. Figure 10 One possible implementation is shown. For example... Figure 10 As shown, two computing devices 100A and 100B are connected via a network. Specifically, they are connected to the network through communication interfaces in each computing device. In this possible implementation, the memory 106 in computing device 100A stores instructions for executing the functions of the analysis module. Simultaneously, the memory 106 in computing device 100B stores instructions for executing the functions of the processing module.
[0179] It should be understood that Figure 10 The functions of the computing device 100A shown can also be performed by multiple computing devices 100. Similarly, the functions of the computing device 100B can also be performed by multiple computing devices 100.
[0180] This application also relates to a computer storage medium storing a program for signal processing, which, when run on a computer, causes the computer to perform steps as performed by the aforementioned execution device, or causes the computer to perform steps as performed by the aforementioned training device.
[0181] This application also relates to a computer program product that stores instructions that, when executed by a computer, cause the computer to perform steps as performed by the aforementioned execution device, or to perform steps as performed by the aforementioned training device.
[0182] The computing device provided in this application embodiment can specifically be a chip, which includes a processing unit and a communication unit. The processing unit can be, for example, a processor, and the communication unit can be, for example, an input / output interface, pins, or circuits. The processing unit can execute computer execution instructions stored in the storage unit to cause the chip in the execution device to execute the data processing method described in the above embodiments, or to cause the chip in the training device to execute the data processing method described in the above embodiments. Optionally, the storage unit is a storage unit within the chip, such as a register or cache. Alternatively, the storage unit can be a storage unit located outside the chip within the wireless access device, such as a read-only memory (ROM) or other types of static storage devices capable of storing static information and instructions, such as random access memory (RAM).
[0183] For details, please refer to Figure 11 , Figure 11 This is a schematic diagram of the structure of a chip provided in an embodiment of this application. The chip can be represented as a neural network processor (NPU) 1100. The NPU 1100 is mounted as a coprocessor on the host CPU, and tasks are assigned by the host CPU. The core part of the NPU is the arithmetic circuit 1103, which is controlled by the controller 1104 to extract matrix data from the memory and perform multiplication operations.
[0184] In some implementations, the arithmetic circuit 1103 internally includes multiple processing engines (PEs). In some implementations, the arithmetic circuit 1103 is a two-dimensional pulsating array. The arithmetic circuit 1103 can also be a one-dimensional pulsating array or other electronic circuitry capable of performing mathematical operations such as multiplication and addition. In some implementations, the arithmetic circuit 1103 is a general-purpose matrix processor.
[0185] For example, suppose we have an input matrix A, a weight matrix B, and an output matrix C. The arithmetic circuit retrieves the corresponding data of matrix B from the weight memory 1102 and caches it in each PE of the arithmetic circuit. The arithmetic circuit retrieves the data of matrix A from the input memory 1101 and performs matrix operations with matrix B. The partial result or the final result of the obtained matrix is stored in the accumulator 1108.
[0186] Unified memory 1106 is used to store input and output data. Weight data is directly transferred to weight memory 1102 via Direct Memory Access Controller (DMAC) 1105. Input data is also transferred to unified memory 1106 via DMAC.
[0187] BIU stands for Bus Interface Unit, which is used for interaction between the AXI bus and the DMAC and the Instruction Fetch Buffer (IFB) 1109.
[0188] The Bus Interface Unit (BIU) 1110 is used by the instruction fetch memory 1109 to fetch instructions from external memory, and also by the memory access controller 1105 to fetch the original data of the input matrix A or the weight matrix B from external memory.
[0189] The DMAC is mainly used to move input data from external memory DDR to unified memory 1106, or to weight data to weight memory 1102, or to input data to input memory 1101.
[0190] The vector computation unit 1107 includes multiple processing units that, when needed, further process the output of the computation circuit 1103, such as vector multiplication, vector addition, exponential operations, logarithmic operations, size comparisons, etc. It is mainly used for computation in non-convolutional / fully connected layers of neural networks, such as Batch Normalization, pixel-level summation, and upsampling of the predicted label plane.
[0191] In some implementations, vector computation unit 1107 can store the processed output vector in unified memory 1106. For example, vector computation unit 1107 can apply a linear function, or a nonlinear function, to the output of computation circuit 1103, such as linear interpolating the predicted label plane extracted from a convolutional layer, or, for example, accumulating a vector of values to generate activation values. In some implementations, vector computation unit 1107 generates normalized values, pixel-level summed values, or both. In some implementations, the processed output vector can be used as activation input to computation circuit 1103, for example, for use in subsequent layers of the neural network.
[0192] The instruction fetch buffer 1109 connected to the controller 1104 is used to store the instructions used by the controller 1104;
[0193] Unified memory 1106, input memory 1101, weight memory 1102, and instruction fetch memory 1109 are all on-chip memories. External memory is proprietary to this NPU hardware architecture.
[0194] The processor mentioned above can be a general-purpose central processing unit, a microprocessor, an ASIC, or one or more integrated circuits used to control the execution of the above program.
[0195] It should also be noted that the device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. In addition, in the device embodiment drawings provided in this application, the connection relationship between modules indicates that they have a communication connection, which can be implemented as one or more communication buses or signal lines.
[0196] Through the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented by means of software plus necessary general-purpose hardware, or it can be implemented by special-purpose hardware including application-specific integrated circuits, special-purpose CPUs, special-purpose GPUs, special-purpose memory, special-purpose components, etc. Generally, any function performed by a computer program can be easily implemented by corresponding hardware, and the specific hardware structure used to implement the same function can be diverse, such as analog circuits, digital circuits, or special-purpose circuits. However, for this application, software program implementation is more often the preferred implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a readable storage medium, such as a computer floppy disk, USB flash drive, mobile hard disk, ROM, RAM, magnetic disk, or optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, training equipment, or network device, etc.) to execute the methods described in the various embodiments of this application.
[0197] In the above embodiments, the implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, in the form of a computer program product.
[0198] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions may be transmitted from one website, computer, training device, or data center to another website, computer, training device, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a training device or data center that integrates one or more available media. The available media may be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., DVDs), or semiconductor media (e.g., solid-state drives (SSDs)).
Claims
1. A data processing method, characterized in that, include: The program code is analyzed to obtain a first analysis result, which is used to indicate the performance acceleration effect of the program code. The program code includes a first function body and a second function body, wherein the second function body is a function called in the first function body. If the performance acceleration effect meets the first condition, the first function body and the second function body are inlined.
2. The method according to claim 1, characterized in that, The performance acceleration effect refers to the performance acceleration effect after the first function body and the second function body are inlined.
3. The method according to claim 1 or 2, characterized in that, The performance acceleration effect satisfies the first condition, including: The performance acceleration effect is greater than the first threshold.
4. The method according to any one of claims 1-3, characterized in that, The performance acceleration effect includes the performance acceleration effect of the first computation graph, which is the computation graph obtained by transforming the program code after function inlining.
5. The method according to any one of claims 1-4, characterized in that, The first analysis result is determined based on the code snippets in the program code that cause performance loss.
6. The method according to claim 5, characterized in that, The code segments that cause performance loss include: split graph points and guard nodes. The split graph points are code segments in the program code that cannot be converted into a computation graph, and the guard nodes are code segments in the program code used to verify the correctness of the program.
7. The method according to claim 6, characterized in that, The performance loss caused by the crack points is determined based on the number of crack points.
8. The method according to claim 6, characterized in that, The performance loss caused by the guard node is determined based on one or more of the following: The number of guard nodes; The type of the guard node, or; The probability of failure to defend the guard node.
9. The method according to any one of claims 1-8, characterized in that, The method further includes: If the performance acceleration effect does not meet the first condition, no function inlining is performed between the first function body and the second function body.
10. The method according to claim 9, characterized in that, The performance acceleration effect does not meet the first condition, including: The performance acceleration effect is less than or equal to the first threshold.
11. A data processing apparatus, characterized in that, include: An analysis module is used to perform code analysis on the program code and obtain a first analysis result. The first analysis result is used to indicate the performance acceleration effect of the program code. The program code includes a first function body and a second function body, wherein the second function body is a function called in the first function body. The processing module is used to perform function inlining on the first function body and the second function body when the performance acceleration effect meets the first condition.
12. The apparatus according to claim 11, characterized in that, The performance acceleration effect refers to the performance acceleration effect after the first function body and the second function body are inlined.
13. The apparatus according to claim 11 or 12, characterized in that, The performance acceleration effect satisfies the first condition, including: The performance acceleration effect is greater than the first threshold.
14. The apparatus according to any one of claims 11-13, characterized in that, The performance acceleration effect includes the performance acceleration effect of the first computation graph, which is the computation graph obtained by transforming the program code after function inlining.
15. The apparatus according to any one of claims 11-14, characterized in that, The first analysis result is determined based on the code snippets in the program code that cause performance loss.
16. The apparatus according to claim 15, characterized in that, The code segments that cause performance loss include: split graph points and guard nodes. The split graph points are code segments in the program code that cannot be converted into a computation graph, and the guard nodes are code segments in the program code used to verify the correctness of the program.
17. The apparatus according to claim 16, characterized in that, The performance loss caused by the crack points is determined based on the number of crack points.
18. The apparatus according to claim 16, characterized in that, The performance loss caused by the guard node is determined based on one or more of the following: The number of guard nodes; The type of the guard node, or; The probability of failure to defend the guard node.
19. The apparatus according to any one of claims 11-18, characterized in that, The processing module is further configured to: If the performance acceleration effect does not meet the first condition, no function inlining is performed between the first function body and the second function body.
20. The apparatus according to claim 19, characterized in that, The performance acceleration effect does not meet the first condition, including: The performance acceleration effect is less than or equal to the first threshold.
21. A computing device, characterized in that, The device includes at least one processor coupled to a memory; The memory is used to store programs or instructions; The at least one processor is used to execute the program or instructions to cause the apparatus to implement the method as described in any one of claims 1 to 10.
22. A computing device cluster, characterized in that, It includes at least one computing device, each computing device including a processor and memory; The processor of the at least one computing device is configured to execute instructions stored in the memory of the at least one computing device to cause the cluster of computing devices to perform the method as described in claims 1-10.
23. A computer storage medium, characterized in that, The computer storage medium stores one or more instructions, which, when executed by one or more computers, cause the one or more computers to perform the operation of the method according to any one of claims 1 to 10.
24. A computer program product, characterized in that, Includes computer-readable instructions that, when executed on a computer device, cause the computer device to perform the method as described in any one of claims 1 to 10.