A model performance optimization method and computing device

Abstract

The application provides a model performance optimization method and a computing device, and relates to the technical field of computers.In the application, a computing device determines a hierarchical index tree of a first model according to an optimization target of the first model, category information of the first model and a first corresponding relationship.The computing device performs data analysis by referring to the level at which a processing node in the hierarchical index tree is located and the hierarchical association between the processing nodes, determines a first link that does not meet a first performance requirement, and obtains a performance optimization suggestion.Based on this mode, the developer or maintainer of the first model can determine the processing node that causes a performance bottleneck in the inference process of the first model, and a feasible performance optimization suggestion is provided, which further reduces the work complexity of the developer or maintainer of the first model and improves the processing efficiency.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a model performance optimization method and computing device. Background Technology

[0002] In recent years, large language models (LLMs) have received widespread attention due to their outstanding performance on various language generation tasks, which has driven the development of various artificial intelligence applications.

[0003] However, with the exponential growth of model parameter size (e.g., from 175 billion parameters to 1.8 trillion parameters), the computational complexity of the inference phase exhibits a superlinear growth trend. In real-world business scenarios, a single inference request requires coordination of multiple subsystems, including graphics processing unit (GPU) computation, memory management, and parallel communication, resulting in end-to-end latency that fails to meet real-time interaction requirements.

[0004] Therefore, how to quickly locate the bottleneck in model inference performance and implement targeted and rapid optimization is a challenge faced by the industry. Summary of the Invention

[0005] This application provides a model performance optimization method and computing device to quickly locate the performance bottleneck of model inference and optimize it in a targeted and rapid manner.

[0006] Firstly, this application provides a model performance optimization method, which can be applied to a computing device. For example, the computing device may be the computing device itself, components within the computing device (e.g., processors, circuits, chips, or chip systems), or logic modules or software capable of implementing all or part of the functions of the computing device. The computing device may be an electronic device or a server; this application does not limit the specific form of the computing device. The execution is as follows:

[0007] Based on the optimization objective of the first model, the category information of the first model, and the first correspondence, the hierarchical index tree of the first model is determined. The first correspondence indicates the hierarchical association of each processing node in the first model, and the hierarchical index tree indicates the level of each processing node in the first model and the hierarchical association between processing nodes. Data analysis is performed on the hierarchical index tree to obtain the first link and performance optimization suggestions. The first link includes multiple first processing nodes that have hierarchical association in the hierarchical index tree. The performance parameters of multiple first processing nodes do not meet the first performance requirements, and the performance optimization suggestions indicate how to adjust the performance parameters of multiple first processing nodes. The first link and performance optimization suggestions are output.

[0008] Based on the above approach, it can assist the developers or maintainers of the first model in identifying the processing nodes that cause performance bottlenecks during the inference process of the first model, and provide feasible performance optimization suggestions, thereby further reducing the workload of the developers or maintainers of the first model and improving processing efficiency.

[0009] In one possible implementation, the computing device collects the performance parameters of each processing node in the hierarchical index tree layer by layer to obtain a first data analysis result. The first data analysis result indicates that the performance parameters of the first processing node do not meet the first performance requirements. Based on the first data analysis result, a first link and performance optimization suggestions are obtained.

[0010] Based on the above method, the computing device collects data layer by layer from each processing node of the hierarchical index tree, which can more accurately filter out the first processing node that does not meet the first performance requirements, so as to quickly identify the processing node that causes the performance bottleneck in the first model inference process.

[0011] In one possible implementation, the plurality of first processing nodes include second processing nodes in the first layer of the hierarchical index tree that do not meet the first requirement of the first performance requirements, and third processing nodes in other layers that are hierarchically related to the second processing nodes and do not meet other requirements of the first performance requirements.

[0012] Based on the above approach, the first processing node includes not only the second processing node in the first layer, but also the third processing node in other layers. Based on this, the first link can be located, which facilitates the computing device to analyze what the root cause of the performance of the first model is and to determine reasonable performance optimization suggestions.

[0013] In one possible implementation, the proportion of performance parameters collected at each processing node of the hierarchical index tree decreases by 20% layer by layer.

[0014] Based on the above method, the computing device can adaptively and dynamically collect and process the performance parameters of the nodes, which greatly reduces the collection overhead and thus reduces the performance loss caused by collection.

[0015] In one possible implementation, the proportion of performance parameters collected for each processing node in the hierarchical index tree is no less than 20%.

[0016] Based on the above method, the proportion of performance parameters collected for each processing node in the hierarchical index tree is no less than 20%. This ensures the amount of performance parameters collected for each processing node and avoids situations where the amount of performance parameters collected for a processing node is too small, making it impossible to determine the first processing node that does not meet the first performance requirement.

[0017] In one possible implementation, the first link also includes multiple first computing nodes associated with multiple first processing nodes, and the performance parameters of the multiple first computing nodes do not meet the second performance requirements.

[0018] For example, a compute node may include instances and / or compute cards, etc.

[0019] Based on the above approach, the first link includes not only the first processing node but also the first computing node, in order to perform drill-down analysis on the root causes of the performance problems of the first model and to perform more granular analysis on the causes of the performance problems of the first model.

[0020] In one possible implementation, the computing device further collects data layer by layer from the computing nodes associated with the multiple first processing nodes to obtain second data analysis results. The second data analysis results indicate that the performance parameters of the multiple first computing nodes do not meet the second performance requirements. Based on the first data analysis results and the second data analysis results, a first link and performance optimization suggestions are obtained.

[0021] Based on the above method, the computing device collects data layer by layer from each processing node of the hierarchical index tree, and also collects data layer by layer from the computing nodes associated with multiple first processing nodes. This allows for more accurate screening of first processing nodes that do not meet the first performance requirements and first computing nodes that do not meet the second performance requirements, so as to quickly identify the processing nodes that cause performance bottlenecks in the first model inference process.

[0022] In one possible implementation, the plurality of first computing nodes include a second computing node associated with a second processing node and a third computing node associated with a third processing node.

[0023] Based on the above approach, the first computing node includes the second computing node associated with the second processing node in the first layer and the third computing node associated with the third processing node. Based on this, the first link can be located, which facilitates the computing device to analyze what the root cause of the performance of the first model is and to determine more detailed performance optimization suggestions.

[0024] In one possible implementation, the proportion of performance parameters collected for the computing nodes corresponding to the processing nodes at each level of the hierarchical index tree is the same as the proportion of performance parameters collected for the processing nodes at the same level.

[0025] Based on the above approach, the performance parameters of processing nodes and computing nodes in the same layer are collected at the same ratio, which can reduce the collection overhead, thereby reducing the performance loss caused by collection, and avoiding the situation where the performance parameters of processing nodes are collected in too few cases, making it impossible to determine the first processing node that does not meet the first performance requirements.

[0026] In one possible implementation, the hierarchical index tree is stored as a queue.

[0027] Based on the above approach, since the queue is characterized by first-in, first-out, the computing device can prioritize collecting the performance parameters of the processing nodes at the first level of the hierarchical index tree according to the queue storage hierarchy tree, so as to collect the performance parameters of the processing nodes at each level of the hierarchical index tree.

[0028] In one possible implementation, the category information of the first model includes the structural information of the first model, and / or the cluster information corresponding to the first model.

[0029] Based on the above approach, when the type information of the first model includes the structural information of the first model and / or the cluster information corresponding to the first model, it is convenient for the computing device to match the appropriate hierarchical index tree, thereby further improving the processing efficiency of determining the root cause of the performance problem of the first model.

[0030] In one possible implementation, the optimization objective of the first model includes latency requirements.

[0031] Secondly, this application also provides a computing device that has the functionality to implement the behavior in the method example of the first aspect described above. The beneficial effects can be found in the description of the first aspect and any possible implementation thereof, and will not be repeated here. The functionality can be implemented by hardware or by hardware executing corresponding software. The hardware or software includes one or more modules corresponding to the above-described functionality. In one possible design, the computing device includes a hierarchical index tree determination module, a root cause analysis module, and a data output module.

[0032] The module includes a hierarchical indicator tree determination module, used to determine the hierarchical indicator tree of the first model based on the optimization objective of the first model, the category information of the first model, and the first correspondence relationship. The first correspondence relationship indicates the hierarchical association of each processing node in the first model, and the hierarchical indicator tree indicates the level of each processing node in the first model and the hierarchical association between each processing node. The root cause analysis module is used to perform data analysis on the hierarchical indicator tree to obtain the first link and performance optimization suggestions. The first link includes multiple first processing nodes that have hierarchical association in the hierarchical indicator tree. The performance parameters of multiple first processing nodes do not meet the first performance requirements, and the performance optimization suggestions indicate the adjustment method for the performance parameters of multiple first processing nodes. The data output module is used to output the first link and performance optimization suggestions.

[0033] In one possible implementation, the root cause analysis module is also used to collect the performance parameters of each processing node in the hierarchical indicator tree layer by layer to obtain the first data analysis result, which indicates that the performance parameters of the first processing node do not meet the first performance requirement; and to obtain the first link and performance optimization suggestions based on the first data analysis result.

[0034] In one possible implementation, the plurality of first processing nodes include second processing nodes in the first layer of the hierarchical index tree that do not meet the first requirement of the first performance requirements, and third processing nodes in other layers that are hierarchically related to the second processing nodes and do not meet other requirements of the first performance requirements.

[0035] In one possible implementation, the proportion of performance parameters collected at each processing node of the hierarchical index tree decreases by 20% layer by layer.

[0036] In one possible implementation, the proportion of performance parameters collected for each processing node in the hierarchical index tree is no less than 20%.

[0037] In one possible implementation, the first link also includes multiple first computing nodes associated with multiple first processing nodes, and the performance parameters of the multiple first computing nodes do not meet the second performance requirements.

[0038] In one possible implementation, the root cause analysis module is further configured to collect data layer by layer from the computing nodes associated with the multiple first processing nodes to obtain second data analysis results, which indicate that the performance parameters of the multiple first computing nodes do not meet the second performance requirements; and to obtain first link and performance optimization suggestions based on the first and second data analysis results.

[0039] In one possible implementation, the plurality of first computing nodes include a second computing node associated with a second processing node and a third computing node associated with a third processing node.

[0040] In one possible implementation, the proportion of performance parameters collected for the computing nodes corresponding to the processing nodes at each level of the hierarchical index tree is the same as the proportion of performance parameters collected for the processing nodes at the same level.

[0041] In one possible implementation, the hierarchical index tree is stored as a queue.

[0042] In one possible implementation, the category information of the first model includes the structural information of the first model, and / or the cluster information corresponding to the first model.

[0043] Thirdly, this application also provides a computing device, which includes at least one processor; and a communication interface communicatively connected to the at least one processor; wherein the at least one processor executes instructions stored in a memory, causing the method described in the first aspect or any possible design of the first aspect to be executed.

[0044] Fourthly, this application also provides a computing device cluster, which includes at least one computing device having the functionality to implement the behavior in the method example of the first aspect described above. The beneficial effects are described in the first aspect and will not be repeated here. Each computing device includes a processor and a memory. The processor is configured to support the computing device in executing the method described in the first aspect or any possible design of the first aspect. The memory is coupled to the processor and stores the necessary program instructions and data of the computing device. The computing device also includes a communication interface for communicating with other devices, such as acquiring data to be detected from a target system.

[0045] Fifthly, this application also provides a computer-readable storage medium storing instructions that, when executed on a computer, cause a computing device to perform the method described in the first aspect or any possible design of the first aspect.

[0046] Sixthly, this application also provides a computer program product containing instructions that, when run on a computing device, cause the computing device to perform the method described in the first aspect or any possible design of the first aspect.

[0047] In a seventh aspect, this application also provides a computer chip connected to a memory, the chip being used to read and execute a software program stored in the memory, and to execute the method described in the first aspect or any possible design of the first aspect.

[0048] For the beneficial effects of aspects two through seven, please refer to the beneficial effects of aspect one, which will not be repeated here. Attached Figure Description

[0049] Figure 1 A schematic diagram of a model performance root cause determination scenario is shown;

[0050] Figure 2 A flowchart illustrating a model performance optimization method provided in an embodiment of this application is shown.

[0051] Figure 3 A schematic diagram of a hierarchical index tree provided in an embodiment of this application is shown;

[0052] Figure 4 A schematic diagram of a hierarchical index tree provided in an embodiment of this application is shown;

[0053] Figure 5 A schematic diagram of a hierarchical index tree provided in an embodiment of this application is shown;

[0054] Figure 6 A schematic diagram of a data acquisition process provided in an embodiment of this application is shown;

[0055] Figure 7 This paper shows a schematic diagram of the structure of a computing device provided in an embodiment of this application;

[0056] Figure 8 A schematic diagram of the structure of a computing device provided in an embodiment of this application is shown. Detailed Implementation

[0057] The technical solutions of the embodiments of this application will now be described with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them.

[0058] In this application embodiment, "at least one" refers to one or more, and "more than one" refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one of the following" or similar expressions refer to any combination of these dozen or more items, including any combination of single or plural items. For example, at least one of a, b, or c can represent: a, b, c, ab, ac, bc, or abc, where a, b, and c can be single or multiple.

[0059] In the embodiments of this application, "send" and "receive" indicate the direction of signal transmission. "Send" can also be understood as the "output" of the chip interface, and "receive" can also be understood as the "input" of the chip interface. In other words, sending and receiving can occur between devices, for example, through buses, traces, or interfaces between components, modules, chips, software modules, or hardware modules within a device. Similar expressions in this application can be understood in a similar way and will not be elaborated further.

[0060] In the embodiments of this application, "when," "if," and "if" all refer to the device taking corresponding actions under certain objective circumstances, not a time limit, nor do they require the device to perform a judgment action, nor do they imply any other limitations. Unless otherwise specified, "if" and "if" are interchangeable, and "when" and "in the case of" are interchangeable. "When" and "if" / "if" are interchangeable. In the embodiments of this application, "*" can be used to represent "multiplication."

[0061] The ordinal numbers such as "first" and "second" mentioned in the embodiments of this application are used to distinguish multiple objects and are not used to limit the size, content, order, timing, priority, or importance of the multiple objects. For example, a first processing node and a second processing node refer to two different processing nodes, and do not indicate that the content, priority, or importance of these two processing nodes are different. Words such as "exemplary" or "for example" are used to indicate that they are examples, illustrations, or explanations. Any embodiment or design scheme described as "exemplary" or "for example" in this application should not be construed as being better or more advantageous than other embodiments or design schemes. Specifically, the use of words such as "exemplary" or "for example" is intended to present the relevant concepts in a specific manner.

[0062] Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion, so that a process, method, system, product, or apparatus that comprises a list of units is not necessarily limited to those units, but may include other units not expressly listed or inherent to those processes, methods, products, or apparatuses. The methods and apparatuses provided in the embodiments of this application are based on the same or similar technical concepts. Since the principles by which the methods and apparatuses solve the problems are similar, implementations of the apparatus and methods can be referred to mutually, and repeated details will not be elaborated further.

[0063] The model performance optimization method provided in this application is applicable to the debugging phase before the model goes live, or the version update phase after the model goes live. Figure 1 A schematic diagram illustrating an application scenario adapted to this application is shown. Before Model 1 goes live, developers optimize its performance to meet latency and / or computing power requirements. Once it is determined that Model 1 meets the latency and / or computing power requirements, it is deployed to electronic devices or servers for user use. When optimizing the performance of Model 1, developers perform data analysis on the performance parameters of the processing nodes in Model 1 based on the computing device to determine the root cause (i.e., the fundamental cause, primary cause, or essential cause) of performance failures in Model 1. Figure 1 This example uses Model 1, and the name of the model is not limited in specific applications. The model can be an artificial intelligence (AI) processing model, such as LLMs.

[0064] To determine the root cause of model performance issues, current technologies indiscriminately collect data from specific executable code segments, the physical devices running the model, or information related to model optimization. This approach may negatively impact the inference efficiency of latency-sensitive models. Furthermore, existing model performance analyses focus on localized performance at the network node or model slice level, while the performance of an inference system is influenced by hardware resources, model structure, and runtime framework. Current technologies lack a comprehensive end-to-end performance analysis approach for the entire inference system, from macro to micro levels. Additionally, existing model performance analyses merely provide results without offering targeted optimization suggestions.

[0065] Based on this, this application provides a model performance optimization method to quickly locate performance bottlenecks in model inference and optimize them accordingly. This model performance optimization method can be applied to computing devices, such as the computing device itself, components within the computing device (e.g., processors, circuits, chips, or chip systems), or logic modules or software that implement all or part of the functions of the computing device. The computing device can be an electronic device or a server; this application does not limit the specific form of the computing device. The processor of the computing device can include one or more of the following: application processor (AP), modem processor, GPU, image signal processor (ISP), controller, memory, video codec, digital signal processor (DSP), baseband processor, or neural network processing unit (NPU). Different processing units can be independent devices or integrated into one or more processors. The controller can be the nerve center and command center of the electronic device. The controller can generate operation control signals based on instruction opcodes and timing signals to control instruction fetching and execution. The processor can also include memory for storing instructions and data. In some embodiments, the memory in the processor is a cache memory. This memory can store instructions or data that the processor has just used or that are used repeatedly. If the processor needs to use the instruction or data again, it can retrieve it directly from the memory. (See also...) Figure 2 Execute as follows:

[0066] Step 201: The computing device determines the hierarchical index tree of the first model based on the optimization objective of the first model, the category information of the first model, and the first correspondence. The first correspondence indicates the hierarchical association of each processing node in the first model, and the hierarchical index tree indicates the level of each processing node in the first model and the hierarchical association between each processing node.

[0067] The first model is an AI processing model, such as a large language model. The optimization objectives of the first model include latency requirements and / or computing power requirements, so that the optimized first model can meet these requirements. The latency requirement can be further refined into the time to first token (TTFT), which is the time from the input question to the generation of the first token, and the time per output token (TPOT), excluding the first token. For example, TTFT is 1 second. The computing power requirement can be refined into the model's computational speed and accuracy; for example, the computing power requirement is that the model's computational accuracy is at the millimeter level. This is only an illustrative example and not a specific limitation. For example, the optimization objectives of the first model may also include result reliability, etc., which are not specifically limited here. After obtaining the optimization objectives of the first model, the computing device can clarify the optimization requirements of the first model in order to specifically identify the reasons for not meeting the optimization objectives.

[0068] The category information of the first model indicates the information associated with the first model. For example, the category information of the first model includes the structural information of the first model (e.g., the model structure of the first model is Transformer, MoE, or Transformer Decoder-Only), and / or the cluster information corresponding to the first model (where the cluster information is as follows: Example 1, eight NPU cards are deployed in one instance, with a total of 10 instances; Example 2, 16 GPU cards are deployed in one instance, with a total of 20 instances). After obtaining the category information of the first model, the computing device can match an appropriate hierarchical indicator tree, further improving the processing efficiency for determining the root causes of the first model's performance problems.

[0069] The first correspondence can be configured based on expert experience and model reasoning scenarios. It can be pre-stored in the computing device, or it can be obtained by the computing device after determining which first model to optimize, by requesting it from a server or other computing device that stores the first correspondence. The specific method by which the computing device obtains the first correspondence is not limited here.

[0070] The first correspondence includes the name of the processing node (e.g., indicated by the NodeName field, such as Latency, TTFT, etc.), the identifier of the processing node (e.g., indicated by the NodeID field, such as Node 0, Node 1, etc.), the computing nodes associated with the processing node (also referred to as associated objects, such as services, instances, and computing cards, etc.) (e.g., indicated by the Objects field), the threshold corresponding to the performance parameters of the processing node (e.g., indicated by the Threshold field, such as (0,2], [2,5]), and the model name (e.g., indicated by the ModelName field, such as Model 1, Model 2), etc. Furthermore, the first correspondence also indicates the information of the parent node (e.g., indicated by the ParentID field, where -1 represents the root node). For example, the Latency performance parameter corresponding to the Latency threshold is used as a condition for anomaly detection; exceeding the threshold is considered an anomaly. This threshold can be a predicted value obtained from modeling based on the actual model structure, hardware resources, and business scenario, or it can be dynamically changed and refreshed based on periodic actual business data, such as using the average data of the previous day as the threshold, automatically refreshing the threshold at 0:00 every day. This is only an illustrative example and not a specific limitation.

[0071] After obtaining the optimization objective and category information of the first model, the computing device determines the first correspondence as shown in Table 1 below. The computing device can then construct a hierarchical index tree based on this first correspondence. Here, the first model is model S, the identifier of the processing node is 1, the name of processing node 1 is Latency (which can be understood as the latency requirement of the first model's optimization objective), processing node 1 is the parent node (-1), and the computing nodes associated with processing node 1 include services and instances. The threshold values ​​corresponding to the performance parameters of processing node 1 are [0, 2s, 5s]. This is only an illustrative example and does not list all the information in Table 1.

[0072] Table 1

[0073]

[0074]

[0075] Based on Table 1 above, the computing device determines the hierarchical index tree of the first model as follows: Figure 3As shown, the first layer of the hierarchical indicator tree has a Latency node as its processing node, and the compute nodes associated with Latency are services and instances. The second layer has TTFT and TPOT nodes as its processing nodes, and the compute nodes associated with TTFT and TPOT are instances. The third layer has compute, communication, and scheduling nodes, which are child nodes of TTFT and TPOT respectively. The compute nodes associated with the third layer have instances and compute cards. The fourth layer has operators 1, 2, and 3 (with compute as the parent node of TTFT), operators 4 and 5 (with communication as the parent node of TTFT), operators 6, 7, and 8 (with scheduling as the parent node of TTFT), operators 1, 2, and 3 (with compute as the parent node of TPOT), operators 4 and 5 (with communication as the parent node of TPOT), and operators 6, 7, and 8 (with scheduling as the parent node of TPOT). The compute nodes associated with the fourth layer have compute cards.

[0076] For example, the hierarchical index tree can be stored as a queue. Since a queue operates on a first-in, first-out (FIFO) principle, storing the hierarchical index tree according to the queue allows the computing device to prioritize collecting performance parameters of the processing nodes at the first level of the hierarchical index tree, thus enabling the collection of performance parameters of the processing nodes at each level. In specific applications, the hierarchical index tree can also be stored in other ways, which are not specifically limited here. For example, steps 1 to 6 can be followed:

[0077] Step 1: The computing device initializes queue M as an empty queue and initializes the root node of the hierarchical index tree T as the optimization objective of the first model.

[0078] Step 2: The computing device queries the list of the first correspondences to determine whether the optimization objective of the first model has a child node. If it does, proceed to step 3; otherwise, proceed to step 5.

[0079] Step 3: The computing device takes the found child node as the child node of the current first model's optimization target, adds the child node of this query to the tail of queue M, and then executes step 4.

[0080] Step 4: Dequeue M dequeues one processing node from the head of the queue and jumps to step 2.

[0081] Step 5: The computing device determines whether queue M is empty. If it is an empty queue, proceed to step 6; otherwise, jump to step 4.

[0082] Step 6: The computing device outputs a hierarchical index tree.

[0083] For example, if the optimization objective of the first model is latency requirements, without considering the associated computing nodes of the processing nodes, the hierarchical index tree is as follows: Figure 4As shown. The first layer of the hierarchical index tree has Latency as its processing node. The second layer has TTFT and TPOT as its processing nodes. The third layer has computation, communication, and scheduling nodes, which are child nodes of TTFT and TPOT, respectively. The fourth layer has operators 1, 2, and 3 with computation as the parent node of TTFT; operators 4 and 5 with communication as the parent node of TTFT; operators 6, 7, and 8 with scheduling as the parent node of TTFT; operators 1, 2, and 3 with computation as the parent node of TPOT; operators 4 and 5 with communication as the parent node of TPOT; and operators 6, 7, and 8 with scheduling as the parent node of TPOT. The fifth layer has computation and transport nodes with operator 1 as the parent node; computation and transport nodes with operator 2 as the parent node; computation and transport nodes with operator 3 as the parent node; computation and transport nodes with operator 4 as the parent node; and computation and transport nodes with operator 5 as the parent node.

[0084] For example, if the optimization objective of the first model is TTFT, without considering the associated computing nodes of the processing nodes, the hierarchical index tree is as follows: Figure 5 As shown. The first layer of the hierarchical index tree has TTFT as its processing node. The second layer's processing nodes include computation, communication, and scheduling. The third layer's processing nodes include operators 1, 2, and 3 (with computation as the parent node), operators 4 and 5 (with communication as the parent node), and operators 6, 7, and 8 (with scheduling as the parent node). The fourth layer's processing nodes include computation and data transfer with operator 1 as the parent node, computation and data transfer with operator 2 as the parent node, computation and data transfer with operator 3 as the parent node, computation and data transfer with operator 4 as the parent node, and computation and data transfer with operator 5 as the parent node.

[0085] The above is merely illustrative and does not specifically limit the form of the hierarchical indicator tree.

[0086] Step 202: The computing device performs data analysis on the hierarchical index tree to obtain the first link and performance optimization suggestions.

[0087] The first link includes multiple first processing nodes that are hierarchically related in the hierarchical index tree. The performance parameters of multiple first processing nodes do not meet the first performance requirements (specifically, different processing nodes may correspond to different first performance requirements, as shown in Table 1 above, where processing node 1 corresponds to the first performance requirement Threshold[0s,2.5s], and processing node 5 corresponds to the first performance requirement Threshold[0ms,30ms]). The performance optimization suggestion indicates how to adjust the performance parameters of multiple first processing nodes.

[0088] In one possible implementation, the computing device collects performance parameters of each processing node in the hierarchical index tree layer by layer to obtain a first data analysis result. The first data analysis result indicates that the performance parameters of the first processing node do not meet the first performance requirement. Then, based on the first data analysis result, the computing device obtains a first link and performance optimization suggestions. By collecting data layer by layer from each processing node in the hierarchical index tree, the computing device can more accurately filter out the first processing nodes that do not meet the first performance requirement, thus quickly identifying the processing nodes causing performance bottlenecks during the first model inference process. The multiple first processing nodes include second processing nodes in the first layer of the hierarchical index tree that do not meet the first requirement of the first performance requirement (e.g., the threshold corresponding to the performance parameters of the aforementioned processing node), and third processing nodes at other levels that are hierarchically related to the second processing nodes and do not meet other requirements of the first performance requirement. In this application, the first processing nodes include not only the second processing nodes in the first layer but also third processing nodes in other layers. Based on this, the first link can be located, facilitating the computing device to analyze the root cause of the first model performance issues and determine reasonable performance optimization suggestions. During layer-by-layer collection, the proportion of performance parameters collected from each processing node in the hierarchical index tree decreases by 20% layer by layer. Based on this, the computing device can adaptively and dynamically collect performance parameters of processing nodes, greatly reducing collection overhead and thus minimizing performance loss caused by collection. Furthermore, the collection ratio of performance parameters corresponding to processing nodes at each level of the hierarchical index tree is no less than 20%. This ensures sufficient collection of performance parameters for each processing node, preventing situations where insufficient data collection makes it impossible to identify the first processing node that does not meet the initial performance requirements.

[0089] It's important to note that when collecting data from processing nodes in a hierarchical indicator tree, three parameters can be used: Indicator, Object, and Ratio. The Indicator is the input parameter for the module generating the data collection requirements. For the first time, the root node is used as the indicator; for subsequent times, child nodes that do not meet the first performance requirement are used. Examples include processing nodes like Latency, TTFT, and TPOT. The Object refers to a data unit to be collected. In an inference cluster, this can refer to a specific cluster, instance, machine, or time period. For the first collection, the entire cluster is used; for subsequent times, the compute nodes associated with child nodes that do not meet the first performance requirement are used. Examples include clusters, instances, and compute cards. The Ratio specifies the amount of data to collect. The initial Ratio is 100%. Considering that finer data granularity has a greater impact on system performance, the Ratio for subsequent times decreases progressively with each indicator level, with a default attenuation rate of 20%. This attenuation rate can be adjusted to other values ​​such as 30%, which is not specifically limited here.

[0090] In a single data collection scenario, the collection metrics clearly define the content to be collected, the collection objects specify the concrete collection targets, and the collection ratio determines the quantity to be collected. Based on this, a small amount of abnormal indicator data from anomalous entities is collected for performance analysis, significantly reducing the impact of data collection on system performance. For example, for a specific inference service cluster, if Indicator = TTFT, ratio = 0.3, and object = instance 1, then the collection requirement could be to collect the first token timeout of all requests in instance 1 at a ratio of 30%. Compared to the existing solution of "collecting the timeout of all tokens across all instances," this reduces the amount of data collected to (1 / instance) * (1 / average output token) * 0.3, thus also reducing the impact on business performance degradation.

[0091] For example, after acquiring the hierarchical index tree, the computing device can collect 100% of the performance parameters of the processing nodes in the first layer of the hierarchical index tree. Through data analysis, it can determine that the performance parameters of the second processing node in the first layer do not meet the first requirement of the first performance requirement; for example, the performance parameters of the second processing node do not meet the corresponding threshold requirement. The computing device can then collect 80% of the performance parameters of the processing nodes in the second layer associated with the second processing node. Through data analysis, it can determine that the processing nodes in the second layer whose performance parameters do not meet the second requirement of the first performance requirement. The above steps are executed sequentially and cyclically, with the collection ratio decreasing by 20% layer by layer, until a third processing node other than the second processing node in the first layer is identified. Figure 4For example, the computing device can collect 100% of the latency performance parameters, then collect 80% of the TTFT and TPOT performance parameters. If the TTFT performance parameters do not meet the first performance requirement, then collect 60% of the computation, communication, and scheduling parameters with TTFT as the parent node. If the computation performance parameters do not meet the first performance requirement, then collect 40% of the computation parameters of operators 1, 2, and 3 with TTFT as the parent node. If the performance parameters of operator 1 do not meet the first performance requirement, then collect 20% of the computation and data transfer parameters with operator 1 as the parent node. If the data transfer performance parameters do not meet the first performance requirement, then the first link is determined to be latency → TTFT → computation → operator 1 → data transfer. Performance optimization suggestions could include merging operators to reduce operator data transfer overhead. This is only an example and not a specific limitation.

[0092] It should be noted that the hierarchical indicator tree also includes calculation nodes, as mentioned above. Figure 3 The hierarchical indicator tree shown indicates that the first link also includes multiple first computing nodes associated with multiple first processing nodes, and the performance parameters of multiple first computing nodes do not meet the second performance requirements. Based on this, it is convenient for the computing device to perform drill-down analysis on the root causes of the first model performance problems, and to conduct a more granular analysis of the reasons for the first model performance problems.

[0093] In another possible implementation, the computing device, in addition to collecting data layer by layer from the processing nodes of the hierarchical index tree, also collects data layer by layer from the computing nodes associated with multiple first processing nodes to obtain second data analysis results. These second data analysis results indicate that the performance parameters of multiple first computing nodes do not meet the second performance requirements. Subsequently, based on the first and second data analysis results, the computing device obtains first links and performance optimization suggestions. Based on this, the computing device can more accurately filter out first processing nodes that do not meet the first performance requirements and first computing nodes that do not meet the second performance requirements, in order to quickly identify the processing nodes causing performance bottlenecks during the first model inference process. The multiple first computing nodes include second computing nodes associated with second processing nodes and third computing nodes associated with third processing nodes. Based on this, the computing device can locate the first links, facilitating analysis of the root causes of the first model performance issues and determining more refined performance optimization suggestions. Furthermore, the proportion of performance parameters collected from the computing nodes corresponding to the processing nodes at each layer of the hierarchical index tree is the same as the proportion of performance parameters collected from the processing nodes at the same layer. The same ratio of performance parameter collection for processing nodes and computing nodes in the same layer can reduce collection overhead, thereby reducing performance loss caused by collection, and avoid the situation where the amount of performance parameter collection for processing nodes is too small, making it impossible to determine the first processing node that does not meet the first performance requirements.

[0094] For example, after acquiring the hierarchical index tree, the computing device can collect 100% of the performance parameters of the processing nodes in the first layer of the hierarchical index tree. Through data analysis, if it is determined that the performance parameters of the second processing node in the first layer do not meet the first requirement of the first performance requirement (e.g., the performance parameters of the second processing node do not meet the corresponding threshold requirement), the computing device can check whether there are any associated computing nodes for the second processing node. If there are associated computing nodes, the computing device collects 100% of the performance parameters of the associated computing nodes until all associated computing nodes of the second processing node have been collected, as described above. Figure 3 First, service performance data can be collected, followed by instance performance data. Then, the computing device can collect 80% of the performance parameters of the second-layer processing nodes associated with the second processing node. Data analysis identifies processing nodes in the second layer whose performance parameters do not meet the second requirement of the first performance requirement. The computing device can then check if there are any associated computing nodes for the second processing node. If so, 80% of the performance parameters of the associated computing nodes are collected until all associated computing nodes of the second processing node are collected. This process is repeated cyclically, decreasing the collection percentage by 20% layer by layer, until a third processing node other than the second processing node in the first processing node is identified, along with the third computing node associated with that third processing node. Based on the above... Figure 3For example, the computing device can collect 100% of the performance parameters of Latency, and 100% of the services and instances associated with Latency. If instance 2 of latency does not meet the second performance requirement, then 80% of the data will be collected from TTFT and TPOT, and 80% of the data will be collected from instances associated with TTFT and TPOT. If the performance parameters of TTFT do not meet the first performance requirement, and instance 3 associated with TTFT does not meet the second performance requirement, then 60% of the data will be collected from computation, communication, and scheduling with TTFT as the parent node, and 60% of the data will be collected from instances and computing cards associated with computation, communication, and scheduling with TTFT as the parent node. If the performance parameters of computation do not meet the first performance requirement, and computing card 8 associated with computation does not meet the second performance requirement, then 40% of the data will be collected from operators 1, 2, and 3 with TTFT as the parent node, and 40% of the data will be collected from computing cards associated with operators 1, 2, and 3. If the performance parameters of operator 1 do not meet the first performance requirement, and computing card 6 associated with operator 1 does not meet the second performance requirement, then 20% of the data will be collected from computation and transport with operator 1 as the parent node, and 20% of the data will be collected from computing cards associated with computation and transport. If the performance parameters of the transport do not meet the first performance requirement, and the associated compute card 10 does not meet the second performance requirement, then the first link is determined as: latency → service → instance 2 → TTFT → instance 3 → compute → instance → compute card 8 → operator 1 → compute card 6 → transport → compute card 10. Performance optimization suggestions could include fusing operators, adjusting compute card 10, and reducing operator transport overhead. This is merely an example and not a specific limitation.

[0095] Specifically, please refer to Figure 6 To collect performance parameters, the computing device first determines whether it's the first time collecting data from a processing node in the hierarchical indicator tree. If so, it initializes the collection parameters for the processing node and outputs the collection requirements. Otherwise, it checks if the currently collected processing node has any associated computing nodes. If associated computing nodes exist, the collection object is updated to the computing node, while the collection indicators and collection ratio remain unchanged, and the collection requirements are output. If no associated computing nodes exist, it checks if child nodes exist. If so, the collection object is updated to the child node, the collection ratio is reduced by 20%, and the collection requirements are output. Otherwise, the loop ends, and the data chain is output. The computing device analyzes the data for each output collection requirement and iterates to the next round of data collection, clarifying the collection parameters.

[0096] Step 203: The computing device outputs the first link and performance optimization suggestions.

[0097] For example, the computing device may display the first link and performance optimization suggestions on the screen, or it may announce the first link and performance optimization suggestions via voice. This is merely illustrative and not intended to be specific.

[0098] In this application, the computing device determines the hierarchical index tree of the first model based on the optimization objective of the first model, the category information of the first model, and the first correspondence. The computing device performs data analysis by referring to the hierarchy of the processing nodes in the hierarchical index tree and the hierarchical relationships between the processing nodes to identify the first link that does not meet the first performance requirements and obtain performance optimization suggestions. This method can assist the developers or maintainers of the first model in identifying the processing nodes that cause performance bottlenecks during the inference process of the first model and provide feasible performance optimization suggestions, further reducing the workload of the developers or maintainers of the first model and improving processing efficiency.

[0099] The foregoing primarily describes the solutions provided by the embodiments of this application from the perspective of device interaction. It is understood that, in order to achieve the above functions, each device may include corresponding hardware structures and / or software modules for executing each function. Those skilled in the art should readily recognize that, in conjunction with the units and algorithm steps of the various examples described in the embodiments disclosed herein, the embodiments of this application can be implemented in hardware or a combination of hardware and computer software. Whether a function is executed in hardware or by computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0100] The embodiments of this application can divide the device into functional units according to the above method examples. For example, each function can be divided into a separate functional unit, or two or more functions can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0101] For example, the structure of a computing device is as follows Figure 7 As shown, it includes a hierarchical indicator tree determination module 701, a root cause analysis module 702, and a data output module 703.

[0102] The hierarchical indicator tree determination module 701 is used to determine the hierarchical indicator tree of the first model based on the optimization objective of the first model, the category information of the first model, and the first correspondence relationship. The first correspondence relationship indicates the hierarchical association of each processing node in the first model, and the hierarchical indicator tree indicates the level of each processing node in the first model and the hierarchical association between each processing node. The root cause analysis module 702 is used to perform data analysis on the hierarchical indicator tree to obtain the first link and performance optimization suggestions. The first link includes multiple first processing nodes that have hierarchical association in the hierarchical indicator tree. The performance parameters of multiple first processing nodes do not meet the first performance requirements, and the performance optimization suggestions indicate the adjustment method for the performance parameters of multiple first processing nodes. The data output module 703 is used to output the first link and performance optimization suggestions.

[0103] The following section uses the hierarchical index tree determination module 701 in the computing device 700 as an example to introduce the implementation of the hierarchical index tree determination module 701. Similarly, the implementation of the root cause analysis module 702 and the data output module 703 can refer to the implementation of the hierarchical index tree determination module 701.

[0104] When implemented in software, the hierarchical indicator tree determination module 701 can be an application or code block running on a computer device. The computer device can be at least one of a physical host, virtual machine, container, or other computing device. Furthermore, the aforementioned computer device can be one or more. For example, the hierarchical indicator tree determination module 701 can be an application running on multiple hosts / virtual machines / containers. It should be noted that the multiple hosts / virtual machines / containers used to run the application can be distributed within the same availability zone (AZ) or in different AZs. The multiple hosts / virtual machines / containers used to run the application can be distributed within the same region or in different regions. Typically, a region can include multiple AZs.

[0105] Similarly, multiple hosts / virtual machines / containers used to run the application can be distributed within the same Virtual Private Cloud (VPC) or across multiple VPCs. Typically, a region can include multiple VPCs, and a VPC can include multiple Availability Zones (AZs).

[0106] When implemented in hardware, the hierarchical index tree determination module 701 may include at least one computing device, such as a server. Alternatively, the hierarchical index tree determination module 701 may also be a device implemented using an application-specific integrated circuit (ASIC) or a programmable logic device (PLD). The PLD may be implemented using a complex programmable logical device (CPLD), a field-programmable gate array (FPGA), generic array logic (GAL), or any combination thereof.

[0107] The hierarchical index tree determination module 701 includes multiple computing devices that can be distributed within the same Availability Zone (AZ) or in different AZs. Similarly, the hierarchical index tree determination module 701 can be distributed within the same region or in different regions. Likewise, the hierarchical index tree determination module 701 can be distributed within the same Virtual Private Cloud (VPC) or in multiple VPCs. These multiple computing devices can be any combination of computing devices such as servers, ASICs, PLDs, CPLDs, FPGAs, and GALs.

[0108] It should be noted that the module division in this embodiment is illustrative and only represents a logical functional division; in actual implementation, there may be other division methods. The functional modules in this embodiment can be integrated into one module, or each module can exist physically separately, or two or more modules can be integrated into one module. For example, the hierarchical index tree determination module 701, the root cause analysis module 702, and the data output module 703 can be integrated into one module, or the root cause analysis module 702 and the data output module 703 can be the same module. The integrated units described above can be implemented in hardware or as software functional units.

[0109] Figure 8 This is a schematic diagram of the structure of a computing device 800 provided in an embodiment of this application. The computing device 800 may be a mobile terminal as listed above, such as a mobile phone. Figure 8As shown, the computing device 800 may include: one or more processors 801; one or more memories 802; a communication interface 803; and one or more computer programs 804. These components can be connected via one or more communication buses 805. The one or more computer programs 804 are stored in the memory 802 and configured to be executed by the one or more processors 801. The one or more computer programs 804 include instructions. The communication interface 803 is used to enable communication between the computing device 800 and other devices; for example, the communication interface may be a transceiver.

[0110] In the embodiments provided above, the methods provided by the embodiments of this application are described from the perspective of a computing device (e.g., a mobile phone) as the executing entity. To implement the functions of the methods provided in the embodiments of this application, the electronic device may include hardware structures and / or software modules, implementing the above functions in the form of hardware structures, software modules, or a combination of hardware structures and software modules. Whether a particular function is implemented in the form of hardware structures, software modules, or a combination of hardware structures and software modules depends on the specific application and design constraints of the technical solution.

[0111] In the above embodiments, implementation can be achieved entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of the present invention are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid state disk (SSD)). Where there is no conflict, the solutions in the above embodiments can be used in combination.

[0112] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0113] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to this application. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0114] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0115] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0116] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the scope and intent of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application is also intended to include such modifications and variations.

Claims

1. A method for optimizing model performance, characterized in that, include: Based on the optimization objective of the first model, the category information of the first model, and the first correspondence, the hierarchical index tree of the first model is determined. The first correspondence indicates the hierarchical association of each processing node in the first model, and the hierarchical index tree indicates the level of each processing node in the first model and the hierarchical association between each processing node. Data analysis is performed on the hierarchical indicator tree to obtain a first link and performance optimization suggestions. The first link includes multiple first processing nodes that are hierarchically related in the hierarchical indicator tree. The performance parameters of the multiple first processing nodes do not meet the first performance requirements. The performance optimization suggestions indicate how to adjust the performance parameters of the multiple first processing nodes. Output the first link and the performance optimization suggestions.

2. The method according to claim 1, characterized in that, The step of performing data analysis on the hierarchical indicator tree to obtain the first link and performance optimization suggestions includes: The performance parameters of each processing node in the hierarchical index tree are collected layer by layer to obtain the first data analysis result, which indicates that the performance parameters of the first processing node do not meet the first performance requirement. Based on the first data analysis results, the first link and the performance optimization suggestions are obtained.

3. The method according to claim 2, characterized in that, The plurality of first processing nodes include a second processing node in the first layer of the hierarchical index tree that does not meet the first requirement of the first performance requirement, and a third processing node in other layers that has a hierarchical association with the second processing node and does not meet other requirements of the first performance requirement.

4. The method according to claim 3, characterized in that, The proportion of performance parameters collected for each processing node in the hierarchical index tree decreases by 20% layer by layer.

5. The method according to claim 3 or 4, characterized in that, The proportion of performance parameters collected for each processing node in the hierarchical index tree is no less than 20%.

6. The method according to any one of claims 3-5, characterized in that, The first link also includes multiple first computing nodes associated with the multiple first processing nodes, and the performance parameters of the multiple first computing nodes do not meet the second performance requirements.

7. The method according to claim 6, characterized in that, The method further includes: The computing nodes associated with the plurality of first processing nodes are collected layer by layer to obtain the second data analysis result, which indicates that the performance parameters of the plurality of first computing nodes do not meet the second performance requirement. The acquisition of the first link and performance optimization suggestions includes: Based on the first data analysis results and the second data analysis results, the first link and the performance optimization suggestions are obtained.

8. The method according to claim 6 or 7, characterized in that, The plurality of first computing nodes include a second computing node associated with the second processing node and a third computing node associated with the third processing node.

9. The method according to claim 7 or 8, characterized in that, The proportion of performance parameters collected for the computing nodes corresponding to the processing nodes at each level of the hierarchical index tree is the same as the proportion of performance parameters collected for the processing nodes at the same level.

10. The method according to any one of claims 1-9, characterized in that, The hierarchical index tree is stored in a queue.

11. The method according to any one of claims 1-10, characterized in that, The category information of the first model includes the structural information of the first model and / or the cluster information corresponding to the first model.

12. The method according to any one of claims 1-11, characterized in that, The optimization objective of the first model includes latency requirements.

13. A computing device, characterized in that, It includes at least one processor; and a communication interface communicatively connected to the at least one processor; the at least one processor causes the method as described in any one of claims 1-12 to be executed by executing instructions stored in memory.

14. A computing device cluster, characterized in that, It includes at least one computing device, each computing device including a processor and memory; The processor of the at least one computing device is configured to execute instructions stored in the memory of the at least one computing device to cause the cluster of computing devices to perform the method as described in any one of claims 1-12.

15. A computer program product containing instructions, characterized in that, When the instructions are executed by the computing device cluster, the computing device causes the computing device to perform the method as described in any one of claims 1-12.

16. A computer-readable storage medium, characterized in that, Includes computer program instructions, which, when executed by a computing device, cause the computing device cluster to perform the method as described in any one of claims 1-12.

Images