AI task analysis method and related equipment

Abstract

The invention provides an analysis method of an AI task and related equipment, which are used for improving the identification efficiency of the performance bottleneck of the AI task. The method comprises the steps that operation information of an AI task is obtained, the AI task comprises a plurality of subtasks, and the operation information comprises at least one of the following items: a dependency relationship among the subtasks, a scheduling strategy of the subtasks on multiple pieces of hardware and the resource utilization rate of each piece of hardware. And identifying the performance bottleneck of the AI task based on the operation information of the AI task. And based on the performance bottleneck of the AI task, generating an optimization suggestion of the AI task.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence (AI), and more particularly to an analysis method and related equipment for AI tasks. Background Technology

[0002] Driven by technological advancements and market forces, AI tasks based on large AI models in heterogeneous computing are developing rapidly, especially in fields such as natural language processing, image recognition, and speech recognition, where they have achieved remarkable success. How to further improve the performance of AI tasks has become a pressing issue.

[0003] In the relevant technical solutions, a performance profiler tool is used to analyze various performance characteristics of the AI ​​task and display them in a visual way. Then, based on human interpretation of the graphs, the performance bottlenecks of the AI ​​task are identified.

[0004] In this approach, the complex structure and numerous modules of AI tasks make the visualization interface displayed by the profiler tool complex, which in turn makes it difficult for manual identification of AI task bottlenecks based on the visualization interface, thus reducing the efficiency of identifying performance bottlenecks of AI tasks. Summary of the Invention

[0005] This application provides an analysis method and related equipment for AI tasks, which can improve the efficiency of identifying performance bottlenecks in AI tasks and generate optimization suggestions for AI tasks, thereby improving the performance of AI tasks.

[0006] Firstly, this application provides a method for analyzing AI tasks, including:

[0007] AI tasks refer to tasks completed using AI technology, including AI model training or inference processes. AI tasks have been widely applied in many fields, such as image recognition, autonomous driving, natural language processing, and robot interaction. The AI ​​tasks in this application run on multiple hardware devices, including various hardware with different specifications and / or types. This means that the computing power running the AI ​​tasks includes heterogeneous computing power, which refers to the computing power provided by various hardware devices with different specifications and / or types. Specifications include storage capacity and computing power; a difference in either is considered a difference in specifications. The execution information of the AI ​​tasks is obtained. An AI task includes multiple subtasks, and correspondingly, the execution information includes at least one of the following: dependencies between subtasks, scheduling strategies for the subtasks on multiple hardware devices, or resource utilization rates of each hardware device. The dependencies between subtasks indicate whether there is a necessary temporal relationship between them. For example, if one subtask must be executed before, after, or simultaneously with another subtask, then the two subtasks are considered to have a dependency relationship. The scheduling strategy for multiple subtasks across multiple hardware components indicates which subtasks are running on each hardware component and the execution order of these subtasks. The resource utilization of each hardware component is determined based on the total available resources and the amount of resources already used. Based on the AI ​​task's runtime information, performance bottlenecks are identified, i.e., potential optimization points. Based on these performance bottlenecks, optimization suggestions are generated for the AI ​​task. These suggestions are used to improve the potential optimization points, thereby enhancing the AI ​​task's performance.

[0008] This application analyzes the runtime information of AI tasks running on heterogeneous computing power to automatically identify performance bottlenecks, eliminating the need for manual bottleneck location and improving the efficiency of bottleneck identification. Furthermore, based on the performance bottlenecks of the AI ​​task, optimization suggestions can be generated to improve its performance.

[0009] In some optional implementations of the first aspect, identifying performance bottlenecks in the AI ​​task essentially involves identifying potential optimization points. This includes identifying whether there are target subtasks among the multiple subtasks of the AI ​​task whose execution order can be adjusted, and / or whether the resource utilization of the multiple hardware components running the AI ​​task is balanced. Based on the identified potential optimization points, corresponding optimization suggestions are generated. Specifically, if the order of the target subtasks among the multiple subtasks is adjustable, and the latency of the AI ​​task after adjustment is less than the latency of the AI ​​task before adjustment, then the generated optimization suggestion indicates adjusting the execution order of the target subtasks. If the resource utilization of the multiple hardware components is uneven, then the optimization suggestion indicates that subtasks running on hardware with high resource utilization be scheduled to hardware with low resource utilization that meets the subtask's running conditions, so that the resource utilization of the multiple hardware components after scheduling is more balanced than the resource utilization of the multiple hardware components before scheduling.

[0010] In this application, the system automatically identifies optimizable points in latency and / or resource utilization of AI tasks and generates corresponding optimization suggestions, enriching the implementation methods and application scenarios of the technical solution in this application.

[0011] In some optional implementations of the first aspect, identifying the performance bottleneck of an AI task based on its runtime information can be based on the dependencies between multiple subtasks. This involves identifying a first and second subtask that have no inherent dependency. In other words, there is no necessary temporal relationship between the first and second subtasks; theoretically, the first subtask can be executed before, after, or simultaneously with the second subtask. If the scheduling strategy on multiple hardware devices instructs the first and second subtasks to execute sequentially, this indicates that during the execution of the current AI task, the second subtask is executed either after the first subtask is completed, or the first subtask is executed after the second subtask is completed. Since there is no dependency between the first and second subtasks, sequential execution actually increases latency. Therefore, identifying the performance bottleneck of an AI task includes both the first and second subtasks.

[0012] In this application, subtasks that are executed sequentially but have no dependencies can be identified as performance bottlenecks of AI tasks, thus identifying the causes of extended AI task execution time. This refines the performance bottleneck of AI tasks down to the granularity of subtasks, which not only increases the accuracy of AI task performance identification but also enhances the pertinence and effectiveness of optimization suggestions.

[0013] In some alternative implementations of the first aspect, where the first and second subtasks, which have no dependencies in the current AI task, are executed sequentially, optimization suggestions are used to indicate that the first and second subtasks be changed from sequential execution to parallel execution.

[0014] In this application, the optimization suggestion instructs the first and second subtasks, which have no dependencies in the current AI task, to be executed in parallel instead of sequentially. This reduces the total latency of the first and second subtasks, thereby reducing the latency of the AI ​​task and improving its performance.

[0015] In some optional implementations of the first aspect, the runtime information of the AI ​​task also includes the time consumed by each of the multiple subtasks. In a scheme where there is no dependency on the first subtask and there are multiple second subtasks that run sequentially with the first subtask in the current AI task, the optimization suggestion for the AI ​​task is used to indicate the subtasks to be determined in parallel with the first subtask based on the order of the time consumed by each subtask in the second subtask from longest to shortest.

[0016] In this application, in the scheme where there are multiple second subtasks, the subtasks to be executed in parallel with the first subtask are determined and modified based on the order of their execution time from longest to shortest. In other words, for these multiple second subtasks, the subtasks with longer execution times can be prioritized for parallel execution with the first subtask. This maximizes the latency reduction while maintaining the same number of modified subtasks, resulting in the best performance improvement for the AI ​​task.

[0017] In some optional implementations of the first aspect, identifying performance bottlenecks of the AI ​​task based on its runtime information can also be based on the resource utilization rate of each hardware component, determining the first hardware component whose resource utilization rate exceeds a first threshold from among multiple hardware components. Here, the first threshold represents the high-water mark of the first hardware component's resource utilization rate, indicating the maximum resource utilization rate of the first hardware component under ideal conditions. The first threshold can be preset or configured by the system; this is not limited here. Furthermore, the first threshold can be the same or different for hardware of different specifications or types; this is not limited here either.

[0018] In this application, based on the resource utilization rate of each hardware component, the hardware with excessively high resource utilization is identified as the performance bottleneck, thus identifying the cause of the unbalanced load during the current AI task. Precisely identifying the performance bottleneck of the AI ​​task down to the hardware level not only increases the accuracy of performance identification but also enhances the targeting and effectiveness of optimization suggestions.

[0019] In some alternative implementations of the first aspect, in the scheme where the resource utilization of the first hardware is greater than the first threshold, the optimization suggestion is also used to instruct the third subtask, which is scheduled to run on the first hardware according to the scheduling policy, to run on the second hardware, where the resource utilization of the second hardware is less than that of the first hardware, and the resource utilization of the second hardware after scheduling is not greater than the first threshold.

[0020] In this application, by scheduling the third subtask running on the first hardware to the second hardware, the resource utilization of the first hardware is reduced. At the same time, the optimization suggestion also ensures that the resource utilization of the second hardware after scheduling is not greater than the first threshold, and will not lead to excessively high resource utilization of the second hardware, thereby further improving the performance of the AI ​​task.

[0021] In some alternative implementations of the first aspect, the second hardware is the candidate hardware with the lowest resource utilization, and the specifications and type of the candidate hardware meet the operational requirements of the third subtask.

[0022] In this application, the hardware with the lowest resource utilization among the hardware that meets the requirements for running the third subtask is used to schedule the third subtask from the first hardware to the second hardware. Compared with scheduling it to other hardware, this is more conducive to achieving load balancing among multiple hardware running AI tasks.

[0023] In some optional implementations of the first aspect, the difference between the resource utilization of the first hardware after scheduling and the resource utilization of the second hardware after scheduling is less than a second threshold. The second threshold can be preset or configured by the system, and the second threshold indicates the range of the difference between the maximum and minimum resource utilization when multiple hardware are in a load-balanced state.

[0024] In this application, the difference between the resource utilization rate of the first hardware after scheduling and the resource utilization rate of the second hardware after scheduling is less than the second threshold, which means that the first hardware and the second hardware are load-balanced, which is more conducive to improving the performance of AI tasks.

[0025] In some optional implementations of the first aspect, the execution information of the AI ​​task also includes the resource consumption of each of the multiple subtasks, with the third subtask being the subtask with the highest resource consumption on the first hardware. That is, for the first hardware with a resource utilization rate greater than a first threshold, the third subtask scheduled for execution on the second hardware is the subtask with the highest resource consumption on the first hardware.

[0026] In this application, the subtask with the highest resource consumption on the first hardware (i.e., the third subtask) is scheduled to the second hardware. Compared with scheduling other subtasks, the resource utilization of the first hardware can be reduced more quickly, that is, the load on the first hardware is reduced, which is conducive to achieving load balancing of multiple hardware.

[0027] In some optional implementations of the first aspect, the aforementioned resource utilization rates include: memory resource utilization rate, video memory resource utilization rate, computing power resource utilization rate, etc.

[0028] In some optional implementations of the first aspect, generating optimization suggestions for the AI ​​task based on its performance bottlenecks can be achieved by inputting the performance bottlenecks into a tuning model to obtain optimization suggestions. These suggestions include optimized code and pseudocode for the AI ​​task. In other words, the optimization suggestions can be represented in code format. The optimized code is new code specifically designed for the AI ​​task, in which performance bottlenecks have already been optimized. The pseudocode indicates suggested modifications to the AI ​​task, or rather, modified code for the AI ​​task's existing code. The tuning model is trained based on performance bottleneck samples, optimization strategy samples, and AI task code samples.

[0029] In some optional implementations of the first aspect, the optimization suggestions for AI tasks can be presented in the form of text, charts, etc., in addition to code format, without being limited here.

[0030] In this application, optimization suggestions for AI tasks can be generated based on a tuning model, improving the end-to-end performance of AI tasks. Furthermore, the optimization suggestions for AI tasks can take various forms, further enriching the implementation methods and application scenarios of the technical solution and enhancing its flexibility. In solutions where the optimization suggestions for AI tasks include optimized code, since the optimized code is already modified, operations and maintenance personnel can execute it directly, requiring minimal technical expertise from them, further enhancing the practicality of the technical solution.

[0031] Secondly, this application provides an analysis apparatus for an AI task, wherein the AI ​​task is run by multiple hardware components, including various hardware components of different specifications and / or types. The apparatus includes:

[0032] The acquisition unit is used to acquire the running information of the AI ​​task. The AI ​​task includes multiple subtasks, and the running information includes at least one of the following: the dependency relationship between the multiple subtasks, the scheduling strategy of the multiple subtasks on multiple hardware, and the resource utilization rate of each hardware.

[0033] The processing unit is used to identify performance bottlenecks of AI tasks based on their runtime information.

[0034] The processing unit is also used to generate optimization suggestions for AI tasks based on performance bottlenecks.

[0035] The analysis device for AI tasks is used to implement the method shown in the first aspect or any possible implementation of the first aspect, as detailed above, and will not be repeated here.

[0036] Thirdly, this application provides a computer device including a processor and a memory, wherein the processor stores instructions that, when executed on the processor, implement the methods shown in the first aspect and any possible implementation thereof.

[0037] Fourthly, this application provides a computer program product containing instructions that, when executed on a processor, implement the method shown in the first aspect and any possible implementation of the first aspect; or, when the instructions are run by a cluster of computer devices, cause the cluster of computer devices to implement the method shown in the first aspect and any possible implementation of the first aspect.

[0038] Fifthly, this application provides a computer-readable storage medium storing computer program instructions that, when executed on a processor, implement the method shown in the first aspect and any possible implementation of the first aspect; or, when executed by a cluster of computer devices, cause the cluster of computer devices to implement the method shown in the first aspect and any possible implementation of the first aspect.

[0039] Sixthly, this application provides a chip including at least one processor and a communication interface, wherein the communication interface and the at least one processor are interconnected via a circuit, and the at least one processor is used to execute computer programs or instructions to perform the methods shown in the first aspect and any possible implementation thereof. The communication interface in the chip can be an input / output interface, pins, or circuits, etc.

[0040] In some possible implementations of the sixth aspect, the chip described above in this application further includes at least one memory storing instructions. This memory can be an internal storage unit of the chip, such as a register or cache, or it can be a storage unit of the chip itself (e.g., read-only memory, random access memory, etc.).

[0041] The beneficial effects shown in any of the second to sixth aspects are similar to those in the first aspect and any possible implementation of the first aspect, and will not be repeated here. Attached Figure Description

[0042] Figure 1 A schematic diagram of the artificial intelligence architecture provided for embodiments of this application;

[0043] Figure 2 A schematic diagram of a system architecture provided for an embodiment of this application;

[0044] Figure 3 A flowchart illustrating the AI ​​task analysis method provided in this application embodiment;

[0045] Figure 4 Another system architecture diagram provided for an embodiment of this application;

[0046] Figure 5 Another flowchart illustrating the AI ​​task analysis method provided in this application embodiment;

[0047] Figure 6 A schematic diagram provided for an embodiment of this application;

[0048] Figure 7 Another schematic diagram provided for an embodiment of this application;

[0049] Figure 8 A schematic diagram of the structure of an AI task analysis device provided in an embodiment of this application;

[0050] Figure 9 A schematic diagram of a computing device for an AI task provided in an embodiment of this application;

[0051] Figure 10 A schematic diagram of the structure of a computing device cluster for an AI task provided in an embodiment of this application;

[0052] Figure 11 This is another schematic diagram of the computing device cluster for AI tasks provided in an embodiment of this application. Detailed Implementation

[0053] This application provides an analysis method and related equipment for AI tasks, which can improve the efficiency of identifying performance bottlenecks in AI tasks and generate optimization suggestions for AI tasks, thereby improving the performance of AI tasks.

[0054] The embodiments of this application will now be described with reference to the accompanying drawings. Those skilled in the art will recognize that, with technological advancements and the emergence of new scenarios, the technical solutions provided in the embodiments of this application are equally applicable to similar technical problems.

[0055] The terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such terms are interchangeable where appropriate; this is merely a way of distinguishing objects with the same attributes in the embodiments of this application. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion, so that a process, method, system, product, or apparatus that comprises a series of units is not necessarily limited to those units, but may include other units not explicitly listed or inherent to those processes, methods, products, or apparatuses. Additionally, "at least one" means one or more, and "more than one" means two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of single or multiple items. For example, at least one of a, b, or c can be expressed as: a, b, c, ab, ac, bc, or abc, where a, b, and c can be single or multiple.

[0056] First, please refer to Figure 1 , Figure 1 This is a schematic diagram of the artificial intelligence architecture provided in an embodiment of this application. The diagram describes the overall workflow of the artificial intelligence system and is applicable to general artificial intelligence field needs.

[0057] The above-mentioned artificial intelligence framework will be elaborated from two dimensions: "intelligent information chain" (horizontal axis) and "IT value chain" (vertical axis).

[0058] The "intelligent information chain" reflects a series of processes from data acquisition to processing. For example, it could be a general process of intelligent information perception, intelligent information representation and formation, intelligent reasoning, intelligent decision-making, and intelligent execution and output. In this process, data undergoes a condensation process of "data—information—knowledge—wisdom."

[0059] The "IT value chain" reflects the value that artificial intelligence brings to the information technology industry, from the underlying infrastructure of human intelligence, information (provided and processed by technology) to the industrial ecosystem of systems.

[0060] (1) Infrastructure.

[0061] The infrastructure provides computing power to support artificial intelligence systems, enabling communication with the external world and providing support through a basic platform. Communication with the outside world is achieved through sensors; computing power is provided by intelligent chips, including central processing units (CPUs), neural network processing units (NPUs), graphics processing units (GPUs), application-specific integrated circuits (ASICs), or field-programmable gate arrays (FPGAs) and other hardware acceleration chips. The basic platform includes distributed computing frameworks and related platform guarantees and support, which may include cloud storage and computing, interconnected networks, etc. For example, sensors communicate with the outside world to acquire data, and this data is provided to intelligent chips in the distributed computing system provided by the basic platform for computation.

[0062] (2) Data.

[0063] The data at the next layer of infrastructure is used to represent the data sources in the field of artificial intelligence. The data involves graphics, images, voice, text, and IoT data from traditional devices, including business data from existing systems and sensor data such as force, displacement, liquid level, temperature, and humidity.

[0064] (3) Data processing.

[0065] Data processing typically includes methods such as data training, machine learning, deep learning, search, reasoning, and decision-making.

[0066] Among them, machine learning and deep learning can perform intelligent information modeling, extraction, preprocessing, and training on data, including symbolization and formalization.

[0067] Reasoning refers to the process in which, in a computer or intelligent system, the machine thinks and solves problems by simulating human intelligent reasoning, based on reasoning control strategies and using formalized information. Typical functions include search and matching.

[0068] Decision-making refers to the process of making decisions based on intelligent information after reasoning, and it typically provides functions such as classification, sorting, and prediction.

[0069] (4) General ability.

[0070] After the data processing mentioned above, the results of the data processing can be used to form some general capabilities, such as algorithms or a general system, for example, translation, text analysis, computer vision processing, speech recognition, image recognition, etc.

[0071] (5) Smart products and industry applications.

[0072] Intelligent products and industry applications refer to products and applications of artificial intelligence systems in various fields. They are the encapsulation of overall artificial intelligence solutions, productizing intelligent information decision-making and realizing practical applications. Their application areas mainly include: intelligent manufacturing, intelligent transportation, smart home, intelligent healthcare, intelligent security, autonomous driving, smart cities, and intelligent terminals.

[0073] The AI ​​task mentioned in this application refers to a task or event completed based on AI technology, such as training or inference of an AI model. AI tasks are applied in multiple fields, such as image recognition, autonomous driving, natural language processing, and robot interaction. This application does not limit the application scenarios of AI tasks.

[0074] In some optional implementations, the AI ​​task analysis method provided in this application can also be applied to the field of cloud computing, combined with... Figure 2 This section describes the system architecture of cloud computing. Please refer to [link / reference]. Figure 2 , Figure 2 This is a schematic diagram of the system architecture provided for an embodiment of this application.

[0075] like Figure 2 As shown, the tenant logs into the cloud service platform 203 via client 201 and the Internet 202 using the account and password registered on the cloud service platform 203. The cloud service platform 203 manages the infrastructure, which includes multiple data centers located in different regions, such as... Figure 1 Region 1, as shown, includes Cloud Data Center 1 and Cloud Data Center 2, while Region 2 includes Cloud Data Center 2 and Cloud Data Center 4. Each cloud data center has multiple servers, on which business instances (including at least one of virtual machines, containers, and dedicated hosts) run.

[0076] In this embodiment, an AI task analysis service is deployed in the business instance. Tenants can purchase or use this cloud service through a client on the cloud service platform 203. Specifically, the tenant sends a call request to the cloud service platform 203, which requests the cloud service, i.e., requests analysis of the AI ​​task, including identifying performance bottlenecks of the AI ​​task and generating optimization suggestions for the AI ​​task. The specific implementation of this service will be described later.

[0077] Please refer to the following. Figure 3 , Figure 3 A flowchart illustrating the AI ​​task analysis method provided in this application embodiment includes:

[0078] 301. Obtain the running information of the AI ​​task, which includes multiple sub-tasks.

[0079] In this embodiment, the computing power for running AI tasks includes heterogeneous computing power, which refers to the computing power provided by various hardware with different specifications and / or types. In other words, the AI ​​task is run by multiple hardware devices, which include various hardware devices with different specifications and / or types.

[0080] Specifications include storage capacity and computing power; differences in either are considered differences in specifications. For example, one central processing unit (CPU) has 2GB of storage, while another CPU has 10GB. These two CPUs are of different specifications, providing heterogeneous computing power. For example, hardware computing power can be represented by floating-point performance, specifically floating-point operations per second (FLOPS). The A100's graphics processing unit (GPU) has a floating-point performance of 19.5 TFLOPS, while the A100's GPU has a floating-point performance of 10.6 TFLOPS; these are GPUs of different specifications.

[0081] Additionally, for example, CPUs, GPUs, and neural network processing units (NPUs) are different types of hardware.

[0082] In the embodiments of this application, the AI ​​task includes multiple subtasks. A subtask refers to an AI task that includes modules, units, functions, network layers, etc., and each subtask can complete a part of the functions in the AI ​​task. For example, if the AI ​​task is an image recognition task, then the subtasks may include data collection subtasks, data preprocessing subtasks, image recognition subtasks, recognition result output subtasks, etc.

[0083] The AI ​​task analysis device acquires the operation information of the AI ​​task, specifically including at least one of the following: the dependency relationship between multiple subtasks, the scheduling strategy of multiple subtasks on multiple hardware, and the resource utilization rate of each hardware.

[0084] The dependency relationships between multiple subtasks indicate whether a dependency exists between different subtasks. Dependencies can also be understood as temporal relationships, meaning whether there is a necessary temporal order between different subtasks. For example, if subtask 1 must be executed before subtask 2, then there is a dependency between subtask 1 and subtask 2. If subtask 1 can be executed concurrently with subtask 2, or can be executed before or after subtask 2, then there is no dependency between subtask 1 and subtask 2.

[0085] The scheduling strategy for multiple subtasks across multiple hardware devices indicates which subtasks are running on each of the multiple hardware devices during the execution of the current AI task, and the execution order of these subtasks.

[0086] Resource utilization rate for each hardware component indicates the proportion of resources used on each hardware component relative to the total resources of that hardware. In this embodiment, resource utilization rate includes: memory resource utilization rate, video memory resource utilization rate, computing power resource utilization rate, etc.

[0087] 302. Identify performance bottlenecks of AI tasks based on their runtime information.

[0088] After acquiring the operational information of the AI ​​task, the analysis device processes this information and automatically identifies the performance bottlenecks. In other words, it identifies potential optimization points for the AI ​​task, or the causes of its performance bottlenecks.

[0089] In summary, identifying performance bottlenecks in AI tasks includes identifying whether there are target subtasks among the multiple subtasks of the AI ​​task whose execution order can be adjusted, and / or identifying whether the resource utilization of the multiple hardware components running the AI ​​task is balanced. In solutions where there are target subtasks whose execution order can be adjusted, these target subtasks can be considered the cause of high latency in the AI ​​task. In solutions where the resource utilization of multiple hardware components is uneven, the uneven hardware components can be considered the cause of uneven hardware load on the AI ​​task, thus leading to reduced performance.

[0090] In some optional implementations, the AI ​​task analysis device identifies a first subtask and a second subtask that have no dependencies on each other, based on the dependencies between multiple subtasks. That is, the first and second subtasks do not necessarily have a sequential order of execution; the first subtask can execute before, after, or simultaneously with the second subtask. In a scheduling strategy across multiple hardware components that sequentially executes the first and second subtasks, this means that during the execution of the current AI task, the first subtask may begin execution after the second subtask has completed, or vice versa. This leads to unnecessary waiting and latency during the AI ​​task's execution. Therefore, in this case, the AI ​​task analysis device identifies the first and second subtasks as performance bottlenecks. The target subtask mentioned earlier also includes the first and second subtasks.

[0091] In this embodiment, subtasks that are executed sequentially but have no dependencies can be identified as performance bottlenecks of AI tasks, thus identifying the reasons for the extended execution time of AI tasks. By pinpointing the performance bottleneck of AI tasks to the granularity of subtasks, not only is the accuracy of performance identification of AI tasks increased, but the pertinence and effectiveness of optimization suggestions are also improved.

[0092] In some alternative implementations, the AI ​​task analysis device identifies a first piece of hardware with a resource utilization rate greater than a first threshold from among multiple pieces of hardware, based on the resource utilization rate of each piece of hardware. This means that the performance bottleneck of the AI ​​task is identified as including the first piece of hardware.

[0093] The first threshold is used to represent the high-water mark of resource utilization of the first hardware, representing the maximum resource utilization of the first hardware under ideal conditions. The first threshold can be preset or configured by the system; there is no specific limitation here. In addition, each hardware has a corresponding first threshold. Multiple first hardware with the same specifications and types can have the same first threshold, and hardware with different specifications or types can have the same or different first thresholds; there is no specific limitation here.

[0094] For example, assuming CPU1 and CPU2 have different specifications, with CPU2 having greater storage capacity than CPU1, then a first threshold for CPU2 can be set higher than that for CPU1. For instance, the first threshold for CPU1 could be 80%, and for CPU2, 85%. That is, if CPU1's storage resource utilization exceeds 80%, CPU1 is considered the primary hardware bottleneck for AI tasks. Similarly, for CPU2, its storage resource utilization must exceed 85% for it to be considered the primary hardware bottleneck for AI tasks. This approach takes into account that different specifications or types of hardware may have different achievable resource utilization limits. Setting different first thresholds for different hardware better reflects the characteristics of the hardware and further enhances the practicality of the technical solution.

[0095] In this embodiment, based on the resource utilization rate of each hardware component, the hardware with excessively high resource utilization is identified as the performance bottleneck, thus identifying the cause of the unbalanced load during the current AI task's operation. Precisely identifying the performance bottleneck of the AI ​​task down to the hardware level not only increases the accuracy of performance identification but also enhances the targeting and effectiveness of optimization suggestions.

[0096] 303. Based on the performance bottlenecks of AI tasks, generate optimization suggestions for AI tasks.

[0097] After identifying performance bottlenecks in an AI task, the analysis device generates optimization suggestions based on these bottlenecks. In short, it inputs the performance bottlenecks into a tuning model to obtain optimization suggestions. These suggestions can take the form of code, pseudocode, text, charts, etc.

[0098] In this context, "code" refers to the optimized code for the AI ​​task, which can be understood as the complete, optimized code for the AI ​​task. Running this code will execute the optimized AI task. "Pseudocode" can be understood as suggested modifications to the original AI task's code, used to modify the original AI task's code. Based on the logic of the pseudocode, operations personnel can modify the original AI task's code. "Optimization suggestions" in text or image format provide easily understandable suggestions for modifying the AI ​​task; however, these still require action from operations personnel to optimize the original AI task.

[0099] Furthermore, the optimization model is trained based on performance bottleneck samples, optimization strategy samples, and AI task code samples. Performance bottleneck samples and optimization strategy samples can be understood as the parameter set and ground truth for training the optimization model, respectively. Through iterative training, the optimization model learns optimization suggestions for different performance bottlenecks. AI task code samples provide the optimization model with code format and content samples specific to AI tasks, allowing the model to understand what AI task code looks like during training, thus enabling it to output optimized code or pseudocode tailored to the AI ​​task.

[0100] The performance bottleneck samples and optimization strategy samples can be derived from expert experience bases, network knowledge, professional literature, and algorithm experience bases. During the training process of the optimization model, professionals can adjust and provide feedback on the optimization strategies output by the model, continuously improving the accuracy of the optimization model.

[0101] In this embodiment, optimization suggestions for AI tasks can be generated based on the tuning model, improving the end-to-end performance of the AI ​​tasks. Furthermore, the optimization suggestions for AI tasks can take various forms, further enriching the implementation methods and application scenarios of the technical solution and enhancing its flexibility. In the scheme where the optimization suggestions for AI tasks include optimized code, since the optimized code is already modified, operations and maintenance personnel can execute it directly, without requiring a high level of technical expertise from them, further enhancing the practicality of the technical solution.

[0102] In this embodiment, the AI ​​task analysis device can provide different optimization suggestions for different types of AI task performance bottlenecks. These are described below:

[0103] In some alternative implementations, as described above, if the performance bottleneck of the AI ​​task includes a first subtask and a second subtask that have no dependencies but are executed sequentially during the current AI task runtime, then an optimization suggestion is made to instruct the first and second subtasks to be changed from sequential execution to parallel execution.

[0104] Optionally, when changing the execution of the first and second subtasks from serial to parallel, the subtask that was originally executed later can be scheduled forward, or the two subtasks can be scheduled to the middle time period. There is no specific limitation here, as long as the total execution time of the first and second subtasks after the modification is less than the total execution time of the first and second subtasks before the modification.

[0105] In this application, the optimization suggestion instructs the first and second subtasks, which have no dependencies in the current AI task, to be executed in parallel instead of sequentially. This reduces the total latency of the first and second subtasks, thereby reducing the latency of the AI ​​task and improving its performance.

[0106] In some optional implementations, the execution information of the AI ​​task also includes the execution time of each of the multiple subtasks, that is, the duration of each subtask from start to finish. The number of second subtasks, which are independent of the first subtask and run sequentially with it in the current AI task, can be multiple. In this approach, the optimization suggestions for the AI ​​task are used to indicate which subtasks should run concurrently with the first subtask, based on the order of their execution times from longest to shortest.

[0107] For example, suppose the first subtask is subtask 1 executed on CPU1, and subtasks 2 through 4 are executed on GPU1, GPU2, and GPU3 respectively. Subtask 1 is executed sequentially with any of subtasks 2 through 4, and there is no dependency between subtask 1 and subtasks 2 through 4. If there is no dependency between any two subtasks in subtask 2 and subtask 4, then the optimization suggestion can instruct these four subtasks to be executed in parallel. If subtasks 3 and 4 must be executed after subtask 2, and the execution time of subtasks 3 and 4 is greater than the execution time of subtask 2, then the optimization suggestion can instruct subtasks 1, 3, and 4 to be executed in parallel. Furthermore, if the execution time of subtask 1 is greater than the execution time of subtask 2 plus the execution time of subtasks 3 and 4 in parallel, then the optimization suggestion can instruct subtasks 2 to be executed sequentially with subtasks 3 and 4, subtasks 3 and 4 in parallel, and subtask 1 to be parallel relative to subtasks 2 through 4.

[0108] In summary, in scenarios where there is no dependency on the first subtask and there are multiple second subtasks that run sequentially with the first subtask in the current AI task, there are various possible optimization suggestions, all of which can shorten the latency of the AI ​​task.

[0109] In this embodiment of the application, in a scheme where there are multiple second subtasks, the subtasks to be executed in parallel with the first subtask are determined and modified based on the order of the execution time of each subtask from longest to shortest. That is, for these multiple second subtasks, the subtask with the longest execution time can be selected to be executed in parallel with the first subtask. In this way, with the same number of modified subtasks, the latency reduction is the greatest, and the performance improvement of the AI ​​task is the best.

[0110] In some optional implementations, as described above, if the performance bottleneck of the AI ​​task includes a first hardware component whose resource utilization exceeds a first threshold, then an optimization suggestion is made to instruct the third subtask, as indicated by the scheduling policy, to be scheduled to run on the first hardware instead of the second hardware. Specifically, the resource utilization of the second hardware is less than that of the first hardware, and after scheduling, the resource utilization of the second hardware does not exceed the first threshold. Furthermore, the specifications and type of the second hardware both meet the operational requirements of the third subtask.

[0111] In other words, when the resource utilization of the first hardware exceeds its high watermark, some subtasks on the first hardware are scheduled to be executed on the second hardware, thereby reducing the resource utilization of the first hardware and increasing the resource utilization of the second hardware. This narrows the difference in resource utilization between the first and second hardware, which is beneficial for achieving load balancing between different hardware.

[0112] In this application, by scheduling the third subtask running on the first hardware to the second hardware, the resource utilization of the first hardware is reduced. At the same time, the optimization suggestion also ensures that the resource utilization of the second hardware after scheduling is not greater than the first threshold, and will not lead to excessively high resource utilization of the second hardware, thereby further improving the performance of the AI ​​task.

[0113] In the embodiments of this application, there are multiple possibilities for the second hardware or the third subtask. The possible solutions are described below:

[0114] Optionally, the second hardware is the candidate hardware with the lowest resource utilization, and the specifications and type of the candidate hardware meet the operational requirements of the third subtask. For example, suppose the candidate hardware includes hardware 1 to hardware 3, where hardware 3 has the lowest resource utilization. Then, scheduling the third subtask from the first hardware to hardware 3 for execution reduces the resource utilization of the first hardware while increasing the resource utilization of hardware 3, thereby reducing the difference in resource utilization between the first hardware and hardware 3.

[0115] In this embodiment, the hardware with the lowest resource utilization among the hardware that meets the requirements for running the third subtask is used to schedule the third subtask from the first hardware to the second hardware. Compared with scheduling it to other hardware, this is more conducive to achieving load balancing among multiple hardware running AI tasks.

[0116] Optionally, the difference between the resource utilization of the first hardware after scheduling and the resource utilization of the second hardware after scheduling is less than a second threshold. The second threshold indicates the range of the difference between the maximum and minimum resource utilization when multiple hardware are under load balancing. The second threshold can be preset or set by the system, and its specific value can also be set based on the needs of the actual application; no specific limitation is made here.

[0117] For example, the second threshold could be 5%, 8%, 10%, etc. Assuming that after scheduling, the resource utilization rate of the first hardware is 60% and the resource utilization rate of the second hardware is 55%, then the difference between these two resource utilization rates is 5%.

[0118] In this embodiment, the difference between the resource utilization rate of the first hardware after scheduling and the resource utilization rate of the second hardware after scheduling is less than the second threshold, which means that the first hardware and the second hardware are load-balanced, which is more conducive to improving the performance of AI tasks.

[0119] Optionally, the AI ​​task's runtime information also includes the resource consumption of each of the multiple subtasks, with the third subtask being the one with the highest resource consumption on the first hardware. This means that when the resource utilization of the first hardware exceeds a first threshold, scheduling the subtask with the highest resource consumption on that hardware to the second hardware can quickly reduce the resource utilization of the first hardware.

[0120] In this embodiment, the subtask with the highest resource consumption on the first hardware (i.e., the third subtask) is scheduled to the second hardware. Compared with scheduling other subtasks, the resource utilization of the first hardware can be reduced more quickly, that is, the load of the first hardware is reduced, which is conducive to achieving load balancing of multiple hardware.

[0121] The previous example briefly introduced the principle of determining the third subtask and the second hardware indicated by the optimization suggestion when the resource utilization of the first hardware is greater than the first threshold. The following will further explain this with specific examples.

[0122] For example, assuming the first threshold is 75%, in the current AI task, the resource utilization rate of the first hardware (hardware 1) is 80%. Subtasks 1 to 4 run on hardware 1, and the resource consumption of these four subtasks on the first hardware accounts for 10%, 15%, 25%, and 30% of the first hardware's resources, respectively. Hardware 2 to 4 are hardware of the same specifications and type as hardware 1, and their corresponding resource threshold is also 75%. In the current AI task, the resource utilization rates of hardware 2 to 4 are 30%, 50%, and 60%, respectively. Then, the second hardware and the third subtask have several possibilities:

[0123] Optionally, the third subtask can be subtask 4, and the second hardware can be hardware 2. After scheduling subtask 4 to hardware 2 for execution, the resource utilization of hardware 1 drops to 50%, and the resource utilization of hardware 2 rises to 60%, neither exceeding the first threshold. This scheduling ensures that the resource utilization of each hardware component does not exceed the first threshold, preventing overload.

[0124] Optionally, the third subtask can be subtask 1, and the second hardware can be hardware 4. When subtask 1 is scheduled to be executed on hardware 3, the resource utilization of hardware 1 drops to 70%, and the resource utilization of hardware 4 rises to 70%, neither of which exceeds the first threshold, and the resource utilization of hardware 1 and hardware 4 is the same.

[0125] Please see Figure 4 , Figure 4 This is a schematic diagram of the system architecture provided for an embodiment of this application. Figure 4 As shown, the system architecture includes a profile module, an automatic location module, and an automatic tuning module.

[0126] The Profile module collects runtime information for AI tasks and sends this information to the automatic localization module. The automatic localization module identifies performance bottlenecks in the AI ​​tasks based on this runtime information. The automatic tuning module generates optimization suggestions for the AI ​​tasks based on these performance bottlenecks. These are explained in detail below:

[0127] The automatic location module includes a dependency analysis submodule, a process time analysis submodule, a task scheduling analysis submodule, and a consumption analysis submodule. Taking the AI ​​task execution information output by the Profiler module as input, the automatic location module calls the aforementioned four submodules respectively, ultimately locating and outputting potential bottlenecks and information on optimizable aspects (such as dependencies, time consumption, and computing power consumption). The detailed functions of the four submodules will be described below:

[0128] The dependency analysis submodule takes the dependencies of multiple subtasks in the current AI task provided by the profiler module as input. It first analyzes and organizes the dependencies between subtasks within each hardware unit, such as the execution order of multiple subtasks in the GPU. Then, it organizes the dependencies between subtasks across different hardware units and across the cluster, such as the dependency between data processing in the CPU and model execution in the GPU. After analysis, the dependency analysis submodule outputs the subtasks in the current AI task whose execution order can be adjusted.

[0129] The process time consumption analysis submodule takes the time consumption of each subtask of the current AI task and the output of the dependency relationship submodule provided by the profiler module as input. It identifies the most time-consuming subtasks or processes by analyzing the time consumption within a single hardware component and the time consumption of cross-hardware communication (such as communication time between different GPUs, or communication time between the CPU and GPU). For example: 1. The total time consumption of sequential execution of modules with no dependencies between the CPU and GPU; 2. The time consumption of extensive communication between GPUs and CPUs or between GPUs. It can also analyze whether different subtasks can be executed sequentially. Finally, it outputs the subtasks in the current AI task whose time consumption can be reduced.

[0130] The task scheduling analysis submodule analyzes the scheduling strategies of multiple subtasks across multiple hardware components. This includes analyzing the scheduling strategies within and between different hardware components, identifying and finding idle or unevenly utilized parts, and identifying potential optimization bottlenecks.

[0131] The resource consumption analysis submodule takes the resource consumption of each subtask and the resource utilization rate of each hardware as input, and combines them with the characteristics of the current AI task to identify and locate whether there are unreasonable resource consumption situations under the current task (such as exceeding or falling below a certain threshold).

[0132] Based on the functions of these four sub-modules, the automatic localization module will identify bottlenecks in AI tasks under heterogeneous computing. In some optional implementations, the automatic localization module can also perform... Figure 5 The process is shown below. Please refer to [link / reference]. Figure 5 , Figure 5 This is a flowchart illustrating the AI ​​task analysis method provided in an embodiment of this application.

[0133] like Figure 5As shown, the automatic localization module analyzes the dependencies between subtasks and the time consumption of each subtask, and then determines whether a subtask F exists. Subtask F refers to a task that depends on other subtasks and has a time gap between it and the dependent subtasks. For example, subtask 1 and subtask 2 are dependent on each other, meaning that subtask 2 must be executed after subtask 1. However, in the current AI task, the time gap between subtask 1 and subtask 2 is greater than the time gap threshold, indicating that the time gap between these two subtasks is too long. The automatic localization module further identifies whether the execution order and number of times of subtask 2 can be changed. If they can be changed, subtask 2 is output as subtask F. Specifically, the execution order of subtask 2 can be moved forward, thereby reducing the time gap between subtask 1 and subtask 2, which can also help shorten the latency of the AI ​​task. If the execution order and number of times of subtask 2 cannot be changed, it means that subtask 2 may also have dependencies on other subtasks. In this case, the process returns to step 3, which is to re-identify a new subtask F. In other words, step 3 is actually about starting with the first subtask of the AI ​​task and identifying the subtasks F in the AI ​​task in sequence.

[0134] Next, we will introduce the automatic tuning module. This module includes a tuning model, which is trained based on data from a database. The database contains performance bottleneck samples, optimization suggestion samples, and AI task code. Data sources for the database include expert experience bases, algorithm experience bases, and network databases. During the training process of the tuning model, professional operations and maintenance personnel can adjust and provide feedback on the optimization suggestions output by the model, iterating and upgrading the model to improve the quality of the final output optimization suggestions.

[0135] Next, please refer to Figure 6 and Figure 7 , Figure 6 and Figure 7 This is a schematic diagram provided for an embodiment of this application.

[0136] exist Figure 6In the illustrated embodiment, a large model inference task is used as the AI ​​task. The execution information of the AI ​​task is analyzed. The automatic localization module in the AI ​​task analysis device analyzes the dependencies between different subtasks within the CPU, GPU, and between the CPU and GPU. It identifies that in the current AI task, the model execution subtask in the GPU has no dependency on the decoding + other processing subtasks and result transmission subtasks in the CPU's post-processing section. Through timing analysis and task flow judgment, the automatic localization module concludes that the time occupied by this part of post-processing can be covered by the model execution subtask, meaning these two parts can be executed in parallel. Therefore, based on the execution strategy of the current AI task, a problem is identified that can be optimized.

[0137] The automatic positioning module inputs the identified problem into the automatic optimization module, which then calls the optimization model to provide suggested solutions for the current bottleneck problem. Specifically, it can be like this: Figure 6 As shown in the diagram below the arrow: the decoding, other processing, and result transmission subtasks in the CPU's post-processing are processed in parallel with the model execution subtasks in the GPU, achieving time reuse in heterogeneous computing and improving performance.

[0138] During the original AI task execution, the scheduling subtask and input processing subtask took 9.3% of the total time, the model execution subtask took 35.62%, the sampling subtask took 40.12%, the decoding + other processing subtask took 11.93%, and the result transmission subtask took 2.81%. After the aforementioned optimization, the actual test results showed the following optimization effects: post-processing decoding time was reduced by 100%, end-to-end throughput was increased by 12% to 20%, and inference incremental latency was reduced by about 15%.

[0139] exist Figure 7 In the illustrated embodiment, the training task of a large model is used as the AI ​​task. The runtime information of the AI ​​task is analyzed. The automatic localization module in the AI ​​task analysis device analyzes the dependencies between different subtasks within the CPU, GPU, between CPU and GPU, and between GPUs. It identifies the following subtasks that have no dependencies: 1) Gradient update in the CPU and the feature fusion module (image neck) in the GPU; 2) I / O operations in the CPU and the god depth net in the GPU; 3) The god depth net of GPU1 and the navigation backbone nets of GPU2 and GPU3. Further time-consuming analysis reveals that these three potential optimization points all have the potential to be converted from serial to parallel processing.

[0140] The automatic positioning module inputs the aforementioned three optimizable parameters into the automatic tuning module. The automatic tuning module then calls the tuning model and provides suggested solutions to the current bottleneck problem. Specifically, this can be done as follows: Figure 7 The diagram below the arrow shows: 1) Parallelize gradient updates in the CPU and feature fusion modules in GPU3 as much as possible. Ensure gradient updates execute after the bird's-eye view (BEV) network, and that the latency between the CPU and GPU meets hardware communication requirements. 2) Parallelize read / write operations in the CPU and the pyramid network in the GPU to achieve time reuse and improve performance. It's understandable that although the pyramid network in GPU1 and the navigation networks in GPU2 and GPU3 are independent, the navigation network depends on read / write operations and therefore must run after these operations.

[0141] Next, please refer to Figure 8 , Figure 8 This is a schematic diagram of the structure of the AI ​​task analysis device provided in an embodiment of this application. The AI ​​task is run by multiple hardware components, which include various hardware components of different specifications and / or types, such as… Figure 8 As shown, the AI ​​task analysis device 800 includes an acquisition unit 801 and a processing unit 802.

[0142] In some optional implementations, the acquisition unit 801 is used to acquire the running information of the AI ​​task, which includes multiple subtasks. The running information includes at least one of the following: the dependency relationship between the multiple subtasks, the scheduling strategy of the multiple subtasks on multiple hardware, and the resource utilization rate of each hardware.

[0143] The processing unit 802 is used to identify performance bottlenecks in AI tasks based on their runtime information. Based on these bottlenecks, it generates optimization suggestions for the AI ​​tasks.

[0144] In some optional implementations, processing unit 802 is specifically configured to: determine a first subtask and a second subtask that do not have a dependency relationship among the multiple subtasks based on the dependencies between the multiple subtasks. If the scheduling policy of the multiple subtasks on multiple hardware devices indicates that the first subtask and the second subtask are sequential, then the performance bottleneck of the AI ​​task is identified to include the first subtask and the second subtask.

[0145] In some alternative implementations, optimization suggestions are provided to indicate that the first and second subtasks be changed from serial execution to parallel execution.

[0146] In some optional implementations, the AI ​​task's runtime information also includes the time taken for each of the multiple subtasks, with the number of second subtasks being multiple. Optimization suggestions are used to indicate which subtasks can be selected to run concurrently with the first subtask, based on the order of the longest to shortest time taken for each subtask in the second subtask.

[0147] In some alternative implementations, the processing unit 802 is specifically configured to determine, based on the resource utilization of each piece of hardware, a first piece of hardware whose resource utilization is greater than a first threshold from a plurality of pieces of hardware.

[0148] In some alternative implementations, the optimization suggestion is also used to instruct the third subtask, which is scheduled to run on the first hardware as indicated by the scheduling policy, to run on the second hardware, where the resource utilization of the second hardware is less than that of the first hardware, and the resource utilization of the second hardware after scheduling is not greater than a first threshold.

[0149] In some alternative implementations, the second hardware is the candidate hardware with the lowest resource utilization, and the specifications and type of the candidate hardware meet the operational requirements of the third subtask.

[0150] In some alternative implementations, the difference between the resource utilization rate of the first hardware after scheduling and the resource utilization rate of the second hardware after scheduling is less than a second threshold.

[0151] In some optional implementations, the runtime information of the AI ​​task also includes the resource consumption of each of the multiple subtasks, with the third subtask being the subtask with the largest resource consumption on the first hardware.

[0152] In some optional implementations, processing unit 802 is specifically used to: input performance bottlenecks into the tuning model to obtain optimization suggestions, which include optimized code and pseudocode for the AI ​​task. The tuning model is trained based on performance bottleneck samples, optimization strategy samples, and AI task code samples.

[0153] The AI ​​task analysis device 800 is used to implement the AI ​​task analysis method provided in the embodiments of this application, as detailed above, and will not be repeated here.

[0154] Both the acquisition unit 801 and the processing unit 802 can be implemented in software or in hardware. For example, the implementation of the processing unit 802 will be described below. Similarly, the implementation of the acquisition unit 801 can be referenced to the implementation of the processing unit 802.

[0155] As an example of a software functional unit, processing unit 802 may include code running on a computing instance. The computing instance may include at least one of a physical host (computing device), a virtual machine, and a container. Further, the aforementioned computing instance may be one or more. For example, processing unit 802 may include code running on multiple hosts / virtual machines / containers. It should be noted that the multiple hosts / virtual machines / containers used to run the code may be distributed in the same region or in different regions. Further, the multiple hosts / virtual machines / containers used to run the code may be distributed in the same availability zone (AZ) or in different AZs, each AZ including one or more geographically proximate data centers. Typically, a region may include multiple AZs.

[0156] Similarly, multiple hosts / virtual machines / containers used to run this code can be distributed within the same Virtual Private Cloud (VPC) or across multiple VPCs. Typically, a VPC is set up within a region. Communication between two VPCs within the same region, as well as between VPCs in different regions, requires a communication gateway to be set up within each VPC to enable interconnection between VPCs.

[0157] As an example of a hardware functional unit, the processing unit 802 may include at least one computing device, such as a server. Alternatively, the processing unit 802 may also be a device implemented using an application-specific integrated circuit (ASIC) or a programmable logic device (PLD). The PLD may be implemented using a complex programmable logical device (CPLD), a field-programmable gate array (FPGA), generic array logic (GAL), or any combination thereof.

[0158] The processing unit 802 includes multiple computing devices that can be distributed in the same region or in different regions. Similarly, the processing unit 802 can be distributed within the same Availability Zone (AZ) or in different AZs. Likewise, the processing unit 802 can be distributed within the same Virtual Private Cloud (VPC) or in multiple VPCs. These multiple computing devices can be any combination of computing devices such as servers, ASICs, PLDs, CPLDs, FPGAs, and GALs.

[0159] Please see Figure 9 , Figure 9 This is a schematic diagram of a computing device provided in an embodiment of this application. The computing device 900 includes a processor 901, a communication interface 902, a bus 903, and a memory 904. The processor 901, the communication interface 902, and the memory 904 communicate with each other via the bus 903. In practical applications, communication can also be achieved through other means such as wireless transmission; the specific method is not limited here.

[0160] The computing device 900 may be a server or a terminal device. It should be understood that this application does not limit the number of processors and memory in the computing device 900.

[0161] Processor 901 may include any one or more processors such as a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP).

[0162] The communication interface 902 uses transceiver modules, such as, but not limited to, network interface cards and transceivers, to enable communication between the computing device 900 and other devices or communication networks.

[0163] The 903 bus can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of representation, Figure 9 The bus 903 is represented by only one line, but this does not mean that there is only one bus or one type of bus. The bus 903 may include a path for transmitting information between various components of the computing device 900 (e.g., memory 904, processor 901, communication interface 902).

[0164] Memory 904 may include volatile memory, such as random access memory (RAM). Memory 904 may also include non-volatile memory, such as read-only memory (ROM), flash memory, hard disk drive (HDD), or solid state drive (SSD).

[0165] Optionally, the memory 904 stores executable program code, and the processor 901 executes the executable program code to implement the functions of the aforementioned acquisition unit 801 and processing unit 802, thereby realizing the analysis method for the AI ​​task. That is, the memory 904 stores instructions for executing the analysis method for the AI ​​task.

[0166] This application also provides a computing device cluster, which includes at least one computing device. The computing device can be a server, such as a central server, an edge server, or a local server in a local data center. In some optional embodiments, the computing device can also be a terminal device such as a desktop computer, a laptop computer, or a smartphone.

[0167] Please see Figure 10 and Figure 11 , Figure 10 and Figure 11 These are all schematic diagrams of the computing device clusters provided in the embodiments of this application.

[0168] like Figure 10 As shown, the computing device cluster includes at least one computing device 900. The memory 904 of one or more computing devices 900 in the computing device cluster may store the same instructions for performing the analysis method for the AI ​​tasks provided in the embodiments of this application.

[0169] In some possible implementations, the memory 904 of one or more computing devices 900 in the computing device cluster may also store partial instructions for the analysis method used to perform AI tasks. In other words, a combination of one or more computing devices 904 can jointly execute the instructions for the analysis method used to perform AI tasks.

[0170] It should be noted that the memory 904 in different computing devices 900 within the computing device cluster can store different instructions, which are used to execute certain functions of the analysis device for AI tasks. That is, the instructions stored in the memory 904 of different computing devices 900 can implement the functions of one or more units among the acquisition unit 801 and the processing unit 802.

[0171] In some possible implementations, one or more computing devices in a computing device cluster can be connected via a network. This network can be a wide area network (WAN) or a local area network (LAN), etc. Figure 11 One possible implementation is shown. For example... Figure 11 As shown, two computing devices 900A and 900B are connected via a network. Specifically, they are connected to the network through communication interfaces in each computing device. In this possible implementation, the memory 904 in computing device 900A stores instructions for executing the functions of the acquisition unit 801. Simultaneously, the memory 904 in computing device 900B stores instructions for executing the functions of the processing unit 802.

[0172] Figure 11 The connection method between the computing device clusters shown can be based on the analysis method of the AI ​​task provided in this application, in which the processing operation and the operation other than the processing operation are executed separately. That is, it is considered that the function of the acquisition unit 801 is executed by the computing device 900A, and the function of the processing unit 802 is executed by the computing device 900B.

[0173] It should be understood that Figure 11 The functions of the computing device 900A shown can also be performed by multiple computing devices 900. Similarly, the functions of the computing device 900B can also be performed by multiple computing devices 900.

[0174] This application also provides another computing device cluster. The connection relationships between the computing devices in this computing device cluster can be similarly referred to... Figure 10 and Figure 11 The connection method of the computing device cluster will not be described in detail here.

[0175] This application also provides a computer program product containing instructions. The computer program product may be a software or program product containing instructions, capable of running on a computing device or stored on any usable medium. When the computer program product is run on at least one computer device, it causes the at least one computer device to perform the analysis method for the aforementioned AI task.

[0176] This application also provides a computer-readable storage medium. The computer-readable storage medium can be any available medium that a computing device can store, or a data storage device such as a data center that includes one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state drive). The computer-readable storage medium includes instructions that instruct the computing device to perform the analysis method for the aforementioned AI task.

[0177] It should also be noted that the device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. In addition, in the device embodiment drawings provided in this application, the connection relationship between modules indicates that they have a communication connection, which can be implemented as one or more communication buses or signal lines.

[0178] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0179] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the protection scope of the technical solutions of the embodiments of this application.

Claims

1. A method for analyzing AI tasks, characterized in that, The AI ​​task is run by multiple hardware components, which include various hardware components with different specifications and / or types. The method includes: Obtain the operation information of the AI ​​task, which includes multiple subtasks. The operation information includes at least one of the following: the dependency relationship between the multiple subtasks, the scheduling strategy of the multiple subtasks on the multiple hardware, and the resource utilization rate of each hardware. Based on the operational information of the AI ​​task, identify the performance bottleneck of the AI ​​task; Based on the performance bottlenecks of the AI ​​task, optimization suggestions for the AI ​​task are generated.

2. The method according to claim 1, characterized in that, The process of identifying performance bottlenecks in the AI ​​task based on its runtime information includes: Based on the dependencies between the multiple subtasks, determine the first and second subtasks that have no dependencies among the multiple subtasks; If the scheduling policy of the multiple subtasks on the multiple hardware indicates that the first subtask and the second subtask are sequential, then the performance bottleneck of the AI ​​task is identified as including the first subtask and the second subtask.

3. The method according to claim 2, characterized in that, The optimization suggestion is used to indicate that the first subtask and the second subtask should be changed from serial execution to parallel execution.

4. The method according to claim 2, characterized in that, The operation information of the AI ​​task also includes the time consumed by each of the multiple subtasks, and the number of the second subtask is multiple; The optimization suggestion is used to indicate which subtasks can be paralleled to the first subtask based on the order of the time taken by each subtask in the second subtask from longest to shortest.

5. The method according to any one of claims 1 to 4, characterized in that, The process of identifying performance bottlenecks in the AI ​​task based on its runtime information includes: Based on the resource utilization rate of each hardware component, a first hardware component with a resource utilization rate greater than a first threshold is determined from the plurality of hardware components.

6. The method according to claim 5, characterized in that, The optimization suggestion is also used to instruct the third subtask, which is scheduled to run on the first hardware as indicated by the scheduling policy, to run on the second hardware, wherein the resource utilization rate of the second hardware after scheduling is not greater than the first threshold.

7. The method according to claim 6, characterized in that, The second hardware is the hardware with the lowest resource utilization among the candidate hardware, and the specifications and type of the candidate hardware meet the operational requirements of the third subtask.

8. The method according to claim 6, characterized in that, The difference between the resource utilization rate of the first hardware after scheduling and the resource utilization rate of the second hardware after scheduling is less than the second threshold.

9. The method according to claim 6, characterized in that, The operation information of the AI ​​task also includes the resource consumption of each of the multiple subtasks, and the third subtask is the subtask with the largest resource consumption on the first hardware.

10. The method according to any one of claims 1 to 9, characterized in that, The process of generating optimization suggestions for the AI ​​task based on its performance bottlenecks includes: The performance bottleneck is input into the tuning model to obtain optimization suggestions, which include optimized code and pseudocode for the AI ​​task. The optimization model was trained based on performance bottleneck samples, optimization strategy samples, and AI task code samples.

11. An analysis device for AI tasks, characterized in that, The AI ​​task is run by multiple hardware components, which include various hardware components of different specifications and / or types. The device includes: An acquisition unit is used to acquire the running information of the AI ​​task, wherein the AI ​​task includes multiple subtasks, and the running information includes at least one of the following: the dependency relationship between the multiple subtasks, the scheduling strategy of the multiple subtasks on the multiple hardware, and the resource utilization rate of each hardware. The processing unit is used to identify the performance bottlenecks of the AI ​​task based on the running information of the AI ​​task; The processing unit is also used to generate optimization suggestions for the AI ​​task based on the performance bottlenecks of the AI ​​task.

12. The apparatus according to claim 11, characterized in that, The processing unit is specifically used for: Based on the dependencies between the multiple subtasks, determine the first and second subtasks that have no dependencies among the multiple subtasks; If the scheduling policy of the multiple subtasks on the multiple hardware indicates that the first subtask and the second subtask are sequential, then the performance bottleneck of the AI ​​task is identified as including the first subtask and the second subtask.

13. The apparatus according to claim 12, characterized in that, The optimization suggestion is used to indicate that the first subtask and the second subtask should be changed from serial execution to parallel execution.

14. The apparatus according to claim 12, characterized in that, The operation information of the AI ​​task also includes the time consumed by each of the multiple subtasks, and the number of the second subtask is multiple; The optimization suggestion is used to indicate which subtasks can be paralleled to the first subtask based on the order of the time taken by each subtask in the second subtask from longest to shortest.

15. The apparatus according to any one of claims 11 to 14, characterized in that, The processing unit is specifically configured to determine, based on the resource utilization rate of each piece of hardware, a first piece of hardware with a resource utilization rate greater than a first threshold from the plurality of hardware pieces.

16. The apparatus according to claim 15, characterized in that, The optimization suggestion is also used to instruct the third subtask, which is scheduled to run on the first hardware as indicated by the scheduling policy, to run on the second hardware, wherein the resource utilization rate of the second hardware after scheduling is not greater than the first threshold.

17. The apparatus according to claim 16, characterized in that, The second hardware is the hardware with the lowest resource utilization among the candidate hardware, and the specifications and type of the candidate hardware meet the operational requirements of the third subtask.

18. The apparatus according to claim 16, characterized in that, The difference between the resource utilization rate of the first hardware after scheduling and the resource utilization rate of the second hardware after scheduling is less than the second threshold.

19. The apparatus according to claim 16, characterized in that, The operation information of the AI ​​task also includes the resource consumption of each of the multiple subtasks, and the third subtask is the subtask with the largest resource consumption on the first hardware.

20. The apparatus according to any one of claims 11 to 19, characterized in that, The processing unit is specifically used for: The performance bottleneck is input into the tuning model to obtain optimization suggestions, which include optimized code and pseudocode for the AI ​​task. The optimization model was trained based on performance bottleneck samples, optimization strategy samples, and AI task code samples.

21. A computing device cluster, characterized in that, It includes at least one computing device, each computing device including a processor and memory; The processor of the at least one computing device is configured to execute instructions stored in the memory of the at least one computing device to cause the cluster of computing devices to perform the method as described in any one of claims 1 to 10.

22. A computer program product containing instructions, characterized in that, When the instructions are executed by the computing device, the computing device performs the method as described in any one of claims 1 to 10; Alternatively, when the instruction is executed by the computing device cluster, the computing device cluster causes the computing device cluster to perform the method as described in any one of claims 1 to 10.

23. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes computer program instructions that, when executed by a computing device, cause the computing device to perform the method as described in any one of claims 1 to 10; Alternatively, when the computer program instructions are executed by a cluster of computing devices, the cluster of computing devices is caused to perform the method as described in any one of claims 1 to 10.

Images