Power consumption control method for artificial intelligence (AI) server and related device

Abstract

The present application discloses a power consumption control method for an artificial intelligence (AI) server, comprising: acquiring processing stages of an AI model in an AI server, wherein the AI model comprises at least two processing stages, the AI server comprises a central processing unit (CPU) and an accelerated processing unit, the accelerated processing unit is used for training the AI model or using the AI model for reasoning, and the CPU is used for controlling the accelerated processing unit to perform AI model training or reasoning; then determining a power consumption adjustment strategy of the CPU on the basis of the processing stages of the AI model; and adjusting a working mode of the CPU on the basis of the power consumption adjustment strategy. According to the method, a power consumption adjustment strategy corresponding to stages can be determined on the basis of the processing stages of an AI model, and a working mode of a CPU is adaptively adjusted on the basis of the power consumption adjustment strategy, so that the power consumption of the CPU is reduced while the performance is guaranteed, thereby optimizing the energy consumption of the CPU.

Description

A power consumption control method and related equipment for artificial intelligence AI server

[0001] This application claims priority to a Chinese patent application filed with the State Intellectual Property Office on February 6, 2024, with application number 202410173876.X and invention name “A method for controlling power consumption of an artificial intelligence (AI) server and related equipment,” the entire contents of which are incorporated by reference into this application. Technical Field

[0002] The present application relates to the field of computer technology, and in particular to a power consumption control method for an artificial intelligence (AI) server, a power consumption controller for an AI server, an AI server, an AI cluster, a computer-readable storage medium, and a computer program product. Background Art

[0003] With the continuous development of AI technology, especially the rise of large models and artificial intelligence-generated content (AIGC), AI computing power has experienced explosive growth. The scale of AI clusters from mainstream vendors typically reaches tens of thousands of accelerators and continues to grow. This has led to a rapid increase in energy consumption in AI clusters, making electricity a major cost in data center operations and maintenance, accounting for up to 20% of the total cost of ownership (TOC).

[0004] The AI ​​server in the AI ​​cluster is the core infrastructure for AI computing in intelligent computing. It is a high-performance computer specifically designed to process and analyze large-scale data and perform machine learning and deep learning tasks. Currently, the AI ​​servers widely used in the market are usually heterogeneous servers. Among them, heterogeneous servers refer to servers that include a central processing unit (CPU) and an accelerator processor. The accelerator processor includes but is not limited to a graphics processing unit (GPU), a neural network processor (Neural Network Processing Unit), and a tensor processing unit (TPU).

[0005] General-purpose servers typically rely on CPUs as their primary computing power, while AI servers rely on accelerators. In AI servers, the CPU is responsible for executing computing tasks and processing data, enabling complex algorithms and model operations. The accelerator provides high-performance parallel computing capabilities, accelerating machine learning and deep learning tasks, and speeding up model training and inference.

[0006] CPU energy optimization for AI servers can include proactively adjusting frequency when temperature or current are too high until temperature and energy consumption reach a stable point. However, this temperature / current-based control scheme is essentially a passive defense measure to ensure AI server security, making it difficult to achieve optimal performance and energy consumption targets.

[0007] Summary of the Invention

[0008] This application provides a power consumption control method for an AI server. This method can determine a power consumption adjustment strategy for the CPU in the AI ​​server based on the processing stage of the AI ​​model in the AI ​​server, and adaptively adjust the CPU operating mode based on the power consumption adjustment strategy. While ensuring performance, the CPU power consumption of the AI ​​server during training or inference is reduced, thereby achieving CPU energy consumption optimization. This application also provides an AI server power consumption controller, AI server, AI cluster, computer-readable storage medium, and computer program product corresponding to the above method.

[0009] In a first aspect, the present application provides a power consumption control method for an AI server. The method can be executed by a power consumption controller of the AI ​​server. The power consumption controller of the AI ​​server can be simply referred to as a power consumption controller. The power consumption controller is used to optimize the power consumption of the AI ​​server, for example, optimizing the power consumption of the CPU in the AI ​​server. The power consumption controller can include multiple deployment methods. One method is to be set inside the AI ​​server as an important component of the AI ​​server. Another method is to be set outside the AI ​​server, for example, to be deployed on a dedicated control server to respectively complete the CPU operating mode decision and control of all AI servers connected to the power consumption controller to optimize the CPU power consumption.

[0010] Specifically, the power consumption controller can obtain the processing stage of the AI ​​model in the AI ​​server. The AI ​​model includes at least two processing stages. The AI ​​server includes a CPU and an accelerator processor. The accelerator processor is used to train the AI ​​model or use the AI ​​model for inference. The CPU is used to control the accelerator processor to perform AI model training or inference. The power consumption controller can determine the CPU's power consumption adjustment strategy based on the AI ​​model's processing stage and then adjust the CPU's operating mode based on the power consumption adjustment strategy.

[0011] This method obtains the processing stage of the AI ​​model in the AI ​​server, determines the power consumption adjustment strategy corresponding to this processing stage, and then adaptively adjusts the CPU operating mode based on the power consumption adjustment strategy. This reduces the CPU power consumption of the AI ​​server during training or inference while ensuring performance, thereby optimizing CPU energy consumption. This method is applicable to different AI applications and has great practicality, reducing the overall operation and maintenance costs of AI clusters in data centers.

[0012] In some possible implementations, the power controller can obtain the training or inference phase of the AI ​​model in various ways. One way is for the power controller to obtain the AI ​​model processing phase reported by the CPU or accelerator processor. Another way is for the power controller to obtain the status of the accelerator processor and determine the AI ​​model processing phase based on the status of the accelerator processor.

[0013] This method supports different ways of obtaining the processing stage of the AI ​​model and adaptively adjusts the operating mode of the AI ​​server's CPU based on the processing stage, optimizing CPU energy consumption while ensuring performance. Furthermore, this method supports multiple ways of identifying the training or inference phase, ensuring high availability.

[0014] In some possible implementations, the AI ​​model in the AI ​​server may be a training model. Accordingly, the processing stages of the training model include an idle stage, a training startup stage, a stable training stage, or a verification stage. During the idle and training startup stages, the utilization rate of the acceleration processor is low, and adjusting the CPU's operating mode has little impact on performance. During the stable training and verification stages, the utilization rate of the acceleration processor is high, and adjusting the CPU's operating mode can have an impact on performance. Therefore, by dividing the AI ​​model's processing stages into the aforementioned stages and adjusting the CPU's operating mode using the power consumption adjustment strategies corresponding to each stage, optimal performance and energy consumption targets can be achieved.

[0015] In some possible implementations, the AI ​​model in the AI ​​server can be an inference model. Accordingly, the processing stages of the inference model include an idle stage, an inference startup stage, or a stable inference stage. In the idle and inference startup stages, the utilization rate of the acceleration processor is low, and adjusting the CPU's operating mode has little impact on performance. In the stable inference stage, the utilization rate of the acceleration processor is high, and adjusting the CPU's operating mode can have an impact on performance. Therefore, by dividing the AI ​​model's processing stages into the above stages and adjusting the CPU's operating mode using the power consumption adjustment strategies corresponding to each stage, optimal performance and energy consumption targets can be achieved.

[0016] It should be noted that the above stages are merely an illustrative division method, and other possible division methods may be used in other possible implementations of the embodiments of the present application. For example, the inference startup stage is relatively short and can be combined with the stable inference stage into one stage, such as the inference in progress stage or the inference-in-progress stage.

[0017] In some possible implementations, when the processing stage is a stable training stage or a verification stage, or when the processing stage is a stable inference stage, the power consumption controller can obtain the state of the acceleration processor and determine the power consumption adjustment strategy of the CPU according to the state of the acceleration processor.

[0018] Since the acceleration processor is performing model training during the stable training phase or the verification phase, and the acceleration processor is performing model reasoning during the stable reasoning phase, in order to avoid adjusting the CPU's working mode to affect the performance of the business, this application obtains the state of the acceleration processor, determines the CPU's power consumption adjustment strategy in combination with the state of the acceleration processor, and adjusts the CPU's working mode according to the power consumption adjustment strategy. In this way, the CPU's working mode can be linked to the state of the acceleration processor. Without directly perceiving the training business or the reasoning business, it is only necessary to observe the external characteristics of the acceleration processor to reduce the CPU power consumption of the AI ​​server during the training process or the reasoning process, thereby achieving the optimization of the CPU's energy consumption while ensuring performance.

[0019] In some possible implementations, when it is determined that the state of the acceleration processor is the first state, the power consumption controller determines that the power consumption adjustment strategy of the CPU is the first strategy based on the first state; when it is determined that the state of the acceleration processor is not the first state, the power consumption controller obtains the state of the CPU, which includes multiple states, and then determines that the power consumption adjustment strategy of the CPU is the second strategy based on the state of the CPU. Among them, the non-first state refers to other states other than the first state, including but not limited to the second state. In actual application, the non-first state may also include other more states, such as the third state. The above-mentioned first state, second state, and third state can be different states divided based on the load, usage rate or temperature of the acceleration processor.

[0020] This method adopts different power consumption adjustment strategies according to different states of the acceleration processor, adaptively adjusts the CPU working mode, and can further improve the energy consumption optimization effect of the CPU.

[0021] In some possible implementations, when the processing stage is an idle stage or a training startup stage, or when the processing stage is an idle stage or an inference startup stage, the power consumption controller can also obtain the state of the CPU and determine the power consumption adjustment strategy of the CPU according to the state of the CPU.

[0022] Since the utilization rate of the acceleration processor is low during the idle phase or the training startup phase or the inference startup phase, adjusting the CPU working mode has little impact on the performance of the business running on the acceleration processor. Therefore, the power consumption controller determines the power consumption adjustment strategy according to the CPU status to maximize the power consumption optimization benefits.

[0023] In some possible implementations, CPU states are classified based on CPU load and busyness. Load represents the number of tasks executed by the CPU per unit time, and busyness represents the CPU utilization rate while executing tasks. This method, by combining CPU load and busyness to classify CPU states, enables fine-grained adjustments to the CPU's operating mode, thereby increasing the benefits of CPU energy optimization.

[0024] In a second aspect, the present application provides a power consumption controller for an AI server. The power consumption controller includes:

[0025] a stage acquisition module, configured to acquire the processing stage of the AI ​​model in the AI ​​server, wherein the AI ​​model includes at least two processing stages, the AI ​​server includes a central processing unit (CPU) and an accelerator processor, the accelerator processor is configured to train the AI ​​model or perform inference using the AI ​​model, and the CPU is configured to control the accelerator processor to perform AI model training or inference;

[0026] a strategy determination module, configured to determine a power consumption adjustment strategy for the CPU according to a processing stage of the AI ​​model;

[0027] A mode adjustment module is used to adjust the working mode of the CPU according to the power consumption adjustment strategy.

[0028] In some possible implementations, the stage acquisition module is specifically configured to:

[0029] Obtaining the processing stage of the AI ​​model reported by the CPU or the acceleration processor; or

[0030] Obtain the state of the acceleration processor, and determine the processing stage of the AI ​​model according to the state of the acceleration processor.

[0031] In some possible implementations, when the AI ​​model is a training model, the processing stage includes an idle stage, a training startup stage, a stable training stage, or a verification stage; when the AI ​​model is an inference model, the processing stage includes an idle stage, an inference startup stage, or a stable inference stage.

[0032] In some possible implementations, the policy determination module is specifically configured to:

[0033] When the processing stage is a stable training stage or a verification stage, or when the processing stage is a stable inference stage, obtaining a state of the acceleration processor;

[0034] A power consumption adjustment strategy of the CPU is determined according to the state of the acceleration processor.

[0035] In some possible implementations, the policy determination module is specifically configured to:

[0036] When it is determined that the state of the acceleration processor is the first state, determining the power consumption adjustment strategy of the CPU to be the first strategy according to the first state;

[0037] When it is determined that the state of the acceleration processor is not the first state, the state of the CPU is acquired, where the CPU includes multiple states, and the power consumption adjustment strategy of the CPU is determined to be the second strategy according to the state of the CPU.

[0038] In some possible implementations, the policy determination module is specifically configured to:

[0039] When the processing stage is an idle stage or a training startup stage, or when the processing stage is an idle stage or an inference startup stage, obtaining a state of the CPU;

[0040] A power consumption adjustment strategy of the CPU is determined according to the state of the CPU.

[0041] In some possible implementations, the state of the CPU is divided according to the load and busyness of the CPU, where the load represents the number of tasks executed by the CPU in unit time, and the busyness represents the utilization rate of the CPU when executing tasks.

[0042] In some possible implementations, the power consumption controller is deployed within the AI ​​server, or on a control server external to the AI ​​server. Deploying the power consumption controller within the AI ​​server can shorten communication links and reduce communication latency, enabling timely adjustments to the CPU's operating mode to meet power consumption optimization requirements. Deploying the power consumption controller on a control server external to the AI ​​server allows a single control server to uniformly control the power consumption of multiple AI servers. This reduces the complexity and improves the robustness of the AI ​​server while avoiding the increased costs associated with deploying a power consumption controller in each AI server.

[0043] In a third aspect, the present application provides a power consumption controller. The power consumption controller includes a processor and a memory. The processor is configured to execute instructions stored in the memory, so that the power consumption controller performs the power consumption control method for an AI server as described in the first aspect or any implementation of the first aspect.

[0044] In a fourth aspect, the present application provides an AI server. The AI ​​server may include a power consumption controller, a CPU, and an accelerator processor. The power consumption controller may execute instructions to perform the power consumption control method for the AI ​​server as described in the first aspect or any implementation of the first aspect, thereby optimizing the energy consumption of the CPU in the AI ​​server.

[0045] In a fifth aspect, the present application provides an AI cluster. The AI ​​cluster includes at least one AI server. The AI ​​server may include a power consumption controller, a CPU, and an accelerator processor. The power consumption controller may execute instructions to perform the power consumption control method for the AI ​​server as described in the first aspect or any implementation of the first aspect, thereby optimizing energy consumption of the CPU in the AI ​​server.

[0046] In a sixth aspect, the present application provides an AI cluster. The AI ​​cluster includes a control server and at least one AI server. The control server includes a power consumption controller, and the AI ​​server includes a CPU and an accelerator processor. The power consumption controller interfaces with the at least one AI server. The power consumption controller can execute instructions to perform the power consumption control method for the AI ​​server as described in the first aspect or any implementation of the first aspect, thereby optimizing energy consumption of the CPU in the at least one AI server interfaced with the power consumption controller.

[0047] In the seventh aspect, the present application provides a computer-readable storage medium, which stores instructions, and the instructions instruct a computing device or a computing device cluster to execute the power consumption control method of the AI ​​server described in the above-mentioned first aspect or any implementation method of the first aspect.

[0048] In an eighth aspect, the present application provides a computer program product comprising instructions, which, when run on a computing device or a computing device cluster, enables the computing device or computing device cluster to execute the power consumption control method of the AI ​​server described in the first aspect or any one of the implementations of the first aspect.

[0049] Based on the implementation methods provided in the above aspects, this application can also be further combined to provide more implementation methods. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] In order to more clearly illustrate the technical methods of the embodiments of the present application, the following briefly introduces the drawings required for use in the embodiments.

[0051] FIG1 is a schematic diagram showing the distribution of CPU and accelerator usage of an AI server during training provided by this application;

[0052] FIG2 is a schematic diagram of a deployment of a power consumption controller provided by the present application;

[0053] FIG3 is a schematic diagram of another deployment of a power consumption controller provided by the present application;

[0054] FIG4 is a flow chart of a power consumption control method provided by the present application;

[0055] FIG5 is a schematic diagram of different stages of a model training scenario provided by this application;

[0056] FIG6 is a schematic diagram of power consumption distribution of an AI server provided in this application;

[0057] FIG7 is a schematic diagram of the structure of a power consumption controller provided by the present application;

[0058] FIG8 is a schematic diagram of the structure of a computing device provided by the present application;

[0059] FIG9 is a schematic diagram of the structure of a computing device cluster provided in this application. DETAILED DESCRIPTION

[0060] The terms "first" and "second" in the embodiments of this application are used for descriptive purposes only and should not be understood to indicate or imply relative importance or implicitly specify the number of technical features indicated. Therefore, features specified as "first" or "second" may explicitly or implicitly include one or more of the features.

[0061] First, some technical terms involved in the embodiments of this application are introduced.

[0062] An artificial intelligence (AI) server is a high-performance computer designed specifically for AI applications. By utilizing advanced hardware architecture, AI servers can rapidly process large amounts of data and complex computing tasks to meet the computing resource requirements of AI applications.

[0063] The hardware architecture of AI servers is usually a heterogeneous architecture. Among them, the heterogeneous architecture can be a hybrid architecture of a central processing unit (CPU) and an acceleration processor. An acceleration processor, also known as an accelerator or accelerator card, refers to a processor that can accelerate computing tasks, including but not limited to a graphics processing unit (GPU), a neural network processing unit (NPU), and a tensor processing unit (TPU). Among them, processors such as CPU, GPU, NPU, TPU can be collectively referred to as XPU.

[0064] Unlike general-purpose servers that rely on CPUs for computing power, AI servers primarily rely on accelerators. The CPU is responsible for executing computational tasks and processing data, enabling complex algorithms and model operations. Accelerators provide high-performance parallel computing capabilities, accelerating tasks like machine learning and deep learning, speeding up model training and inference. While accelerators consume a larger portion of AI server energy than CPUs, CPU energy consumption remains a significant component of AI server energy consumption. Reducing CPU energy consumption in AI servers can also bring significant cost-effectiveness to AI clusters.

[0065] Currently, AI server CPU energy consumption optimization solutions include temperature / current-based control solutions. Specifically, the CPU can run at the default frequency. When the temperature / current is too high, the control system actively adjusts the frequency until the temperature and energy consumption reach a stable point. This solution is essentially an overcurrent protection / overtemperature protection. Its purpose is to ensure the safety of the AI ​​server and prevent the AI ​​server from shutting down due to temperature or current exceeding the corresponding threshold values. This makes it difficult to achieve optimal performance and energy consumption targets. For example, when the load is heavy and the current is high during a certain period of time, the control system reduces the frequency, which can cause a decrease in inference speed, greatly affecting the performance of the AI ​​server.

[0066] Based on this, the industry has also provided CPU energy consumption optimization solutions based on performance prediction. This CPU energy consumption optimization solution uses performance prediction to dynamically manage CPU frequency to save CPU power consumption, thereby reducing CPU energy consumption. Specifically, using a model training scenario as an example, during model training, the control system collects signals related to the system load, such as CPU utilization, to calculate the current system load and predict the performance required by the AI ​​server in the next time period. The predicted performance is then converted into the required CPU frequency, and the CPU chip frequency and other settings are adjusted.

[0067] However, these performance prediction-based solutions require modeling and analysis of the computational process, requiring in-depth understanding of the AI ​​computational process. In practice, due to the diversity of AI applications, each requires an optimal performance prediction algorithm. However, performance prediction algorithms are difficult to generalize from one AI application to another, making it difficult to achieve optimal results.

[0068] The inventors of this application monitored the real-time resource usage of AI servers during the training of multiple large models and found that the usage of the AI ​​server's CPU and accelerator processor during the training process had significant characteristics as shown in Figure 1:

[0069] 1. During the training startup phase, the CPU usage will increase instantaneously, while the accelerator processor usage will be low.

[0070] 2. During the stable training and verification phases, the CPU usage is low and the acceleration processor usage is high.

[0071] 3. In the idle phase, the CPU and accelerator usage are low.

[0072] In view of this, the present application provides a power consumption control method. This method obtains the processing stage of the AI ​​model in the AI ​​server, determines the power consumption adjustment strategy corresponding to the processing stage of the AI ​​model, and then adaptively adjusts the CPU operating mode according to the power consumption adjustment strategy. In this way, the CPU power consumption of the AI ​​server during the training or inference process can be reduced as much as possible while ensuring performance, thereby optimizing the CPU energy consumption. This method is applicable to different AI applications, has great practicality, and reduces the overall operation and maintenance costs of the AI ​​cluster in the data center.

[0073] The power consumption control method of the present application can be applied to model training or model reasoning scenarios. In the model training scenario, the AI ​​model can be a training model, and in the model reasoning scenario, the AI ​​model can be an inference model, wherein the inference model can be a model trained in the training scenario. It should be noted that the above-mentioned AI model can be a large model, and the large model of the present application refers to a model with large-scale parameters, such as a model with a parameter scale of hundreds of billions, trillions or more. In some examples, the large model may include a generative pre-trained transformer model (GPT), such as GPT used for chatting. In some examples, the AI ​​model in the model training scenario or the model reasoning scenario may also be a model other than a large model, such as a small model with tens of billions of parameters or a small model with a parameter scale of less than tens of billions.

[0074] The power consumption control method of the AI ​​server of the present application can be executed by the power consumption controller of the AI ​​server. For ease of description, the power consumption controller of the AI ​​server can also be referred to as the power consumption controller. The power consumption controller is used to complete the operating mode decision and control of the CPU of the AI ​​server. The power consumption controller can be implemented by software or hardware.

[0075] The power consumption controller can include multiple deployment modes, which are described below with reference to the accompanying drawings.

[0076] One deployment method may be to deploy a power consumption controller within an AI server as a built-in component of the AI ​​server, completing CPU operating mode decisions and control for the host AI server where the power consumption controller is located. Referring to FIG2 , a schematic diagram of a power consumption controller deployment is shown. The power consumption controller is deployed within an AI server, which includes a power consumption controller, at least one CPU on the host side, and at least one accelerator processor on the device side. The accelerator processor may be, for example, an NPU. An operating system (OS) is deployed on both the host side and the device side. The power consumption controller can obtain the status of the accelerator processor based on the usage rate, load, or temperature of the accelerator processor collected by the device OS. Based on the status of the accelerator processor, the processing stage of the AI ​​model in the AI ​​server can be obtained. The power consumption controller can determine the power consumption adjustment policy of the CPU based on the processing stage of the AI ​​model. In some cases, the power consumption controller can also obtain the status of the CPU, for example, based on the CPU usage rate (also known as occupancy rate) and CPU load collected by the host OS, and determine the power consumption adjustment policy of the CPU based on the CPU status. The power consumption controller can adjust the CPU operating mode based on the power consumption adjustment policy. For example, the power consumption controller may determine parameters of the operating mode according to the power consumption adjustment policy, and set the operating mode of the CPU according to the parameters.

[0077] Another deployment method is to deploy the power consumption controller outside of the AI ​​server, for example, on a dedicated control server, to determine and control the CPU operating mode of all AI servers connected to the power consumption controller. Referring to FIG3 , another schematic diagram of power consumption controller deployment is shown. The power consumption controller is deployed on the control server and is connected to at least one AI server, for example, AI server 1, AI server 2, and AI server N. Taking one AI server as an example, the AI ​​server includes at least one CPU on the host side and at least one accelerator processor on the device side. The host and device sides are each deployed with an operating system, specifically a host OS and a device OS. The power consumption controller can obtain the accelerator processor's status based on the accelerator processor's usage, load, or temperature collected by the device OS. Based on the accelerator processor's status, the processing stage of the AI ​​model in the AI ​​server can be determined. The power consumption controller can determine the CPU's power consumption adjustment policy based on the AI ​​model's processing stage. Furthermore, the power consumption controller can obtain the CPU's status based on the CPU's usage and load collected by the host OS, and determine the CPU's power consumption adjustment policy based on the CPU's status. The power consumption controller can adjust the CPU's operating mode based on the power consumption adjustment policy.

[0078] It should be noted that Figures 2 and 3 illustrate the example of a power controller obtaining the status of the acceleration processor and determining the processing stage of the AI ​​model based on the status of the acceleration processor. In actual applications, the power controller can also receive the processing stage of the AI ​​model reported by the CPU or acceleration processor. The CPU and acceleration processor can determine the processing stage of the AI ​​model based on the status of the acceleration processor and report the processing stage of the AI ​​model to the power controller.

[0079] In order to make the technical solution of the present application clearer and easier to understand, the power consumption control method of the AI ​​server of the present application is introduced below with reference to the accompanying drawings.

[0080] FIG4 is a flowchart of a method for controlling power consumption of an AI server. The method is illustrated using a model training scenario as an example and specifically includes the following steps:

[0081] S402: The power consumption controller obtains the processing stage of the AI ​​model in the AI ​​server. If the processing stage is the stable training stage or the verification stage, S404 is executed; if the processing stage is the idle stage or the training startup stage, S412 is executed.

[0082] In a model training scenario, the AI ​​model can be a training model. This training model can be a large model with hundreds of billions or trillions of parameters, or a small model with fewer than a hundred billion parameters. The AI ​​model includes at least two processing stages. The processing stages of the AI ​​model can reflect the operating status of the AI ​​server and, therefore, can be used to control the power consumption of the AI ​​server.

[0083] Specifically, referring to Figure 5, in the model training scenario, the processing stage of the AI ​​model can be divided into an idle stage and a training stage, wherein the training stage can be further divided into a training startup stage, a stable training stage, and a verification stage.

[0084] Considering that the state of the accelerator processor may be different at different processing stages, the processing stage of the AI ​​model may be determined based on the state of the accelerator processor. The state of the accelerator processor may be characterized by at least one of the load, usage rate, or temperature of the accelerator processor.

[0085] The power controller can determine the processing stage of the AI ​​model in a variety of ways. Each of these is explained below.

[0086] The first method is that the power consumption controller determines the processing stage of the AI ​​model by itself. Specifically, the power consumption controller can obtain the status of the acceleration processor, and then determine the processing stage of the AI ​​model based on the status of the acceleration processor. The status of the acceleration processor can be characterized by at least one of the utilization rate, load or temperature of the acceleration processor. The utilization rate of the acceleration processor is the utilization rate of the acceleration processor when executing tasks. The load of the acceleration processor represents the number of tasks executed by the acceleration processor per unit time. The load of the acceleration processor can be the average load of the acceleration processor.

[0087] For ease of understanding, this application uses an example of determining the processing stage of the AI ​​model based on the utilization rate of the acceleration processor. For example, when the utilization rate of the acceleration processor is less than a first value, the power consumption controller determines that the AI ​​model is in the idle stage or the training startup stage; when the utilization rate of the acceleration processor is greater than or equal to the first value, the power consumption controller determines that the AI ​​server is in the stable training stage or the verification stage. Among them, the first value can be set based on experience, for example, the first value can be set to 10%. It should be noted that the first value can also be adjusted during the power consumption control process.

[0088] In other possible implementations of the present application, the power consumption controller may also determine the processing stage of the AI ​​model based on the load or temperature of the acceleration processor. Alternatively, the power consumption controller may also determine the processing stage of the AI ​​model based on a combination of the usage rate, load, or temperature of the acceleration processor.

[0089] Among them, real-time monitoring of indicators such as the usage rate, load, and temperature of the acceleration processor can also include multiple implementation methods. One method is routine inspection. Specifically, the power consumption controller can periodically and proactively query the OS running on the acceleration processor about the usage rate, load, or temperature of the acceleration processor. Another implementation method is fast reporting. Specifically, a threshold value can be set for the monitored indicator. When the OS running on the acceleration processor detects that the indicator exceeds the limit (the indicator exceeds the threshold value), it actively reports the indicator to the power consumption controller, such as the usage rate, load, or temperature of the acceleration processor.

[0090] The second method is that the power consumption controller receives the processing stage of the AI ​​model reported by the CPU or the acceleration processor. Take the example of the processing stage of the AI ​​model reported by the acceleration processor. The OS running on the acceleration processor can obtain the status of the acceleration processor, such as the utilization rate, load or temperature of the acceleration processor, and then determine the processing stage of the AI ​​model based on the utilization rate, load or temperature, and then report the processing stage of the AI ​​model to the power consumption controller. Among them, the specific implementation of the acceleration processor (or the OS running on the acceleration processor) determining the processing stage of the AI ​​model based on the utilization rate, load or temperature of the acceleration processor can refer to the relevant content description of the power consumption controller determining the processing stage of the AI ​​model based on the status of the acceleration processor. I will not go into details here.

[0091] S404: The power consumption controller obtains the state of the acceleration processor. If the state of the acceleration processor is the first state, execute S406; if the state of the acceleration processor is not the first state, execute S408 to S410.

[0092] If the processing stage is the stable training stage or the verification stage, it means that the acceleration processor is training the model or verifying the trained model. To avoid affecting performance, the power consumption controller can obtain the status of the acceleration processor and adjust the CPU operating mode based on the status of the acceleration processor. This optimizes CPU power consumption while ensuring the performance of the AI ​​server.

[0093] The power consumption controller can determine the power consumption adjustment strategy in different ways based on the different states of the acceleration processor. A non-first state refers to a state other than the first state, including but not limited to the second state. In actual applications, the non-first state may also include other states, such as the third state. The first state, second state, and third state described above can be different states based on the load, usage rate, or temperature of the acceleration processor.

[0094] Specifically, when the state of the acceleration processor is the first state, the power consumption controller executes S406 to determine that the power consumption strategy of the CPU is the first strategy. When the state of the acceleration processor is not the first state, for example, when it is the second state, the power consumption controller executes S408 to S410 to determine that the power consumption adjustment strategy of the CPU is the second strategy.

[0095] In some examples, the first state may be that the difference between the usage of the acceleration processor and the usage of the previous acceleration processor (e.g., the usage of the acceleration processor in the previous cycle) is less than a first threshold, and the second state may be that the usage of the acceleration processor is greater than or equal to the first threshold. The first threshold is less than 0 and can be set based on experience, for example, the first threshold can be set to -10%.

[0096] In some possible implementations, the second state may be that the usage rate of the acceleration processor is greater than or equal to a first threshold, and the difference between the temperature of the acceleration processor and a threshold value is less than a second threshold. The second threshold value can be set based on experience, for example, it can be set to -2 degrees Celsius (°C). In this case, the acceleration processor may also include a third state, in which the difference between the usage rate of the acceleration processor and the usage rate of the previous acceleration processor is greater than or equal to the first threshold, and the temperature of the acceleration processor is greater than or equal to the threshold value.

[0097] It should be noted that when the power consumption controller has obtained the status of the acceleration processor when determining the training phase of the AI ​​model, there is no need to repeatedly obtain the status of the acceleration processor. When the power consumption controller receives the training phase of the AI ​​model reported by the CPU or the acceleration processor, the power consumption controller can obtain the status of the acceleration processor.

[0098] S406: The power consumption controller determines that the power consumption adjustment strategy of the CPU is the first strategy according to the first state.

[0099] The first state includes a difference between the usage of the acceleration processor and the usage of the last acceleration processor being less than a first threshold, indicating a sudden drop in the usage of the acceleration processor. The power consumption controller may determine that the power consumption adjustment strategy for the CPU is to restore the CPU operating mode to the last backup operating mode. In this case, the first strategy may be to adjust the CPU operating mode to the last backup operating mode.

[0100] S408: The power consumption controller obtains the CPU status.

[0101] The second state includes that the difference between the usage rate of the acceleration processor and the usage rate of the last acceleration processor is greater than or equal to the first threshold. Furthermore, the second state can be that the difference between the usage rate of the acceleration processor and the usage rate of the last acceleration processor is greater than or equal to the first threshold, and the difference between the temperature of the acceleration processor and the threshold value is less than the second threshold. Among them, the difference in usage rate is greater than or equal to the first threshold, indicating that the usage rate of the acceleration processor has not dropped sharply, and the difference between the temperature of the acceleration processor and the threshold value is less than the second threshold, indicating that the temperature of the acceleration processor has not exceeded the limit and will not exceed the limit immediately. Therefore, the power consumption controller can obtain the state of the CPU to determine the power consumption adjustment strategy of the CPU.

[0102] The state of the CPU can be divided according to the load and busyness of the CPU. Among them, the load represents the number of tasks executed by the CPU in a unit of time, based on this, it can also be called the average load. Furthermore, the above load can also be standardized. For example, the power consumption controller can periodically (for example, 1 second as a cycle) standardize the average load of the CPU. Among them, the standardized load = the average load of the CPU / the number of CPU working cores. Among them, the working core refers to the CPU core in the working state. The busyness represents the CPU usage rate when the CPU is executing tasks.

[0103] Among them, different CPU loads can correspond to different overload levels, and different CPU usage rates can correspond to different busy levels. The power consumption controller can divide the CPU overload level into levels according to the CPU load, and divide the CPU busy level into levels according to the CPU usage rate (for example, the average usage rate). For example, each level of overload level is respectively set with its own threshold value, and the power consumption controller can compare the load with the threshold value of at least one level of overload level to determine the level of overload level. Similarly, each level of busyness is respectively set with its own threshold value, and the power consumption controller can compare the CPU usage rate with the threshold value of at least one level of busyness to determine the level of busyness.

[0104] In some examples, the degree of overload may include the following levels: normal load, slight overload, moderate overload, and severe overload, where the value range of the standardized load corresponding to normal load is [0, 0.7], the value range of the standardized load corresponding to slight overload is (0.7, 1], the value range of the standardized load corresponding to moderate overload is (1, 5], and the value range of the standardized load corresponding to severe overload is >5. Similarly, the degree of busyness may include the following levels: idle, normal, slightly busy, and severely busy, where the value range of the average CPU usage corresponding to idle is [0%, 5%), the value range of the average CPU usage corresponding to normal is [5%, 75%), the value range of the average CPU usage corresponding to slightly busy is [75%, 95%, and the value range of the average CPU usage corresponding to severely busy is [95%, 100%).

[0105] Based on this, the state of the CPU can be represented by at least one of an overload level and a busy level.

[0106] S410: The power consumption controller determines that the power consumption adjustment strategy of the CPU is a second strategy according to the state of the CPU.

[0107] Specifically, the power consumption controller may determine the CPU power consumption adjustment strategy to be the second strategy according to the state of the CPU and through a mapping relationship between the overload level or busyness level in the stable training phase or the verification phase and the power consumption adjustment strategy.

[0108] The mapping relationship between the overload level or busyness level and the power consumption adjustment strategy in the stable training phase or the verification phase is as follows:

[0109] If the busyness is at the first level, such as idle or normal, the power consumption adjustment strategy of the CPU may be to take the CPU cores of the CPU offline or to lower the operating frequency of the CPU. In order to avoid affecting the performance, the power consumption adjustment strategy may be to adjust the number of working cores first, and when the number of working cores reaches a minimum number (such as a limit such as the minimum number of working cores), then consider adjusting the operating frequency. For example, when the number of CPU cores in working state is greater than a set number, the power consumption adjustment strategy is to take the CPU cores of the CPU offline until the number of CPU cores in working state is less than or equal to the set number; when the number of CPU cores in working state is less than or equal to the set number, the power consumption adjustment strategy is to lower the operating frequency of the CPU to the lowest operating frequency. Among them, the set number can be set based on experience. In some examples, when the CPU has 32 cores or 64 cores, the set number can be 8.

[0110] Furthermore, when the first level is idle, the power consumption adjustment strategy of the CPU may be to offline a first number of CPU cores or to reduce the operating frequency of the CPU by a first amplitude. When the first level is normal, the power consumption adjustment strategy of the CPU may be to offline a second number of CPU cores or to reduce the operating frequency of the CPU by a second amplitude. The first number is greater than the second number, and the first amplitude is greater than the second amplitude. The above-mentioned first number, second number, first amplitude, and second amplitude can be set according to empirical values. For example, when the CPU has 32 cores or 64 cores, the first number may be 2, the second number may be 1, the first amplitude may be 2 amplitude adjustment units, and the second amplitude may be 1 amplitude adjustment unit.

[0111] If the busyness level is the second level and the overload level is also the second level, the CPU power consumption adjustment strategy may be to increase the CPU operating frequency or change the operating mode of the CPU core of the online CPU. The second busyness level may include slightly busy or severely busy, and the second overload level may include moderate overload or severe overload.

[0112] When the busyness is severely busy and the overload is moderately overloaded or severely overloaded, the power consumption adjustment strategy of the CPU may be to bring the CPU cores of the CPU online to a maximum number and increase the operating frequency of the CPU to a maximum operating frequency.

[0113] When the busyness is slightly busy and the overload is moderately overloaded, the power consumption adjustment strategy of the CPU may be to bring online a first proportion of offline CPU cores or to increase the operating frequency of the CPU according to a third amplitude. When the busyness is slightly busy and the overload is severely overloaded, the power consumption adjustment strategy of the CPU may be to bring online a second proportion of offline CPU cores or to increase the operating frequency of the CPU according to a fourth amplitude. Among them, the first ratio is smaller than the second ratio, and the third amplitude is smaller than the fourth amplitude. The above-mentioned first ratio, second ratio, third amplitude, and fourth amplitude can be set based on experience. For example, when the CPU has 32 cores or 64 cores, the first ratio may be 1 / 4, the second ratio may be 1 / 2, the first amplitude may be 8 amplitude adjustment units, and the second amplitude may be 16 amplitude adjustment units.

[0114] When the busyness is at the second level and the overload is also at the second level, the CPU power consumption adjustment strategy may be to prioritize increasing the operating frequency before considering bringing CPU cores online. Specifically, the CPU power consumption adjustment strategy may be to increase the CPU operating frequency to the maximum operating frequency; when the CPU is at the highest operating frequency, the CPU power consumption adjustment strategy may be to bring CPU cores online until the number of CPU cores in operation reaches the maximum number.

[0115] In some possible implementations, if the overload level is at a first level, such as a normal load or a slight overload, and the busyness level is at a second level, such as a slightly busy or severely busy level, the CPU power consumption adjustment strategy may be to maintain the last operating mode. It should be noted that if the difference between the utilization rate of the acceleration processor and the utilization rate of the last acceleration processor is greater than or equal to a first threshold, and the difference between the temperature of the acceleration processor and the threshold value is greater than or equal to a second threshold and less than zero, indicating that the temperature is about to exceed the limit, the CPU power consumption adjustment strategy may also be to maintain the current operating mode.

[0116] In some possible implementations, the accelerator card may be in a third state. In the third state, the difference between the usage of the accelerator processor and the usage of the previous accelerator processor is greater than or equal to a first threshold, and the temperature of the accelerator processor is greater than or equal to a threshold, indicating that the temperature has exceeded a limit. The power consumption controller may determine a power consumption adjustment strategy for the CPU to be to deactivate the CPU core of the CPU or to reduce the operating frequency of the CPU.

[0117] Among them, the number of CPU cores taken offline and the amplitude of lowering the operating frequency can be set based on experience. In some examples, the status of the CPU may also include the number of remaining working cores, referred to as the remaining working cores. If the remaining working cores are greater than the set number, the power consumption adjustment strategy may be to take the CPU working cores offline until only the set number of working cores are left. If the remaining working cores are not greater than the set number, the power consumption adjustment strategy may be to lower the CPU operating frequency until the lowest operating frequency. Among them, the set number can be set based on experience. For example, in a 32-core or 64-core scenario, the set number can be 8. The number of CPU cores taken offline and the amplitude of lowering the CPU operating frequency can be set based on experience. For example, the number of CPU cores taken offline can be 2, and the amplitude of lowering the operating frequency can be 2 amplitude adjustment units.

[0118] S412: The power consumption controller obtains the CPU status.

[0119] The training phase is the idle phase or the training startup phase, indicating that the acceleration processor is not performing model training or has just started model training. In this case, the power consumption controller adjusts the CPU operating mode, which has little impact on the performance of the AI ​​server. Based on this, the power consumption controller can obtain the CPU status and adjust the CPU operating mode based on the CPU status to optimize the CPU power consumption as much as possible.

[0120] The specific implementation of the power consumption controller obtaining the CPU status can be found in the description of S408, which will not be repeated here.

[0121] S414: The power consumption controller determines a power consumption adjustment strategy of the CPU according to the state of the CPU.

[0122] Specifically, if the difference between the current load of the CPU (for example, the current average service) and the last load of the CPU (for example, the last average load) is greater than the third threshold, the power consumption controller can determine the power consumption adjustment strategy of the CPU to be to online the maximum number of CPU cores of the CPU and increase the operating frequency of the CPU to the highest operating frequency. Among them, the third threshold can be set based on experience. In some examples, the third threshold can be set to 2. In this case, the power consumption adjustment strategy can be to online all CPU cores at one time and adjust the operating frequency of the CPU to the highest operating frequency at one time. Otherwise, the power consumption controller can determine the power consumption adjustment strategy of the CPU based on the state of the CPU, through the mapping relationship between the level of overload or the level of busyness in the idle phase or the training startup phase and the power consumption adjustment strategy.

[0123] The mapping relationship between the overload level or busyness level and the power consumption adjustment strategy in the idle phase or the training startup phase is as follows:

[0124] If the busyness is at the first level, such as idle or normal, the power consumption adjustment strategy may be to lower the CPU's operating frequency or to take the CPU core offline. During the Idle phase or the training startup phase, the power consumption adjustment strategy may be to prioritize adjusting the operating frequency, and when the operating frequency reaches the minimum operating frequency, consider adjusting the number of CPU cores. For example, the power consumption controller may determine that the CPU's power consumption adjustment strategy is to lower the CPU's operating frequency. If the CPU's operating frequency reaches the minimum operating frequency, the power consumption controller may determine that the CPU's power consumption adjustment strategy is to take the CPU core offline, until the minimum number of CPU cores in operation is reached.

[0125] Furthermore, when the first level is idle, the power consumption controller may determine that the power consumption adjustment strategy of the CPU is to reduce the operating frequency of the CPU by a first amplitude or to offline a first number of CPU cores. When the first level is normal, the power consumption controller may determine that the power consumption adjustment strategy of the CPU is to reduce the operating frequency of the CPU by a second amplitude or to offline a second number of CPU cores. The first number is greater than the second number, and the first amplitude is greater than the second amplitude. For example, the first number may be 2, the second number may be 1, the first amplitude may be 2 amplitude adjustment units, and the second amplitude may be 1 amplitude adjustment unit.

[0126] If the busyness is at the second level and the overload is also at the second level, the power consumption controller may determine the CPU power consumption adjustment strategy to be to bring the CPU core online or increase the CPU operating frequency. The second busyness level may include slightly busy or severely busy, and the second overload level may include moderate overload or severe overload.

[0127] When the busy level is severely busy and the overload level is moderately overloaded or severely overloaded, the power consumption controller may determine the power consumption adjustment strategy of the CPU to bring online the maximum number of CPU cores of the CPU (i.e., bring online all CPU cores) and increase the CPU operating frequency to the highest operating frequency.

[0128] When the busyness is slightly busy and the overload is moderately overloaded, the power consumption controller can determine the power consumption adjustment strategy of the CPU to bring online a first proportion of offline CPU cores or to increase the operating frequency of the CPU according to a third amplitude. When the busyness is slightly busy and the overload is severely overloaded, the power consumption controller can determine the power consumption adjustment strategy of the CPU to bring online a second proportion of offline CPU cores or to increase the operating frequency of the CPU according to a fourth amplitude. The first ratio is smaller than the second ratio, and the third amplitude is smaller than the fourth amplitude. For example, the first ratio can be 1 / 4, the second ratio can be 1 / 2, the first amplitude can be 8 amplitude adjustment units, and the second amplitude can be 16 amplitude adjustment units.

[0129] In some possible implementations, if the overload level is the first level, such as normal load or slight overload, and the busy level is the second level, such as slightly busy or severely busy, the CPU power consumption adjustment strategy may be to maintain the last working mode.

[0130] The above S404 to S414 are some specific implementation methods of the power consumption controller determining the power consumption adjustment strategy of the CPU according to the training stage of the AI ​​model. In other possible implementation methods of the embodiment of the present application, the power consumption adjustment strategy of the CPU can also be determined by other methods, and this embodiment does not limit this.

[0131] S416: The power consumption controller adjusts the CPU working mode according to the power consumption adjustment strategy.

[0132] Specifically, the power consumption controller may provide a setting interface for setting an operating mode, such as setting mode parameters of the operating mode. The power consumption controller may adjust or switch the operating mode through the setting interface. For example, the power consumption controller may determine the mode parameters of the operating mode according to a power consumption adjustment policy, and then set the mode parameters of the operating mode to the mode parameters determined by the power consumption adjustment policy through the setting interface.

[0133] Furthermore, in order to ensure safety and reliability, the power consumption controller may first back up the current working mode of the CPU, for example, back up the mode parameters of the current working mode, and then adjust or switch the working mode of the CPU through the setting interface.

[0134] Based on the above description, it can be seen that the power consumption control method of the present application can obtain the processing stage of the AI ​​model in the AI ​​server, determine the power consumption adjustment strategy suitable for the stage according to the processing stage of the AI ​​model, and adaptively adjust the working mode of the CPU according to the power consumption adjustment strategy, so as to achieve the reduction of the CPU power consumption of the AI ​​server while ensuring performance, thereby achieving the energy consumption optimization of the CPU. Furthermore, when the processing stage of the AI ​​model is the stable training stage or the verification stage, the working mode of the CPU can be determined by the state of the acceleration processor, that is, the working mode of the CPU in the AI ​​server is linked to the state of the acceleration processor. In this way, without directly perceiving the business, it is only necessary to observe the external characteristics of the accelerated processing, reduce the CPU power consumption of the AI ​​server during the training process, and thereby achieve the energy consumption optimization of the CPU.

[0135] The embodiment shown in FIG4 primarily describes the power consumption control method for the AI ​​server provided in this application using a model training scenario. The power consumption control method for the AI ​​server provided in this application can also be applied to a model inference scenario. The main difference between the model inference scenario and the model training scenario is that the AI ​​model includes at least two processing stages, which can be an idle stage, an inference startup stage, or a stable inference stage. The inference startup stage or the stable inference stage can also be divided into a single inference stage, such as an inference-in-progress stage or an inference-in-progress stage. The power consumption controller can obtain the processing stage of the AI ​​model in the model inference scenario in a manner similar to that used in the model training scenario. The power consumption controller then determines the CPU power consumption adjustment policy based on the processing stage of the AI ​​model. For details on how the power consumption controller determines the power consumption adjustment policy when the processing stage is the idle stage or the inference startup stage, refer to the specific implementation of the power consumption controller determining the power consumption adjustment policy when the processing stage is the idle stage or the training startup stage. For details on how the power consumption controller determines the power consumption adjustment policy when the processing stage is the stable inference stage, refer to the specific implementation of the power consumption controller determining the power consumption adjustment policy when the processing stage is the stable training stage or the verification stage. This will not be further described here.

[0136] For ease of understanding, this application also provides an example for illustration. In this example, the number of CPU cores in the CPU is 64, that is, the CPU is a 64-core CPU. The minimum number of CPU cores (working cores) in the CPU that are in working state is 8, and the maximum number of CPU cores in working state is 64.

[0137] When the power controller determines that the AI ​​model is in the stable training or verification phase based on the usage of the acceleration processor, if the difference between the usage of the acceleration processor and the last usage of the acceleration processor is less than -10%, the CPU operating mode is restored to the last operating mode. Otherwise, the CPU operating mode in this phase is determined as follows:

[0138] When the difference between the temperature of the acceleration processor and the preset threshold value is less than -2°C, the power consumption controller can adjust the CPU working mode according to the power consumption adjustment policy in the following table.

[0139] Table 1 CPU power consumption adjustment strategy

[0140] When the difference between the temperature of the acceleration processor and its preset threshold is less than 0, it means that the temperature is about to exceed the limit, and the power consumption controller can keep the current working mode of the CPU unchanged.

[0141] The above describes the specific implementation of the power consumption controller adjusting the CPU working mode during the stable training phase and the verification phase. The following describes the specific implementation of the power consumption controller adjusting the CPU working mode during the idle phase and the training startup phase.

[0142] When the power controller determines that the AI ​​model is in the idle phase or training startup phase based on the utilization rate of the acceleration processor, if the difference between the CPU load and the previous CPU load is greater than 2, the CPU operating mode is adjusted to all CPU cores online and the CPU operates at the maximum operating frequency. Otherwise, the CPU operating mode in this phase is determined as follows:

[0143] Table 2 CPU power consumption adjustment strategy

[0144] As can be seen from the above table, the power consumption controller's strategy for adjusting the operating mode is to adjust slowly and quickly. It should also be noted that when taking a CPU core offline or lowering the operating frequency, the above table uses the example of prioritizing the operating frequency and then adjusting the number of CPU cores. When bringing a CPU core online or raising the operating frequency, the above table uses the example of prioritizing the number of CPU cores and then adjusting the operating frequency. In actual applications, the power consumption controller may also prioritize adjusting the number of CPU cores when taking a CPU offline or lowering the operating frequency, and then consider adjusting the operating frequency. The power consumption controller may also prioritize adjusting the operating frequency and then consider adjusting the number of CPU cores when bringing a CPU online or raising the operating frequency.

[0145] It should be noted that the power consumption controller determines the power consumption adjustment strategy of the CPU based on the collected indicators. The decision-making process can be periodic or triggered when the indicators change, which is not limited in this embodiment.

[0146] In order to verify the technical effect of the technical solution of the present application, relevant experiments were designed for verification. An example of power consumption control during large model training is given by an AI server including 4 acceleration processors (abbreviated as 4 cards) and an AI server including 8 acceleration processors (abbreviated as 8 cards). In this example, the power consumption controller can obtain the load, usage rate or temperature of the acceleration processor in the AI ​​server, and determine the processing stage of the AI ​​model in the AI ​​server based on the load, usage rate or temperature of the acceleration processor. When the processing stage of the AI ​​model is the stable training stage or the verification stage, if the difference between the temperature of the acceleration processor and the preset threshold value is less than -2°C, the power consumption controller can adjust the CPU working mode according to the power consumption adjustment strategy in Table 1. When the processing stage of the AI ​​model is the training startup stage or the idle stage, if the difference between the current CPU average load and the last CPU average load is ≤2, the power consumption controller can adjust the CPU working mode according to the power consumption adjustment strategy in Table 2. Figure 6 also shows the power consumption distribution of the AI ​​server, in which the CPU power consumption can account for more than 20%. If the technical solution of the present application is used for tuning, the power consumption can be reduced by about 5%, which is a significant benefit.

[0147] Based on the aforementioned power consumption control method, the present application also provides a power consumption controller. The following describes the power consumption controller from the perspective of functional modularization. As shown in FIG7 , the power consumption controller 700 includes:

[0148] A stage acquisition module 702 is configured to acquire the processing stage of an AI model in an AI server, wherein the AI ​​model includes at least two processing stages, and the AI ​​server includes a CPU and an accelerator processor. The accelerator processor is configured to train the AI ​​model or perform inference using the AI ​​model, and the CPU is configured to control the accelerator processor to perform AI model training or inference.

[0149] A strategy determination module 704 is configured to determine a CPU power consumption adjustment strategy based on the processing stage of the AI ​​model;

[0150] The mode adjustment module 706 is configured to adjust the CPU operating mode according to the power consumption adjustment policy.

[0151] Among them, the stage acquisition module 702 is used to implement the content description related to S402 in the embodiment shown in Figure 4, the strategy determination module 704 is used to implement the content description related to S404 to S414 in the embodiment shown in Figure 4, and the mode adjustment module 706 is used to implement the content description related to S416.

[0152] Exemplarily, the above-mentioned stage acquisition module 702, strategy determination module 704, and mode adjustment module 706 can be implemented by hardware or by software.

[0153] When implemented via software, the phase acquisition module 702, policy determination module 704, and mode adjustment module 706 can be applications running on a computer device, such as a computing engine. These applications can be provided to users in the form of virtualization services. Virtualization services can include virtual machine (VM) services, bare metal server (BMS) services, and container services. VM services can use virtualization technology to create a virtual machine (VM) resource pool on multiple physical hosts, providing users with VMs on demand. BMS services use virtualized BMS resource pools on multiple physical hosts, providing users with BMSs on demand. Container services use virtualized container resource pools on multiple physical hosts, providing users with containers on demand. A VM is a simulated virtual computer, or logically a single computer. BMS is a scalable, high-performance computing service with computing performance comparable to traditional physical machines and secure physical isolation. Containers are a kernel virtualization technology that provides lightweight virtualization to isolate user space, processes, and resources. It should be understood that the VM service, BMS service and container service in the above-mentioned virtualization services are only specific examples. In actual applications, virtualization services can also be other lightweight or heavyweight virtualization services, which are not specifically limited here.

[0154] When implemented through hardware, the phase acquisition module 702, the policy determination module 704, and the mode adjustment module 706 may include at least one computing device, such as a server. Alternatively, the phase acquisition module 702, the policy determination module 704, and the mode adjustment module 706 may be implemented using an application-specific integrated circuit (ASIC) or a programmable logic device (PLD). The PLD may be a complex programmable logical device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), or any combination thereof.

[0155] In some possible implementations, the stage acquisition module 702 is specifically configured to:

[0156] Get the processing stage of the AI ​​model reported by the CPU or accelerator; or,

[0157] Obtain the status of the acceleration processor, and determine the processing stage of the AI ​​model according to the status of the acceleration processor.

[0158] In some possible implementations, when the AI ​​model is a training model, the processing stage includes an idle stage, a training startup stage, a stable training stage, or a verification stage; when the AI ​​model is an inference model, the processing stage includes an idle stage, an inference startup stage, or a stable inference stage.

[0159] In some possible implementations, the policy determination module 704 is specifically configured to:

[0160] When the processing stage is a stable training stage or a validation stage, or when the processing stage is a stable inference stage, obtaining a state of the acceleration processor;

[0161] The power consumption adjustment policy of the CPU is determined according to the state of the acceleration processor.

[0162] In some possible implementations, the policy determination module 704 is specifically configured to:

[0163] When it is determined that the state of the acceleration processor is the first state, determining the power consumption adjustment strategy of the CPU to be the first strategy according to the first state;

[0164] When it is determined that the state of the acceleration processor is not the first state, the state of the CPU is acquired, where the CPU includes multiple states, and the power consumption adjustment strategy of the CPU is determined to be the second strategy according to the state of the CPU.

[0165] In some possible implementations, the policy determination module 704 is specifically configured to:

[0166] When the processing stage is in the idle stage or the training startup stage, or when the processing stage is in the idle stage or the inference startup stage, obtain the CPU status;

[0167] A power consumption adjustment strategy of the CPU is determined according to the state of the CPU.

[0168] In some possible implementations, the state of the CPU is divided according to the load and busyness of the CPU, where the load represents the number of tasks executed by the CPU in a unit time, and the busyness represents the utilization rate of the CPU when executing tasks.

[0169] In some possible implementations, the power consumption controller is deployed within the AI ​​server, or on a control server external to the AI ​​server. Deploying the power consumption controller within the AI ​​server can shorten communication links and reduce communication latency, enabling timely adjustments to the CPU's operating mode to meet power consumption optimization requirements. Deploying the power consumption controller on a control server external to the AI ​​server allows a single control server to uniformly control the power consumption of multiple AI servers. This reduces the complexity and improves the robustness of the AI ​​server while avoiding the increased costs associated with deploying a power consumption controller in each AI server.

[0170] The present application also provides a computing device 800. The computing device 800 can be an AI server for model training or model inference. As shown in Figure 8, the computing device 800 includes: a bus 802, a power consumption controller 803, a central processing unit 804, an acceleration processor 805, a memory 806, and a communication interface 808. The processor 804, the acceleration processor 805, the memory 806, and the communication interface 808 communicate with each other via the bus 802. It should be understood that the present application does not limit the number of processors and memories in the computing device 800.

[0171] Bus 802 may be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, among others. Buses may be classified as address buses, data buses, control buses, and the like. For ease of illustration, FIG8 illustrates a single bus line, but this does not imply a single bus or type of bus. Bus 802 may include a path for transmitting information between various components of computing device 800 (e.g., memory 806, central processing unit 804, accelerator processing unit 805, and communication interface 808).

[0172] The power consumption controller 803 is a key component of the computing device 800 for optimizing power consumption. Its function is to optimize the power consumption of the AI ​​server, for example, optimizing the power consumption of the central processing unit 804 (or CPU) in the AI ​​server. The power consumption controller 803 may include a processor and a memory. The memory may store instructions for the stage acquisition module 702, the policy determination module 704, and the mode adjustment module 706 shown in FIG7 . The processor reads these instructions to implement the power consumption control method for the AI ​​server.

[0173] The central processing unit (CPU) 804 is one of the main components of the computing device 800. Its primary function is to interpret computer instructions and process data in computer software. The CPU 804 can include various instruction set architectures (ISAs), which define the instruction set, registers, data types, address modes, exception handling, and other features supported by the CPU. The CPU architecture determines the CPU's functionality, performance, compatibility, and energy efficiency.

[0174] The acceleration processor 805 is another major component of the computing device 800. It can be a processing unit that assists the traditional central processing unit in handling special types of computing tasks, such as graphics processing and vector calculations. The acceleration processor 805, also known as an accelerator or accelerator card, includes but is not limited to any one or more processors such as a graphics processing unit (GPU), a neural network processing unit (NPU), and a tensor processing unit (TPU). CPUs, GPUs, NPUs, TPUs, and other processors can be collectively referred to as XPUs.

[0175] The memory 806 may include a volatile memory, such as a random access memory (RAM). The memory 806 may also include a non-volatile memory, such as a read-only memory (ROM), a flash memory, a hard disk drive (HDD), or a solid state drive (SSD). The memory 806 stores executable program code, and the processor 804 executes the executable program code to implement the aforementioned power consumption control method. Specifically, the memory 806 stores instructions for the power consumption controller 803 to execute the power consumption control method.

[0176] The communication interface 808 uses a transceiver module such as, but not limited to, a network interface card or a transceiver to implement communication between the computing device 800 and other devices or a communication network.

[0177] The present application also provides a computing device cluster. The computing device cluster includes at least one computing device. The computing device may be the AI ​​server shown in FIG8 , and accordingly, the computing device cluster may be an AI cluster.

[0178] As shown in Figure 9, the computing device cluster includes at least one computing device 800. The memory 806 in one or more computing devices 800 in the computing device cluster may store the same power consumption controller for executing instructions of the power consumption control method.

[0179] In some possible implementations, one or more computing devices 800 in the computing device cluster may also be used to execute some of the instructions of the power consumption controller for executing the power consumption control method. In other words, the combination of one or more computing devices 800 may jointly execute the instructions of the power consumption controller for executing the power consumption control method.

[0180] It should be noted that the memories 806 in different computing devices 800 in the computing device cluster may store different instructions for executing part of the functions of the power consumption controller.

[0181] Figures 8 and 9 illustrate an example in which a power consumption controller is provided within an AI server. In other possible implementations of the embodiments of this application, the power consumption controller may also be independent of the AI ​​server. Based on this, this application also provides a power consumption controller. The power consumption controller may include a processor and a memory. The memory may store instructions for the stage acquisition module 702, the policy determination module 704, and the mode adjustment module 706 shown in Figure 7. The processor reads these instructions to implement the power consumption control method for the AI ​​server.

[0182] The embodiment of the present application also provides a computer-readable storage medium. The computer-readable storage medium can be any available medium that can be stored by a computing device or a data storage device such as a data center containing one or more available media. The available medium can be a magnetic medium (for example, a floppy disk, a hard disk, a tape), an optical medium (for example, a DVD), or a semiconductor medium (for example, a solid-state drive). The computer-readable storage medium includes instructions that instruct the computing device to execute the above-mentioned method for executing the power consumption control method applied to the power consumption controller.

[0183] The present application also provides a computer program product including instructions. The computer program product may be software or a program product including instructions that can be run on a computing device or stored in any available medium. When the computer program product is run on at least one computing device, the at least one computing device executes the power consumption control method described above.

[0184] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the protection scope of the technical solutions of the various embodiments of the present invention.

Claims

1. A method for controlling power consumption of an artificial intelligence (AI) server, characterized in that: The method comprises: Obtaining a processing stage of an AI model in the AI server, wherein the AI model includes at least two processing stages, the AI server includes a central processing unit (CPU) and an accelerator processor (CPU), the accelerator processor is used to train the AI model or use the AI model for inference, and the CPU is used to control the accelerator processor to perform AI model training or inference; Determining a power consumption adjustment strategy for the CPU according to a processing stage of the AI model; The operating mode of the CPU is adjusted according to the power consumption adjustment policy.

2. The method according to claim 1, characterized in that The processing stage of obtaining the AI model in the AI server includes: Obtaining the processing stage of the AI model reported by the CPU or the acceleration processor; or Obtain the state of the acceleration processor, and determine the processing stage of the AI model according to the state of the acceleration processor.

3. The method according to claim 1 or 2, characterized in that When the AI model is a training model, the processing stage includes an idle stage, a training startup stage, a stable training stage or a verification stage; when the AI model is an inference model, the processing stage includes an idle stage, an inference startup stage or a stable inference stage.

4. The method according to any one of claims 1 to 3, characterized in that When the processing stage is a stable training stage or a verification stage, or when the processing stage is a stable inference stage, determining the power consumption adjustment strategy of the CPU according to the processing stage of the AI model includes: Obtaining the status of the acceleration processor; A power consumption adjustment strategy of the CPU is determined according to the state of the acceleration processor.

5. The method according to claim 4, characterized in that The determining of the power consumption adjustment strategy of the CPU according to the state of the acceleration processor includes: When it is determined that the state of the acceleration processor is the first state, determining the power consumption adjustment strategy of the CPU to be the first strategy according to the first state; When it is determined that the state of the acceleration processor is not the first state, the state of the CPU is acquired, where the CPU includes multiple states, and the power consumption adjustment strategy of the CPU is determined to be the second strategy according to the state of the CPU.

6. The method according to any one of claims 1 to 5, characterized in that When the processing stage is an idle stage or a training startup stage, or when the processing stage is an idle stage or an inference startup stage, determining the power consumption adjustment strategy of the CPU according to the processing stage of the AI model includes: Get the status of the CPU; A power consumption adjustment strategy of the CPU is determined according to the state of the CPU.

7. The method according to claim 5 or 6, characterized in that The CPU status is divided according to the load and busyness of the CPU. The load represents the number of tasks executed by the CPU in a unit time, and the busyness represents the utilization rate of the CPU when the CPU executes tasks.

8. A power consumption controller for an artificial intelligence (AI) server, characterized in that: The power consumption controller includes: a stage acquisition module, configured to acquire the processing stage of the AI model in the AI server, wherein the AI model includes at least two processing stages, the AI server includes a central processing unit (CPU) and an accelerator processor, the accelerator processor is configured to train the AI model or perform inference using the AI model, and the CPU is configured to control the accelerator processor to perform AI model training or inference; a strategy determination module, configured to determine a power consumption adjustment strategy for the CPU according to a processing stage of the AI model; A mode adjustment module is used to adjust the working mode of the CPU according to the power consumption adjustment strategy.

9. The power consumption controller according to claim 8, characterized in that: The stage acquisition module is specifically used for: Obtaining the processing stage of the AI model reported by the CPU or the acceleration processor; or Obtain the state of the acceleration processor, and determine the processing stage of the AI model according to the state of the acceleration processor.

10. The power consumption controller according to claim 8 or 9, characterized in that: When the AI model is a training model, the processing stage includes an idle stage, a training startup stage, a stable training stage or a verification stage; when the AI model is an inference model, the processing stage includes an idle stage, an inference startup stage or a stable inference stage.

11. The power consumption controller according to any one of claims 8 to 10, characterized in that: The strategy determination module is specifically used to: When the processing stage is a stable training stage or a verification stage, or when the processing stage is a stable inference stage, obtaining a state of the acceleration processor; A power consumption adjustment strategy of the CPU is determined according to the state of the acceleration processor.

12. The power consumption controller according to claim 11, characterized in that: The strategy determination module is specifically used to: When it is determined that the state of the acceleration processor is the first state, determining the power consumption adjustment strategy of the CPU to be the first strategy according to the first state; When it is determined that the state of the acceleration processor is not the first state, the state of the CPU is acquired, where the CPU includes multiple states, and the power consumption adjustment strategy of the CPU is determined to be the second strategy according to the state of the CPU.

13. The power consumption controller according to any one of claims 8 to 11, characterized in that: The strategy determination module is specifically used to: When the processing stage is an idle stage or a training startup stage, or when the processing stage is an idle stage or an inference startup stage, obtaining a state of the CPU; A power consumption adjustment strategy of the CPU is determined according to the state of the CPU.

14. The power consumption controller according to claim 12 or 13, characterized in that: The CPU status is divided according to the load and busyness of the CPU. The load represents the number of tasks executed by the CPU in a unit time, and the busyness represents the utilization rate of the CPU when the CPU executes tasks.

15. The power consumption controller according to any one of claims 8 to 14, characterized in that: The power consumption controller is deployed in the AI server, or the power consumption controller is deployed in a control server outside the AI server.

16. An artificial intelligence (AI) server, characterized in that: The AI server includes a central processing unit (CPU), an acceleration processor (ACC), and a power consumption controller. The power consumption controller executes computer-readable instructions to enable the AI server to execute the power consumption control method of the AI server according to any one of claims 1 to 7.

17. A computer-readable storage medium, characterized in that The method comprises computer-readable instructions; the computer-readable instructions are used to implement the power consumption control method of the AI server according to any one of claims 1 to 7.

18. A computer program product, characterized in that The method comprises computer-readable instructions; the computer-readable instructions are used to implement the power consumption control method of the AI server according to any one of claims 1 to 7.

Images