Low-energy-consumption neural network reasoning method and device based on service request density prediction
Through the method based on service request density prediction, high-performance or low-energy operators in neural network inference are dynamically selected, which solves the problems of high energy consumption and high power consumption of neural network inference platform, and reduces server power consumption and improves service capabilities. Effect.
Patent Information
- Application Number
- CN202410785986.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-06-18
- Publication Date
- 2024-11-22
AI Technical Summary
High energy consumption and high power consumption operators in neural network inference platforms lead to high cloud inference costs, large heat dissipation overhead, and limited cluster inference service capabilities.
Through the method based on the prediction of service request density, a prediction function of the change in service request density over time is built, high-performance or low-energy consumption operators are dynamically selected, and operator selection is optimized using dynamic programming algorithms to ensure that service delay requirements are required to reduce server power consumption at the same time.
It realizes that when the service delay requirements are met, server power consumption is reduced, cluster energy efficiency is improved, heat dissipation overhead is reduced, and inference service capabilities are improved.
Smart Images

Figure CN119003111A_ABST
Abstract
Description
[Technical field]
[0001] The present invention relates to the field of intelligent information technology, and in particular to a low-energy consumption neural network reasoning method and device based on service request density prediction. [Background technology]
[0002] The operators used in current neural network cloud inference services are all high-performance operators. Although all services can be completed at the fastest speed, both power consumption and energy consumption are maintained at a very high level. With the emergence of large generative models, the demand for neural network inference services is growing, and high energy consumption and high power consumption operators bring challenges to the cloud inference platform:
[0003] 1) High energy consumption operators refer to operators that consume a lot of energy when running. Operator energy consumption = operator power consumption x operator delay. The higher the energy consumption, the greater the electricity cost generated by the GPU onboard, which means a greater inference cost.
[0004] 2) High-power operators refer to operators with high GPU onboard wattage during operation. On the one hand, this will cause excessive heat dissipation energy consumption of the cluster. In some typical computer clusters, heat dissipation energy consumption often accounts for about 50% of the total power consumption of the cluster. The heat dissipation cost is linearly (water cooling) or super-linearly (air cooling) with the GPU power consumption depending on the type of cluster heat dissipation device. On the other hand, before the construction of a computer cluster, the power supply and supporting energy supply devices are designed based on future power consumption forecasts. If the power consumption of the GPU card is too high, the number of load cards that the cluster power supply can support will be fewer, which will ultimately result in the weaker reasoning service capability of the cluster.
[0005] Therefore, a method is needed to dynamically select high-performance or low-energy operators based on service scenarios, so as to reduce cluster energy consumption and improve service capabilities.
[0006] The following relevant patents were retrieved by searching the patent database.
[0007] Related search result 1: U.S. Patent Document Number, US20230117143A1, Name, Efficient learning and using of topologies of neural networks in machine learning, Abstract, A mechanism is described for facilitating learning and application of neural network topologies in machine learning at autonomous machines. A method of embodiments, as described herein, includes monitoring and detecting structure learning of neural networks relating to machine learning operations at a computing device having a processor, and generating a cursive generative model based on one or more topologies of one or more of the neural networks. The method may further include converting the generative model into a discriminative model. This document relates to the application of neural network topologies, but does not involve high-performance or low-energy neural network operators.
[0008] Related search result 2: US Patent Document Number US20200005135A1, Title Optimizing inference for deep - learning neural networks in a heterogeneous system, Abstract Systems, methods, and devices for deploying an artificial neural network (ANN). Candidate ANNs are generated for performing an inference task based on specifications of a target inference device. Trained ANNs are generated by training the candidate ANNs to perform the inference task on an inference device conforming to the specifications. Characteristics describing the trained ANNs performance of the inference task on a device conforming to the specifications are determined. Profiles that reflect the characteristics of each trained ANN are stored. The stored profiles are queried based on requirements of an application to select an ANN from among the trained ANNs. The selected ANN is deployed on an inference device conforming to the target inference device specifications. Input data is communicated to the deployed ANN from the application.An output is generated using the deployed ANN, and the output is communicated to the application. This document aims at accelerating neural network reasoning in heterogeneous systems and does not consider the issue of low power consumption.
[0009] The present invention aims at the technical problem that a neural network inference platform has high energy consumption and high power consumption operators, and makes technical improvements to a neural network inference method and device. [Summary of the invention]
[0010] The purpose of the present invention is to provide a neural network reasoning method that can reduce server power consumption while meeting service delay requirements.
[0011] To achieve the above object, the technical solution adopted by the present invention is a low-energy neural network reasoning method based on service request density prediction, comprising the following steps:
[0012] S1. The neural network inference server constructs a service request density prediction function group that changes the service request density over time during the inference service time period according to the tidal phenomenon of the neural network running service request density;
[0013] S2, the neural network inference server selects operators with different performance for each operation in the directed acyclic graph running the neural network;
[0014] S3. The neural network running scheduler on the neural network inference server selects an operator with appropriate performance for each operation under the delay condition that meets the service requirements based on the predicted service request density of the next time the neural network runs, ultimately achieving the lowest power consumption while the neural network runs to meet the delay requirements.
[0015] Preferably, the tidal phenomenon of the service request density in step S1 refers to: the service request density presents a Poisson distribution in the short term, and the service request density has a tidal effect in the long term, with more requests during the day and fewer requests at night.
[0016] Preferably, the service request density prediction function group constructed in step S1 includes:
[0017] The service request density prediction function is used to predict the service request density Current_state at the current time based on the service request density last_state of the previous operation of the neural network;
[0018] The probability function of the service request density increase at the current time is used to give the probability possibility of the service request density increase at the current time based on historical experience Possibility4risin;
[0019] A service request density adjustment function is used to obtain a service request density adjustment k=Current_k(time) related to the current time time;
[0020] next_state=Current_state+k*possibility, the predicted service request density of the next operation run by the neural network is equal to the service request density of the current operation plus the predicted service request density increase.
[0021] Preferably, step S2 includes the following sub-steps:
[0022] S21. Use the open source operator generation framework to search for all possible operator implementations and find available operators with large performance differences in power consumption and latency.
[0023] S22. For each operation, select a high-performance operator with the highest performance and the highest power consumption, a balanced operator with slightly weaker performance and lower power consumption, and a low-power operator with the lowest performance and the lowest power consumption.
[0024] Preferably, the operator selection problem in step S3 is mathematically described as follows: there are N operations in the directed acyclic graph of the neural network, and each operation has high-performance operators, balanced operators, and low-power operators to choose from. Then the entire neural network runs with 3 to the power of N inference schemes, satisfying the minimum total operator running time Kernels=argmin(totalInferenceEnergy(Kernels)), and satisfying the total operator running time to meet the latency requirement Constraint: totalInferenceTime(Kernels)>servicerequired latency.
[0025] Preferably, step S3: using a dynamic programming algorithm to solve the operator selection problem, the delay and power consumption of operators with different performance are measured in advance to form a lookup table, which is convenient for the use of the dynamic programming algorithm, and finally the operator selection in the current scenario is given by real-time table lookup.
[0026] Another object of the present invention is to provide a neural network reasoning device that can reduce server power consumption while meeting service delay requirements.
[0027] To achieve the above-mentioned another purpose, the technical solution adopted by the present invention is a low-energy neural network reasoning device based on service request density prediction, which is used to execute the above-mentioned low-energy neural network reasoning method based on service request density prediction.
[0028] The beneficial effects of the low-energy neural network reasoning method and device based on service request density prediction of the present invention are as follows: according to the tidal phenomenon of requests in the reasoning service, the change of the density of requests over time in the reasoning service time period is accurately predicted; by predicting the traffic of the neural network cloud reasoning service, the high-performance / low-energy operator used for neural network reasoning is selected according to the predicted traffic size, which can reduce the server power consumption and achieve the lowest service energy consumption while meeting the service delay requirements.
Brief Description of the Drawings
[0029] Figure 1 It is a step diagram of a low-energy neural network inference method based on service request density prediction.
[0030] Figure 2 This is a schematic diagram of different characteristic operator selection for a low-energy neural network inference method based on service request density prediction.
[0031] Figure 3 It is a low-energy neural network inference method based on service request density prediction. It dynamically selects different characteristic operators for each operation. Schematic diagram, where op1, op2, and op3 are different operations in the directed acyclic graph of the neural network operation, kernel0 is a high-performance operator, kernel1 is a balanced operator, and kernel2 is a low-power operator. [Specific implementation method]
[0032] The features and exemplary embodiments of various aspects of the present invention will be described in detail below. In the detailed description below, many specific details are proposed to provide a comprehensive understanding of the present invention. But it is obvious to those skilled in the art that the present invention can be implemented without the need for some of these specific details. The following description of the embodiments is only to provide a better understanding of the present invention by illustrating examples of the present invention. The present invention is by no means limited to any specific configuration and algorithm proposed below, but covers any modification, replacement and improvement of elements, parts and algorithms without departing from the concept of the present invention. In the accompanying drawings and the following description, known structures and technologies are not shown to avoid unnecessary ambiguity to the present invention.
[0033] Example
[0034] This embodiment implements a low-energy neural network reasoning method based on service request density prediction.
[0035] Figure 1 It is a step diagram of a low-energy neural network inference method based on service request density prediction. Figure 1As shown, the method of this embodiment accurately predicts the change of request density over time within the inference service time period based on the tidal phenomenon of requests in the inference service, and designs a method that can dynamically select inference operators according to the request density, which can reduce server power consumption while meeting the service delay requirements.
[0036] 1. Observation of service request density
[0037] According to the statistics of typical server request traffic, we found that:
[0038] 1) Service traffic in the short term (in minutes) presents a Poisson distribution (a discrete probability distribution commonly seen in statistics and probability);
[0039] 2) Service traffic has a tidal effect in the long term (in hours), with more requests during the day and fewer requests at night.
[0040] 2. Service request (traffic) density prediction function of this embodiment
[0041] Function pseudo code:
[0042] def request_density_prediction(time,last_state):#time refers to the current time, last_state refers to the traffic density of the previous state
[0043] possibility = Possibility4rising(time) #Possibility4rising can give the probability of traffic increase at the current time based on historical experience
[0044] k=Current_k(time) #k is a flow density adjustment value related to the current time
[0045] next_state=Current_state+k*possibility#The traffic density of the next state is equal to the current traffic plus the predicted traffic increase
[0046] return next_state#Output next_state. The operator selection for the next time is determined according to next_state.
[0047] 3. Method for selecting operators with different characteristics in this embodiment (high-performance operators or low-power operators)
[0048] Figure 2 This is a schematic diagram of different characteristic operators selected based on low-energy neural network reasoning method for service request density prediction. Figure 2As shown, taking the matrix multiplication operator of the same size as an example, this embodiment uses an open source operator generation framework to search for all possible operator implementations, and finds that the available operators found have great differences in power consumption and latency:
[0049] 1) The operator of ① has the best performance but the highest power consumption;
[0050] 2) The power consumption of operators gradually decreases along the black line, but the performance also gradually deteriorates. Compared with the highest performance operator, the performance of the operators before ② on the black line is reduced by 10% to 20%, but the power consumption is reduced by 30% to 40%;
[0051] 3) This embodiment selects the operators at the left starting point of the black line and the two turning points ② and ③ as high-performance, balanced, and low-power operators, respectively. During neural network reasoning, the appropriate operator will be selected according to the service density of the current scenario.
[0052] 4. This embodiment dynamically selects the framework of high-performance or low-energy consumption operators
[0053] Figure 3 It is a low-energy neural network reasoning method based on service request density prediction to dynamically select different characteristic operators for each operation. Figure 3 As shown in the figure, op is the operation (operator) in the directed acyclic graph of the neural network operation, such as convolution, matrix multiplication, etc.; the operator (kernel) is the operator with different characteristics generated by the operation (operator) using the operator generation method in step 3, kernel0 is a high-performance operator (highest performance, highest power consumption), kernel1 is a balanced operator (slightly weaker performance, lower power consumption), and kernel2 is a low-power operator (lowest performance, lowest power consumption).
[0054] The scheduler selects the appropriate kernel for each operation based on the predicted traffic next_state of the next state and the latency required by the service, ultimately achieving the lowest power consumption while meeting the latency requirements.
[0055] 5. Kernel selection algorithm in the neural network of this embodiment
[0056] Since there are hundreds of operators in the directed acyclic graph of a neural network, and each op has three different operators (kernels) to choose from, the entire network has 3^N (3 to the power of N) inference schemes, where N is the number of ops in the network. The problem of selecting an operator (kernel) can be described mathematically as follows:
[0057] Kernels = argmin (totalInferenceEnergy (Kernels)) # kernel (operator) running total time should be minimum;
[0058] Constraint: totalInferenceTime(Kernels)>service required latency#The total kernel (operator) running time should meet the system's latency requirements.
[0059] This embodiment uses a dynamic programming algorithm to solve this problem. The latency and energy of the kernels need to be measured in advance to form a lookup table to facilitate the use of the dynamic programming algorithm, and finally give the kernel selection in the current scenario in real time.
[0060] The core feature of this embodiment is to protect the framework of dynamically selecting high-performance or low-energy operators. Through this framework, it is possible to select a suitable operator (kernel) for each operation (operator) under the delay condition that meets the service requirements, and ultimately achieve the lowest power consumption while meeting the delay requirements.
[0061] Those skilled in the art will appreciate that all or part of the steps to implement the above embodiments may be accomplished by hardware, or may be accomplished by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, wherein the storage medium may be a magnetic disk, an optical disk, a read-only memory (ROM) or a random access memory (RAM), etc.
[0062] The above is only a preferred embodiment of the present invention. It should be pointed out that ordinary technicians in this technical field can make several improvements and supplements without departing from the principle of the present invention. These improvements and supplements should also be regarded as the scope of protection of the present invention.
Claims
1. A low-energy neural network reasoning method based on service request density prediction, characterized in that The following steps are involved: S1. The neural network inference server constructs a service request density prediction function group that changes the service request density over time during the inference service time period according to the tidal phenomenon of the neural network running service request density; S2, the neural network inference server selects operators with different performance for each operation in the directed acyclic graph running the neural network; S3. The neural network running scheduler on the neural network inference server selects an operator with appropriate performance for each operation under the delay condition that meets the service requirements based on the predicted service request density of the next time the neural network runs, ultimately achieving the lowest power consumption while the neural network runs to meet the delay requirements.
2. According to claim 1, a low-energy neural network reasoning method based on service request density prediction is characterized in that The tidal phenomenon of the service request density in step S1 refers to: the service request density presents a Poisson distribution in the short term, and the service request density has a tidal effect in the long term, with more requests during the day and fewer requests at night.
3. A low-energy neural network reasoning method based on service request density prediction according to claim 2, characterized in that The service request density prediction function group constructed in step S1 includes: The service request density prediction function is used to predict the service request density Current_state at the current time based on the service request density last_state of the previous operation of the neural network; The probability function of the service request density increase at the current time is used to give the probability possibility of the service request density increase at the current time based on historical experience Possibility4risin; A service request density adjustment function is used to obtain a service request density adjustment k=Current_k(time) related to the current time time; next_state=Current_state+k*possibility, the predicted service request density of the next operation run by the neural network is equal to the service request density of the current operation plus the predicted service request density increase.
4. A low-energy neural network reasoning method based on service request density prediction according to claim 3, characterized in that Step S2 includes the following sub-steps: S21. Use the open source operator generation framework to search for all possible operator implementations and find available operators with large performance differences in power consumption and latency. S22. For each operation, select a high-performance operator with the highest performance and the highest power consumption, a balanced operator with slightly weaker performance and lower power consumption, and a low-power operator with the lowest performance and the lowest power consumption.
5. A low-energy neural network reasoning method based on service request density prediction according to claim 4, characterized in that Mathematical description of the operator selection problem in step S3: There are N operations in the directed acyclic graph of the neural network, and each operation has high-performance operators, balanced operators, and low-power operators to choose from. Then the entire neural network runs with 3N inference schemes, which satisfies the minimum total operator running time Kernels = argmin(totalInferenceEnergy(Kernels)) and the total operator running time meets the latency requirement Constraint: totalInferenceTime(Kernels)>service requiredlatency.
6. A low-energy neural network reasoning method based on service request density prediction according to claim 5, characterized in that Step S3: Use a dynamic programming algorithm to solve the operator selection problem. The delay and power consumption of operators with different performance are measured in advance to form a lookup table to facilitate the use of the dynamic programming algorithm. Finally, the real-time table lookup gives the operator selection in the current scenario.
7. A low-energy neural network reasoning device based on service request density prediction, characterized in that: Used to execute a low-energy neural network reasoning method based on service request density prediction as described in any one of claims 1 to 6.
