Usage performance evaluation method and device, equipment, storage medium and product

Abstract

The invention relates to a use performance evaluation method and device, equipment, a storage medium and a product. The method comprises the following steps: acquiring a task type of a computing power task; the task type comprises a training task or a reasoning task; determining computing power energy efficiency information corresponding to the task type according to the task type; the computing power energy efficiency information is acquired under the condition of executing the computing power task; and evaluating use performance parameters of the computing power task according to the computing power energy efficiency information. By adopting the method, the real performance of AI large model training and reasoning under the green computing power condition can be accurately evaluated, the model effect is not sacrificed while energy conservation and carbon reduction are ensured, and the'low-carbon, efficient and auditable 'green computing goal is achieved.

Description

Technical Field

[0001] This application relates to the field of emerging information technology, and in particular to a method, apparatus, device, storage medium and product for evaluating performance. Background Technology

[0002] With the advent of the Artificial Intelligence (AI) era, the computing power of data centers has surged. Assessing the carbon emissions and resource utilization efficiency generated by this surge in computing power has become an important research direction for energy conservation and emission reduction. However, as green electricity is gradually applied to data centers, they acquire green computing power. The computing power used for large-scale model training and inference tasks based on this green computing power requires evaluation of its performance during execution.

[0003] Current assessment methods mainly rely on macro-level certification or simply on the geographical matching of computing resources, which is no longer sufficient to meet the performance evaluation needs of green computing power in large-scale model training and inference scenarios in the AI ​​era.

[0004] Therefore, how to accurately evaluate the performance of green computing power in large model training and inference scenarios in the AI ​​era has become an urgent problem to be solved. Summary of the Invention

[0005] Therefore, it is necessary to provide a method, device, equipment, storage medium, and product for evaluating the performance of green computing power in large model training and inference scenarios in the AI ​​era, in order to address the above-mentioned technical problems.

[0006] Firstly, this application provides a method for evaluating performance, including:

[0007] The task type for acquiring computing power; the task type includes training tasks or inference tasks.

[0008] Based on the task type, determine the computing power efficiency information corresponding to the task type; the computing power efficiency information is collected when executing computing power tasks;

[0009] Based on computing power energy efficiency information, evaluate the performance parameters of computing power tasks.

[0010] In one embodiment, the task type is a training task, and the computing power efficiency information includes at least one of the following computing power efficiency parameters:

[0011] Throughput, unit energy consumption, carbon emission intensity, and hardware loss.

[0012] In one embodiment, the task type is a reasoning task, and the computing power efficiency information includes at least one of the following computing power efficiency parameters:

[0013] Monitor latency, energy consumption per query request, and start / stop losses.

[0014] In one embodiment, the aforementioned computing power efficiency information includes multiple computing power efficiency parameters;

[0015] Based on computing power efficiency information, evaluate the performance parameters of computing power tasks, including:

[0016] Obtain the weights corresponding to multiple computing power efficiency parameters;

[0017] Based on the weights, multiple computing power energy efficiency parameters are weighted to obtain the performance parameters of the computing power task.

[0018] In one embodiment, the method further includes:

[0019] If the performance parameters of the computing power task do not meet the preset performance parameters, then determine the task execution strategy for optimizing computing power energy efficiency information;

[0020] The computing power task is re-executed based on the task execution strategy to obtain new computing power energy efficiency information of the computing power task, and then the execution is returned to evaluate the usage performance parameters of the computing power task based on the computing power energy efficiency information, until the usage performance parameters of the computing power task meet the preset performance parameters.

[0021] In one embodiment, the above-described task execution strategy for optimizing computing power efficiency information includes:

[0022] Based on the difference between the performance parameters used by the computing task and the preset performance parameters, and the performance parameters used by the computing task, a task execution strategy for optimizing computing power energy efficiency information is determined.

[0023] In one embodiment, determining the computing power efficiency information corresponding to the task type includes:

[0024] Based on the acquisition plugin or probe, collect computing power information corresponding to the task type;

[0025] Based on the power grid carbon tracing system, collect power information corresponding to the task type;

[0026] Based on the network infrastructure system, network information corresponding to the task type is collected.

[0027] Secondly, this application also provides a performance evaluation device, comprising:

[0028] The acquisition module is used to acquire the task type of the computing power task; the task type includes training tasks or inference tasks.

[0029] The determination module is used to determine the computing power efficiency information corresponding to the task type; the computing power efficiency information is collected when executing computing power tasks;

[0030] The evaluation module is used to evaluate the performance parameters of computing tasks based on computing power energy efficiency information.

[0031] Thirdly, this application also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to perform the following steps:

[0032] The task type for acquiring computing power; the task type includes training tasks or inference tasks.

[0033] Based on the task type, determine the computing power efficiency information corresponding to the task type; the computing power efficiency information is collected when executing computing power tasks;

[0034] Based on computing power energy efficiency information, evaluate the performance parameters of computing power tasks.

[0035] Fourthly, this application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, performs the following steps:

[0036] The task type for acquiring computing power; the task type includes training tasks or inference tasks.

[0037] Based on the task type, determine the computing power efficiency information corresponding to the task type; the computing power efficiency information is collected when executing computing power tasks;

[0038] Based on computing power energy efficiency information, evaluate the performance parameters of computing power tasks.

[0039] Fifthly, this application also provides a computer program product, including a computer program that, when executed by a processor, performs the following steps:

[0040] The task type for acquiring computing power; the task type includes training tasks or inference tasks.

[0041] Based on the task type, determine the computing power efficiency information corresponding to the task type; the computing power efficiency information is collected when executing computing power tasks;

[0042] Based on computing power energy efficiency information, evaluate the performance parameters of computing power tasks.

[0043] The aforementioned performance evaluation methods, devices, equipment, storage media, and products distinguish the task types of computing power tasks, determine the computing power energy efficiency information corresponding to the task types, and evaluate the performance parameters of computing power tasks based on the computing power energy efficiency information corresponding to the task types. This enables accurate evaluation of the real performance of AI large model training and inference under green computing power conditions, ensuring that energy conservation and carbon reduction are not sacrificed without sacrificing model performance, and achieving the goal of "low-carbon, high-efficiency, and auditable" green computing. Attached Figure Description

[0044] To more clearly illustrate the technical solutions in the embodiments of this application or related technologies, the drawings used in the description of the embodiments of this application or related technologies will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0045] Figure 1 This is a diagram illustrating the application environment of a performance evaluation method used in one embodiment.

[0046] Figure 2 This is a flowchart illustrating a performance evaluation method used in one embodiment;

[0047] Figure 3 This is a schematic diagram of the performance evaluation index system in one embodiment;

[0048] Figure 4 This is a flowchart illustrating the performance evaluation method used in another embodiment;

[0049] Figure 5 This is a flowchart illustrating the performance evaluation method used in another embodiment;

[0050] Figure 6 This is a flowchart illustrating the performance evaluation method used in another embodiment;

[0051] Figure 7 This is a flowchart illustrating the performance evaluation method used in another embodiment;

[0052] Figure 8 This is a structural block diagram of a performance evaluation device used in one embodiment. Detailed Implementation

[0053] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0054] With the advent of the Artificial Intelligence (AI) era, the computing power of data centers has surged. Assessing the carbon emissions and resource utilization efficiency generated by this surge in computing power has become an important research direction for energy conservation and emission reduction. However, as green electricity is gradually applied to data centers, they acquire green computing power. The computing power used for large-scale model training and inference tasks based on this green computing power requires evaluation of its performance during execution.

[0055] Current assessment methods mainly rely on macro-level certification or simply on the geographical matching of computing resources, which is no longer sufficient to meet the performance evaluation needs of green computing power in large-scale model training and inference scenarios in the AI ​​era.

[0056] Therefore, accurately evaluating the performance of green computing power in large-scale model training and inference scenarios in the AI ​​era has become an urgent problem to be solved. This application provides a performance evaluation method aimed at accurately assessing the performance of green computing power in large-scale model training and inference scenarios in the AI ​​era.

[0057] Having described the background technology of the performance evaluation method provided in the embodiments of this application, the implementation environment involved in the performance evaluation method provided in the embodiments of this application will be briefly described below. The performance evaluation method provided in the embodiments of this application can be applied to, for example... Figure 1 The internal structure diagram of the computer device shown can be as follows: Figure 1As shown, the computer device includes a processor, memory, input / output interfaces, a communication interface, a display unit, and an input device. The processor, memory, and input / output interfaces are connected via a system bus, and the communication interface, display unit, and input device are also connected to the system bus via the input / output interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides the environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The input / output interfaces are used for exchanging information between the processor and external devices, including compact discs (CDs), digital versatile discs (DVDs), or universal serial bus (USB) flash drives. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, mobile cellular networks, near field communication (NFC), or other technologies. When the computer program is executed by the processor, it implements a performance evaluation method. The display unit of this computer device is used to form a visually visible image and can be a display screen, a projection device, or a virtual reality imaging device. The display screen can be an LCD screen or an e-ink screen. The input device of this computer device can be a touch layer covering the display screen, or buttons, a trackball, or a touchpad set on the casing of the computer device, or an external keyboard, touchpad, or mouse, etc.

[0058] The technical solution of this application and how the technical solution of this application solves the above-mentioned technical problems are described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will now be described with reference to the accompanying drawings.

[0059] In one embodiment, such as Figure 2 As shown, a performance evaluation method is provided, which is then applied to... Figure 1 Taking a computer device as an example, the explanation includes the following steps:

[0060] S201. Obtain the task type of the computing power task; the task type includes training task or inference task.

[0061] Among them, the training task refers to the task that the computer device is performing to train a certain model. For example, the training task can be the training task of a Long Short-Term Memory (LSTM) neural network model, etc.

[0062] Inference tasks refer to tasks in which computer devices use trained models to perform operations such as recognition and classification. For example, an inference task could be an image classification task using an LSTM neural network model.

[0063] In this embodiment, before analyzing the performance parameters of the computing task, it is necessary to first obtain the task type of the computing task. Optionally, the task type of the computing task can be obtained by analyzing the operating data of the processor in the computer device, or it can be determined by the type identifier of the computing task in the processor of the computer device. It should be noted that this application does not limit the specific method of obtaining the task type of the computing task. As long as the method can obtain the task type of the computing task, it is within the protection scope of this application.

[0064] S202. Based on the task type, determine the computing power efficiency information corresponding to the task type; the computing power efficiency information is collected when the computing power task is executed.

[0065] When the task type is inference task, the computing power efficiency information includes at least one of the following computing power efficiency parameters: monitoring latency, energy consumption per query request, and start-stop loss.

[0066] When the task type is a training task, the computing power efficiency information includes at least one of the following computing power efficiency parameters: throughput, unit energy consumption, carbon emission intensity, and hardware loss.

[0067] In this embodiment, if the task type of the computing power task obtained above is a reasoning task, then the computing power energy efficiency information corresponding to the reasoning task is determined, and the computing power energy efficiency information corresponding to the reasoning task is collected from the acquisition plug-in or probe deployed on the computer device, the power grid carbon tracking system and the network basic capability system, namely at least one of monitoring latency, single query request energy consumption and start-stop loss.

[0068] Optionally, if the task type of the computing power task obtained above is a training task, then the computing power energy efficiency information corresponding to the training task is determined, and the computing power energy efficiency information corresponding to the training task is collected from the acquisition plug-in or probe deployed on the computer device, the power grid carbon tracking system and the network infrastructure capability system, namely at least one of throughput, unit energy consumption, carbon emission intensity and hardware loss.

[0069] It should be noted that, see Figure 3 The evaluation indicators corresponding to computing power energy efficiency information include the following three categories;

[0070] (1) Efficiency indicators: computing power efficiency, the effective computing power generated per unit of electricity consumption; power efficiency, the electricity resource consumption corresponding to the unit computing power demand;

[0071] (2) Green indicators: Green electricity consumption rate, the proportion of renewable energy used in computing power clusters; Carbon intensity, the carbon emissions per unit of computing power output;

[0072] (3) Spatiotemporal indicators: resource matching degree, the spatiotemporal overlap rate of power supply and computing power demand (such as the proportion of computing power load during peak hours of wind and solar power generation); dispatch delay, the response time from power fluctuations to computing power adjustment.

[0073] S203. Based on the computing power energy efficiency information, evaluate the performance parameters of the computing power task.

[0074] In this embodiment, after obtaining computing power energy efficiency information, the computing power energy efficiency information can be analyzed to evaluate the performance parameters of computing power tasks.

[0075] Optionally, if the computing power and energy efficiency information corresponding to the training task is obtained, the throughput, unit energy consumption, carbon emission intensity, and hardware loss can be weighted and calculated to obtain the performance parameters of the training task. Optionally, if the computing power and energy efficiency information corresponding to the inference task is obtained, the monitoring latency, energy consumption per query request, and start / stop loss can be weighted and calculated to obtain the performance parameters of the training task.

[0076] In this embodiment, by distinguishing the task types of computing power tasks, determining the computing power energy efficiency information corresponding to the task type, and evaluating the usage performance parameters of computing power tasks based on the computing power energy efficiency information corresponding to the task type, it is possible to accurately evaluate the real performance of AI large model training and inference under green computing power conditions, ensuring that energy saving and carbon reduction are achieved without sacrificing model performance, and realizing the green computing goal of "low carbon, high efficiency, and auditability".

[0077] In this embodiment, when the computing power efficiency information includes multiple computing power efficiency parameters, the performance parameters of the current task can be evaluated based on these multiple parameters. (See [link to relevant documentation]). Figure 4 The aforementioned S203 includes:

[0078] S301. Obtain the weights corresponding to multiple computing power efficiency parameters.

[0079] In this embodiment, when it is necessary to evaluate the performance parameters of computing power tasks, the weights corresponding to multiple computing power energy efficiency parameters can be obtained first.

[0080] Optionally, the weights corresponding to each computing power energy efficiency parameter can be predetermined, or the weights corresponding to each computing power energy efficiency parameter can be determined based on the entropy weight method. It should be noted that this application does not restrict the method of obtaining the weights corresponding to each computing power energy efficiency parameter; any method that can determine the weights corresponding to multiple computing power energy efficiency parameters is within the scope of protection of this application.

[0081] Optionally, for training tasks, the computing power efficiency parameters include throughput, unit energy consumption, carbon emission intensity, and hardware loss. Therefore, it is necessary to obtain the weights a1 for throughput, b1 for unit energy consumption, c1 for carbon emission intensity, and d1 for hardware loss.

[0082] Optionally, for inference tasks, the computing power efficiency parameters include monitoring latency, energy consumption per query request, and start-stop loss. Therefore, it is necessary to obtain the weights a2 of monitoring latency, b2 of energy consumption per query request, and c2 of start-stop loss.

[0083] S302. Based on the weights, multiple computing power energy efficiency parameters are weighted to obtain the performance parameters of the computing power task.

[0084] In this embodiment, after obtaining the weights corresponding to each computing power energy efficiency parameter, the computing power energy efficiency parameters can be weighted based on their respective weights to obtain the performance parameters of the computing power task.

[0085] Optionally, for the training task, throughput A is weighted according to throughput weight a1, unit energy consumption B is weighted according to unit energy consumption weight b1, carbon emission intensity C is weighted according to carbon emission intensity weight c1, and hardware loss D is weighted according to hardware loss weight d1. The weighted results are then summed to obtain the performance parameters of the training task. For example, the performance parameter X of the training task can be determined using the following formula (1):

[0086]

[0087] Optionally, for the inference task, the monitoring latency E is weighted according to the weight a2 of the monitoring latency, the energy consumption of a single query request is weighted according to the weight b2 of the energy consumption of a single query request, and the start-stop loss is weighted according to the weight c2 of the start-stop loss, and the weighted results are summed to obtain the performance parameters of the inference task. For example, the performance parameters Y of the inference task can be determined by the following formula (2):

[0088]

[0089] In this embodiment, by obtaining the weights corresponding to each computing power energy efficiency parameter, and performing weighted processing on each computing power energy efficiency parameter based on the weights corresponding to each computing power energy efficiency parameter, the performance parameters of the computing power task are obtained. This ensures that the real performance of AI large model training and inference under green computing power conditions can be accurately evaluated, ensuring that energy saving and carbon reduction are achieved without sacrificing model performance, and realizing the green computing goal of "low carbon, high efficiency, and auditability".

[0090] In this embodiment, if the performance parameters of the computing task do not meet the preset performance parameters, a corresponding optimization strategy can be executed to ensure that the performance parameters of the computing task meet the preset performance parameters. This process is described in detail in [reference needed]. Figure 5 The above method also includes:

[0091] S204. If the performance parameters of the computing power task do not meet the preset performance parameters, then determine the task execution strategy for optimizing computing power energy efficiency information.

[0092] In this embodiment, after obtaining the performance parameters of the computing power task, it is determined whether the performance parameters of the computing power task meet the preset performance parameters. If the performance parameters of the computing power task meet the preset performance parameters, no operation is performed. If the performance parameters of the computing power task do not meet the preset performance parameters, a task execution strategy for optimizing computing power energy efficiency information is determined based on the difference between the performance parameters of the computing power task and the preset performance parameters.

[0093] S205. Re-execute the computing power task based on the task execution strategy, obtain new computing power energy efficiency information of the computing power task, and return to execute S203 until the usage performance parameters of the computing power task meet the preset performance parameters.

[0094] In this embodiment, after determining the task execution strategy, the computing task can be re-executed based on the task execution strategy. During the execution of the computing task, or after the execution of the computing task, new computing power efficiency information of the computing task is obtained, and the usage performance parameters of the computing task are re-evaluated based on the computing power efficiency information. Then, it is determined again whether the usage performance parameters of the computing task meet the preset performance parameters. If the usage performance parameters of the computing task meet the preset performance parameters, no operation is performed. If the usage performance parameters of the computing task do not meet the preset performance parameters, a task execution strategy for optimizing the computing power efficiency information is determined based on the difference between the usage performance parameters of the computing task and the preset performance parameters. The computing task is re-executed based on the task execution strategy, and during the execution of the computing task, or after the execution of the computing task, new computing power efficiency information of the computing task is obtained, and the usage performance parameters of the computing task are re-evaluated based on the computing power efficiency information. Then, it is determined again whether the usage performance parameters of the computing task meet the preset performance parameters, ..., until the usage performance parameters of the computing task meet the preset performance parameters.

[0095] In this embodiment, when the performance parameters of the computing task do not meet the preset performance parameters, a task execution strategy for optimizing computing power energy efficiency information is determined, and the computing task is re-executed based on the task execution strategy. Then, new computing power energy efficiency information of the computing task is obtained, and the performance parameters of the computing task are evaluated based on the computing power energy efficiency information until the performance parameters of the computing task meet the preset performance parameters. This ensures that the performance parameters of the computing task always meet the preset performance parameters, thereby ensuring that energy saving and carbon reduction are not sacrificed without sacrificing model performance, and achieving the green computing goal of "low carbon, high efficiency, and auditability".

[0096] In this embodiment, the detailed process of determining the task execution strategy for optimizing computing power efficiency information can also be explained. In an exemplary embodiment, the above-mentioned S204 includes:

[0097] Based on the difference between the performance parameters used by the computing task and the preset performance parameters, and the performance parameters used by the computing task, a task execution strategy for optimizing computing power energy efficiency information is determined.

[0098] In this embodiment, when the performance parameters of the computing power task do not meet the preset performance parameters, the difference between the performance parameters of the computing power task and the preset performance parameters is obtained, and a task execution strategy for optimizing computing power efficiency information is determined based on the difference between the performance parameters of the computing power task and the preset performance parameters and the performance parameters of the computing power task.

[0099] Optionally, by comparing the deviation between the performance parameters of the computing power task and the preset performance parameters in real time, strategies such as frequency adjustment, precision reduction, parallelism modification, or batch size adjustment can be automatically generated and distributed to minimize power consumption while ensuring that performance is not reduced, and the energy efficiency baseline is refreshed simultaneously.

[0100] In this embodiment, the difference between the performance parameters used by the computing power task and the preset performance parameters, and the performance parameters used by the computing power task, are used to determine the task execution strategy for optimizing computing power energy efficiency information. This ensures that the real performance of AI large model training and inference under green computing power conditions can be accurately evaluated, ensuring that energy saving and carbon reduction are achieved without sacrificing model performance, and realizing the green computing goal of "low carbon, high efficiency, and auditability".

[0101] In this embodiment, the detailed process of obtaining computing power efficiency information corresponding to the task type can also be explained. In an exemplary embodiment, such as Figure 6 As shown, the above steps include:

[0102] S501. Based on the acquisition plugin or probe, collect computing power information corresponding to the task type.

[0103] The computing power information includes task type (e.g., training task, inference task, etc.), task priority, task energy consumption, and graphics processing unit (GPU) resource usage (e.g., GPU resource information, GPU status information, and power consumption information during task execution, etc.). It should be noted that the corresponding computing power information needs to be obtained for both training and inference tasks.

[0104] In this embodiment, a power consumption acquisition plugin or probe can be deployed on a computer device to collect computing power information corresponding to the task type.

[0105] S502. Based on the power grid carbon tracing system, collect power information corresponding to the task type.

[0106] The electricity information includes electricity price data, green electricity consumption ratio, green electricity share, clean energy share, and electricity load. It should be noted that both training and inference types require obtaining the corresponding electricity information.

[0107] In this embodiment, the computer device can be connected to the power grid carbon tracing system. When the computer device needs to collect electricity information, it can send a query command to the power grid carbon tracing system to query the electricity information corresponding to the task type.

[0108] S503. Based on the network infrastructure system, collect network information corresponding to the task type.

[0109] The network information includes bandwidth, latency, jitter, and other data related to the execution of computing tasks. It's important to note that this network information is required for both training and inference tasks.

[0110] In this embodiment, the computer device can be connected to the network infrastructure system. When the computer device needs to collect network information, it can send a query command to the network infrastructure system to query the network information corresponding to the task type.

[0111] It should be noted that the computing power information, power information, and network information corresponding to the task type together constitute the computing power energy efficiency information corresponding to the task type.

[0112] In this embodiment, computing power energy efficiency information corresponding to the task type is obtained through a data acquisition plug-in, probe, power grid carbon tracking system and network infrastructure system, providing a data foundation for subsequent evaluation of the performance parameters of computing power tasks based on computing power energy efficiency information.

[0113] In one exemplary embodiment, such as Figure 7 It also provides a method for evaluating performance, including:

[0114] T1. The task type for acquiring computing power; the task type includes training tasks or inference tasks.

[0115] T2. Based on the task type, determine multiple computing power efficiency parameters corresponding to the task type; these multiple computing power efficiency parameters are collected when executing computing power tasks.

[0116] T3. Obtain the weights corresponding to multiple computing power efficiency parameters;

[0117] T4. Based on the weights, multiple computing power energy efficiency parameters are weighted to obtain the performance parameters of the computing power task.

[0118] T5. If the performance parameters of the computing power task do not meet the preset performance parameters, then the task execution strategy for optimizing computing power energy efficiency information is determined based on the difference between the performance parameters of the computing power task and the preset performance parameters, and the performance parameters of the computing power task.

[0119] T6. Re-execute the computing power task based on the task execution strategy, obtain new computing power energy efficiency information of the computing power task, and return to execute step T4 until the usage performance parameters of the computing power task meet the preset performance parameters.

[0120] It should be noted that the descriptions of T1-T6 above can be found in the relevant descriptions in the above embodiments, and their effects are similar, so they will not be repeated here.

[0121] This embodiment proposes a novel indicator-based evaluation system integrating computing power, power, and network. Compared to traditional evaluation methods that only focus on the performance of intelligent computing services themselves, this method innovatively links parameters such as network indicators, power indicators, and computing power indicators directly to high-energy-consuming tasks, constructing a quantifiable performance evaluation system and optimization method. Through refined process testing of large-scale model training and inference in different scenarios, an intuitive mapping from parameters such as efficiency, greenness, and spatiotemporal characteristics to green electricity usage performance is achieved, enabling users to accurately and dynamically match the best computing resources based on actual AI business needs. In addition, an innovative system for evaluating and optimizing the performance of green computing power utilization in large-scale model training and inference scenarios is proposed. This system breaks through the limitations of traditional single data sources and single objectives. It innovatively proposes an advanced comprehensive evaluation and optimization system that integrates heterogeneous data collection from computing, network, and electricity, a dual-modal accurate dynamic evaluation engine for large-scale model training / inference scenarios, and intelligent optimization. This system solves the problems of mismatch between computing power resources and green electricity consumption, and low computing efficiency that arise after the emergence of large-scale model business. It establishes an evaluation-optimization-feedback-re-evaluation mechanism between green computing power utilization performance indicators and large-scale model training and inference tasks, continuously optimizing the performance of green computing power utilization, promoting green electricity consumption, and reducing carbon emissions.

[0122] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0123] Based on the same inventive concept, this application also provides a performance evaluation apparatus for implementing the performance evaluation method described above. The solution provided by this apparatus is similar to the implementation described in the above method; therefore, the specific limitations of one or more performance evaluation apparatus embodiments provided below can be found in the limitations of the performance evaluation method described above, and will not be repeated here.

[0124] In one exemplary embodiment, such as Figure 8 As shown, a performance evaluation device is provided, comprising: an acquisition module 10, a determination module 11, and an evaluation module 12, wherein:

[0125] The acquisition module 10 is used to acquire the task type of the computing power task; the task type includes training task or inference task.

[0126] The determination module 11 is used to determine the computing power efficiency information corresponding to the task type based on the task type; the computing power efficiency information is collected when executing computing power tasks.

[0127] Evaluation module 12 is used to evaluate the performance parameters of computing power tasks based on computing power energy efficiency information.

[0128] In an exemplary embodiment, the task type described above is a training task, and the computing power efficiency information includes at least one of the following computing power efficiency parameters:

[0129] Throughput, unit energy consumption, carbon emission intensity, and hardware loss.

[0130] In an exemplary embodiment, the task type described above is an inference task, and the computing power efficiency information includes at least one of the following computing power efficiency parameters:

[0131] Monitor latency, energy consumption per query request, and start / stop losses.

[0132] In one exemplary embodiment, the computing power efficiency information mentioned above includes multiple computing power efficiency parameters;

[0133] The aforementioned evaluation module 12 includes:

[0134] The acquisition unit is specifically used to acquire the weights corresponding to multiple computing power efficiency parameters.

[0135] The processing unit is specifically used to perform weighted processing on multiple computing power efficiency parameters to obtain the performance parameters of the computing power task.

[0136] In one exemplary embodiment, the above-described apparatus further includes:

[0137] The strategy determination module is used to determine the task execution strategy for optimizing computing power energy efficiency information when the performance parameters of the computing power task do not meet the preset performance parameters.

[0138] The task execution module is used to re-execute the computing power task based on the task execution strategy, obtain new computing power energy efficiency information of the computing power task, and return to the execution evaluation module 12 until the usage performance parameters of the computing power task meet the preset performance parameters.

[0139] In an exemplary embodiment, the strategy determination module is further configured to determine a task execution strategy for optimizing computing power energy efficiency information based on the difference between the usage performance parameters of the computing power task and the preset performance parameters, and the usage performance parameters of the computing power task.

[0140] In an exemplary embodiment, the determining module 11 includes:

[0141] The first acquisition unit is specifically used to acquire computing power information corresponding to the task type based on acquisition plugins or probes.

[0142] The second acquisition unit is specifically used to acquire power information corresponding to the task type based on the power grid carbon tracing system.

[0143] The third acquisition unit is specifically used to acquire network information corresponding to the task type based on the network infrastructure system.

[0144] Each module in the aforementioned performance evaluation device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can invoke and execute the operations corresponding to each module.

[0145] In one exemplary embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 1 As shown, this computer device includes a processor, memory, input / output interfaces (I / O), and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is also connected to the system bus via the I / O interfaces. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides the environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The database stores computing power and energy efficiency data. The I / O interfaces are used for exchanging information between the processor and external devices. The communication interface is used for communicating with external terminals via a network connection. When the computer program is executed by the processor, it implements a performance evaluation method.

[0146] Those skilled in the art will understand that Figure 1 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0147] In one exemplary embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to perform the following steps:

[0148] The task type for acquiring computing power; the task type includes training tasks or inference tasks.

[0149] Based on the task type, determine the computing power efficiency information corresponding to the task type; the computing power efficiency information is collected when executing computing power tasks;

[0150] Based on computing power energy efficiency information, evaluate the performance parameters of computing power tasks.

[0151] In one embodiment, the processor, when executing a computer program, also performs the following steps:

[0152] The task type is a training task, and the computing power efficiency information includes at least one of the following computing power efficiency parameters:

[0153] Throughput, unit energy consumption, carbon emission intensity, and hardware loss.

[0154] In one embodiment, the processor, when executing a computer program, also performs the following steps:

[0155] The task type is inference task, and the computing power efficiency information includes at least one of the following computing power efficiency parameters:

[0156] Monitor latency, energy consumption per query request, and start / stop losses.

[0157] In one embodiment, the computing power efficiency information includes multiple computing power efficiency parameters; when the processor executes the computer program, it also performs the following steps:

[0158] Obtain the weights corresponding to multiple computing power efficiency parameters;

[0159] Based on the weights, multiple computing power energy efficiency parameters are weighted to obtain the performance parameters of the computing power task.

[0160] In one embodiment, the processor, when executing a computer program, also performs the following steps:

[0161] If the performance parameters of the computing power task do not meet the preset performance parameters, then determine the task execution strategy for optimizing computing power energy efficiency information;

[0162] The computing power task is re-executed based on the task execution strategy to obtain new computing power energy efficiency information of the computing power task, and then the execution is returned to evaluate the usage performance parameters of the computing power task based on the computing power energy efficiency information, until the usage performance parameters of the computing power task meet the preset performance parameters.

[0163] In one embodiment, the processor, when executing a computer program, also performs the following steps:

[0164] Based on the difference between the performance parameters used by the computing task and the preset performance parameters, and the performance parameters used by the computing task, a task execution strategy for optimizing computing power energy efficiency information is determined.

[0165] In one embodiment, the processor, when executing a computer program, also performs the following steps:

[0166] Based on the acquisition plugin or probe, collect computing power information corresponding to the task type;

[0167] Based on the power grid carbon tracing system, collect power information corresponding to the task type;

[0168] Based on the network infrastructure system, network information corresponding to the task type is collected.

[0169] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, the computer program performing the following steps when executed by a processor:

[0170] The task type for acquiring computing power; the task type includes training tasks or inference tasks.

[0171] Based on the task type, determine the computing power efficiency information corresponding to the task type; the computing power efficiency information is collected when executing computing power tasks;

[0172] Based on computing power energy efficiency information, evaluate the performance parameters of computing power tasks.

[0173] In one embodiment, when the computer program is executed by a processor, it also performs the following steps:

[0174] The task type is a training task, and the computing power efficiency information includes at least one of the following computing power efficiency parameters:

[0175] Throughput, unit energy consumption, carbon emission intensity, and hardware loss.

[0176] In one embodiment, when the computer program is executed by a processor, it also performs the following steps:

[0177] The task type is inference task, and the computing power efficiency information includes at least one of the following computing power efficiency parameters:

[0178] Monitor latency, energy consumption per query request, and start / stop losses.

[0179] In one embodiment, when the computer program is executed by a processor, it also performs the following steps:

[0180] Obtain the weights corresponding to multiple computing power efficiency parameters;

[0181] Based on the weights, multiple computing power energy efficiency parameters are weighted to obtain the performance parameters of the computing power task.

[0182] In one embodiment, when the computer program is executed by a processor, it also performs the following steps:

[0183] If the performance parameters of the computing power task do not meet the preset performance parameters, then determine the task execution strategy for optimizing computing power energy efficiency information;

[0184] The computing power task is re-executed based on the task execution strategy to obtain new computing power energy efficiency information of the computing power task, and then the execution is returned to evaluate the usage performance parameters of the computing power task based on the computing power energy efficiency information, until the usage performance parameters of the computing power task meet the preset performance parameters.

[0185] In one embodiment, when the computer program is executed by a processor, it also performs the following steps:

[0186] Based on the difference between the performance parameters used by the computing task and the preset performance parameters, and the performance parameters used by the computing task, a task execution strategy for optimizing computing power energy efficiency information is determined.

[0187] In one embodiment, when the computer program is executed by a processor, it also performs the following steps:

[0188] Based on the acquisition plugin or probe, collect computing power information corresponding to the task type;

[0189] Based on the power grid carbon tracing system, collect power information corresponding to the task type;

[0190] Based on the network infrastructure system, network information corresponding to the task type is collected.

[0191] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, performs the following steps:

[0192] The task type for acquiring computing power; the task type includes training tasks or inference tasks.

[0193] Based on the task type, determine the computing power efficiency information corresponding to the task type; the computing power efficiency information is collected when executing computing power tasks;

[0194] Based on computing power energy efficiency information, evaluate the performance parameters of computing power tasks.

[0195] In one embodiment, when the computer program is executed by a processor, it also performs the following steps:

[0196] The task type is a training task, and the computing power efficiency information includes at least one of the following computing power efficiency parameters:

[0197] Throughput, unit energy consumption, carbon emission intensity, and hardware loss.

[0198] In one embodiment, when the computer program is executed by a processor, it also performs the following steps:

[0199] The task type is inference task, and the computing power efficiency information includes at least one of the following computing power efficiency parameters:

[0200] Monitor latency, energy consumption per query request, and start / stop losses.

[0201] In one embodiment, when the computer program is executed by a processor, it also performs the following steps:

[0202] Obtain the weights corresponding to multiple computing power efficiency parameters;

[0203] Based on the weights, multiple computing power energy efficiency parameters are weighted to obtain the performance parameters of the computing power task.

[0204] In one embodiment, when the computer program is executed by a processor, it also performs the following steps:

[0205] If the performance parameters of the computing power task do not meet the preset performance parameters, then determine the task execution strategy for optimizing computing power energy efficiency information;

[0206] The computing power task is re-executed based on the task execution strategy to obtain new computing power energy efficiency information of the computing power task, and then the execution is returned to evaluate the usage performance parameters of the computing power task based on the computing power energy efficiency information, until the usage performance parameters of the computing power task meet the preset performance parameters.

[0207] In one embodiment, when the computer program is executed by a processor, it also performs the following steps:

[0208] Based on the difference between the performance parameters used by the computing task and the preset performance parameters, and the performance parameters used by the computing task, a task execution strategy for optimizing computing power energy efficiency information is determined.

[0209] In one embodiment, when the computer program is executed by a processor, it also performs the following steps:

[0210] Based on the acquisition plugin or probe, collect computing power information corresponding to the task type;

[0211] Based on the power grid carbon tracing system, collect power information corresponding to the task type;

[0212] Based on the network infrastructure system, network information corresponding to the task type is collected.

[0213] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, artificial intelligence (AI) processors, etc., and are not limited to these.

[0214] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0215] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A method of evaluating the performance of use, characterized by, The method comprises: obtaining a task type of a computing power task; the task type comprises a training task or an inference task; determining computing power energy efficiency information corresponding to the task type according to the task type; the computing power energy efficiency information is collected under the condition of executing the computing power task; evaluating a use performance parameter of the computing power task according to the computing power energy efficiency information.

2. The method of claim 1, wherein the task type is the training task, and the computing power energy efficiency information comprises at least one of the following computing power energy efficiency parameters: throughput, unit energy consumption, carbon emission intensity, and hardware loss.

3. The method of claim 1, wherein the task type is the inference task, and the computing power energy efficiency information comprises at least one of the following computing power energy efficiency parameters: monitoring time delay, single query request energy consumption, and start-stop loss.

4. The method of claim 2 or 3, wherein the computing power energy efficiency information comprises a plurality of computing power energy efficiency parameters; and the evaluating the use performance parameter of the computing power task according to the computing power energy efficiency information comprises: obtaining weights corresponding to the plurality of computing power energy efficiency parameters; and performing weighted processing on the plurality of computing power energy efficiency parameters based on the weights to obtain the use performance parameter of the computing power task. The method further comprises: if the use performance parameter of the computing power task does not meet a preset performance parameter, determining a task execution strategy for optimizing the computing power energy efficiency information; 5. The method of claim 1, wherein, re-executing the computing power task based on the task execution strategy, obtaining new computing power energy efficiency information of the computing power task, and returning to the step of evaluating the use performance parameter of the computing power task according to the computing power energy efficiency information until the use performance parameter of the computing power task meets the preset performance parameter. The determining the task execution strategy for optimizing the computing power energy efficiency information comprises: determining the task execution strategy for optimizing the computing power energy efficiency information according to a difference between the use performance parameter of the computing power task and a preset performance parameter and the use performance parameter of the computing power task.

6. The method of claim 5, wherein, The determining the computing power energy efficiency information corresponding to the task type comprises: collecting computing power information corresponding to the task type based on a collection plug-in or a probe; 7. The method of claim 1, wherein, collecting power information corresponding to the task type based on a power grid carbon tracking system; and collecting network information corresponding to the task type based on a network infrastructure capability system. The device comprises: an obtaining module configured to obtain a task type of a computing power task; the task type comprises a training task or an inference task; 8. An evaluation device for the use performance, characterized in that a determining module configured to determine computing power energy efficiency information corresponding to the task type according to the task type; the computing power energy efficiency information is collected under the condition of executing the computing power task; an evaluating module configured to evaluate a use performance parameter of the computing power task according to the computing power energy efficiency information. The processor executes the computer program to implement the steps of the method of any one of claims 1 to 7. The computer program is executed by the processor to implement the steps of the method of any one of claims 1 to 7. 9.A computer device, comprising a memory and a processor, wherein the memory stores a computer program, and the computer device is configured to perform the method according to any one of claims 1-8 when the computer program is executed by the processor. ​ 10. A computer-readable storage medium having stored thereon a computer program, characterized in that, ​

Images