Charging method and device based on calculation management platform, electronic equipment and medium

Abstract

The invention discloses a charging method and device based on a computing management platform, electronic equipment and a medium, and the charging method is applied to a high-performance computing management platform and comprises the steps: receiving a job task request, and monitoring the operation condition of a job task; obtaining operation information of the job task according to the operation condition of the job task; determining resource information required for completing the job task according to the running information; according to the resource information and a charging cycle strategy defined by a user, judging whether to execute fee deduction; and generating a bill according to the billing cycle strategy and the bill generation strategy. According to the method, the operation task scheduling system and the charging system can be decoupled to adapt to any operation scheduling system, meanwhile, the operation task can be flexibly monitored and analyzed in real time, periodic charging is performed according to the charging strategy set by the user, and the method is convenient, flexible, reliable and practical, realizes unified charging and does not cause expense omission.

Description

[0001] The present invention relates to the field of computer technology, and in particular to a charging method, device, electronic device and medium based on a computing management platform. Background Art

[0002] The high-performance computing management platform can integrate hardware resources and business resources for high-performance computing business development and operation and maintenance scenarios, realize unified allocation and scheduling of computing resources, comprehensive cluster monitoring, diversified job submission and management, and precise and flexible billing management. It is a stable, efficient and easy-to-use full-stack platform.

[0003] At present, high-performance computing management platforms provide high-performance data parallel processing for applications using big data, which will be distributed in thousands or even tens of thousands of cluster nodes. In order to better manage the use of platform clusters and reasonably allocate resources, a job scheduling system will be introduced into the platform. Different scheduling systems have different information statistics and scheduling strategies for job tasks. When designing a billing system, designers often design billing strategies based on the characteristics of the job scheduling system used and the characteristics of the platform, but this will lead to a high degree of coupling between the billing system and the scheduling system used by the platform. As a result, the scheduling system will change and the billing system will also become invalid. At this time, the designer has to redesign the billing system according to the new scheduling system.

[0004] In addition, the billing strategy design of the billing system is often relatively simple, and users cannot flexibly charge according to their own billing strategy and billing cycle. The resource rate setting is also relatively simple, and it is impossible to flexibly charge for services according to changes in resource rates. It is impossible to unify the billing of jobs submitted by the platform and jobs submitted by the Linux system layer, resulting in omissions in expenses. Summary of the invention

[0005] The technical problem to be solved by the present invention is to overcome the defects of the prior art and provide a billing method, device, electronic device and medium based on a computing management platform, which can decouple the job task scheduling system and the billing system to adapt to the job scheduling system, and at the same time can flexibly monitor and analyze the job tasks in real time, and perform periodic billing according to the billing strategy set by the user, which is convenient and flexible, achieves unified billing, does not lead to cost omissions, and is reliable to use.

[0006] In order to solve the above technical problems, the present invention provides the following technical solutions:

[0007] The present invention provides a charging method based on a computing management platform, which is applied to a high-performance computing management platform and includes the following steps:

[0008] Step A: Receive job task requests and monitor the resource information of the job tasks. The resource information includes: resource type CPU or GPU, resource partition, resource QOS, number of nodes, number of CPUs, memory size, number of GPUs, and storage size of a single node.

[0009] Step B: According to the operation status of the job task, the BP neural network model is used to predict the operation information of the job task according to the operation parameters of the job task. The operation information includes: operation start time, end time, CPU time and wall time, as well as resource utilization rate during operation;

[0010] Step C: Determine whether the user usage rate meets the set threshold based on the predicted operation information. If the operation usage rate is lower than the threshold, send a warning to the user and provide task parameter suggestions. After the user confirms, determine the resource information required to complete the job task;

[0011] Step D: Determine whether to perform fee deduction based on resource information and user-defined billing cycle policy;

[0012] Step E: Generate an original bill according to the billing cycle policy and the bill generation policy;

[0013] Step F: Generate a final bill based on the preferential policy for the user set by the administrator.

[0014] As a preferred technical solution of the present invention, the step B further comprises: obtaining a billing period from the management node, where the billing period is a billing time interval preset by the user;

[0015] According to the running status of the job task, the attributes of the job task are analyzed in each billing cycle, and the parameter information of the job task is recorded.

[0016] As a preferred technical solution of the present invention, the step C further comprises:

[0017] Analyze the parameter information of the job task to determine the resource information required to complete the job task.

[0018] As a preferred technical solution of the present invention, the step C further comprises:

[0019] Using the BP neural network model to determine whether the resource utilization rate of the job task exceeds a set threshold according to the job task operation parameters, and obtaining a first determination result;

[0020] The first judgment result includes that the operation task utilization rate is higher than the threshold value and that the operation task utilization rate is lower than the threshold value.

[0021] As a preferred technical solution of the present invention, if the first judgment result is that the job task is higher than the threshold, the task is submitted to the resource scheduling node to perform the task operation. After the task is completed, the operation information of the job task is obtained according to the operation status of the job task, and the actual operation time is the second billing operation time;

[0022] Obtaining a first billing job running time and a second billing job running time, wherein the first billing job running time is the job running time predicted by the BP neural network model of the job task, and the second billing job running time is the actual billing job running time of the job task;

[0023] Determine whether the difference between the running time of the first billing job and the running time of the second billing job is greater than a threshold;

[0024] If the difference between the running time of the first billing job and the running time of the second billing job is greater than the threshold, the running time of the second billing job is recorded in the database, accounting details are generated according to the running time of the second billing job, and the accounting sheet of the job task is updated;

[0025] Update the BP neural network model according to the recorded billing job running time and correct the model prediction result weight;

[0026] If the difference between the running time of the first billing job and the running time of the second billing job is less than the threshold, the running time of the second billing job is recorded in the database, accounting details are generated according to the running time of the second billing job, and the accounting sheet of the job task is updated;

[0027] Update the job task's accounting book.

[0028] As a preferred technical solution of the present invention, the step D also includes: in response to the balance of the account from which the deduction is executed being less than the balance reminder threshold, sending a balance reminder message to the account, the account including an organization account or a user account.

[0029] As a preferred technical solution of the present invention, the step E also includes:

[0030] Get the preset payment cycle, which is the time interval for payment statistics set by the user;

[0031] According to the preset payment cycle, the account will collect the bill details of the preset payment cycle, and the account will be paid out regularly according to the bill details;

[0032] Accounts include organizational accounts or user accounts.

[0033] The present invention also provides a charging device based on a computing management platform, comprising:

[0034] The communication module is used to receive job task requests and monitor the running status of job tasks;

[0035] The management module is used to obtain the operation information of the job task according to the operation status of the job task;

[0036] The resource acquisition module is used to determine the resource information required to complete the job task based on the operation information;

[0037] The deduction module is used to determine whether to perform deduction based on resource information and user-defined billing cycle policies;

[0038] The billing module is used to generate bills according to the billing cycle strategy and bill generation strategy.

[0039] The present invention also provides an electronic device based on a computing management platform, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the above-mentioned billing method based on the computing management platform is executed.

[0040] The present invention also provides a computer-readable storage medium storing computer-executable instructions. When the computer-readable storage medium is run on a computer, the computer executes the above-mentioned billing method based on the computing management platform.

[0041] Compared with the prior art, the present invention has the following beneficial effects:

[0042] The present invention can flexibly monitor, analyze and collect statistical information on job tasks submitted by the front end or Linux layer of the high-performance computing management platform in real time, and can decouple the job task scheduling system and the billing system to adapt to any job scheduling system; at the same time, periodic billing can be performed according to the billing strategy set by the user, which is convenient and flexible and can achieve unified billing; reasonable charges can also be made according to the usage of resources and the setting of rates for different resources, and regular billing can be made according to the billing cycle set by the user, which is highly flexible, reliable and practical, and provides a good user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] The accompanying drawings are used to provide a further understanding of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation of the present invention. In the accompanying drawings:

[0044] Figure 1 is a flow chart of the charging method of the present invention; DETAILED DESCRIPTION

[0045] The preferred embodiments of the present invention are described below in conjunction with the accompanying drawings. It should be understood that the preferred embodiments described herein are only used to illustrate and explain the present invention, and are not used to limit the present invention.

[0046] Example 1

[0047] like Figure 1 As shown, the present invention provides a billing method based on a computing management platform, which is applied to a high-performance computing management platform and includes the following steps:

[0048] Step A: Receive job task requests and monitor the resource information of the job tasks. The resource information includes: resource type CPU or GPU, resource partition, resource QOS, number of nodes, number of CPUs, memory size, number of GPUs, and storage size of a single node.

[0049] Step B: According to the operation status of the job task, the BP neural network model is used to predict the operation information of the job task according to the operation parameters of the job task. The operation information includes: operation start time, end time, CPU time and wall time, as well as resource utilization rate during operation;

[0050] Step C: Determine whether the user usage rate meets the set threshold based on the predicted operation information. If the operation usage rate is lower than the threshold, send a warning to the user and provide task parameter suggestions. After the user confirms, determine the resource information required to complete the job task;

[0051] Step D: Determine whether to perform fee deduction based on resource information and user-defined billing cycle policy;

[0052] Step E: Generate an original bill according to the billing cycle policy and the bill generation policy;

[0053] Step F: Generate a final bill based on the preferential policy for the user set by the administrator.

[0054] Further, step B also includes: obtaining a billing cycle from the management node, the billing cycle being a billing time interval preset by the user;

[0055] According to the running status of the job task, the attributes of the job task are analyzed in each billing cycle, and the parameter information of the job task is recorded.

[0056] Further, step C also includes:

[0057] Analyze the parameter information of the job task to determine the resource information required to complete the job task.

[0058] Further, step C also includes:

[0059] Using the BP neural network model to determine whether the resource utilization rate of the job task exceeds a set threshold according to the job task operation parameters, and obtaining a first determination result;

[0060] The first judgment result includes that the operation task utilization rate is higher than the threshold value and that the operation task utilization rate is lower than the threshold value.

[0061] Further, if the first judgment result is that the job task is higher than the threshold, the task is submitted to the resource scheduling node to run the task job. After the task is completed, the running information of the job task is obtained according to the running status of the job task. The actual running time is the running time of the second billing job.

[0062] Obtaining a first billing job running time and a second billing job running time, wherein the first billing job running time is the job running time predicted by the BP neural network model of the job task, and the second billing job running time is the actual billing job running time of the job task;

[0063] Determine whether the difference between the running time of the first billing job and the running time of the second billing job is greater than a threshold;

[0064] If the difference between the running time of the first billing job and the running time of the second billing job is greater than the threshold, the running time of the second billing job is recorded in the database, accounting details are generated according to the running time of the second billing job, and the accounting sheet of the job task is updated;

[0065] Update the BP neural network model according to the recorded billing job running time and correct the model prediction result weight;

[0066] If the difference between the running time of the first billing job and the running time of the second billing job is less than the threshold, the running time of the second billing job is recorded in the database, accounting details are generated according to the running time of the second billing job, and the accounting sheet of the job task is updated;

[0067] Update the job task's accounting book.

[0068] Further, step D also includes: in response to the balance of the account from which the deduction is executed being less than the balance reminder threshold, sending a balance reminder message to the account, where the account includes an organization account or a user account.

[0069] Further, step E also includes:

[0070] Get the preset payment cycle, which is the time interval for payment statistics set by the user;

[0071] According to the preset payment cycle, the account will collect the bill details of the preset payment cycle, and the account will be paid out regularly according to the bill details;

[0072] Accounts include organizational accounts or user accounts.

[0073] Specifically, the implementation environment of the embodiment includes:

[0074] Computing management platform: A high-performance computing cluster consisting of multiple server nodes that supports the Linux operating system and mainstream resource scheduling frameworks (such as Slurm, OpenPBS, Kubernetes) and mainstream high-performance computing frameworks (such as MPI, OpenMP, Spark, MapReduce, OpenCL) and heterogeneous computing frameworks (such as CUDA, ROCm).

[0075] Client: Supports web interface, command line interface and API interface, allowing users to submit job task requests, monitor job status and view billing information.

[0076] Database: used to store job task information, user account information, billing data, etc.

[0077] Implementation steps:

[0078] Step 1: Receive a job request

[0079] Users submit job task requests through the web interface, command line interface, or API, including the job type, required resource type and quantity, etc.

[0080] Step 2: Verify whether the job task application resources comply with the rules

[0081] The communication module receives the request, verifies the user's authority, and stores the request information in the database. If the user submits an incorrect partition or the resource application submitted exceeds the resource limit of the partition where the user is located, the task submission is aborted and a warning message is returned, which contains the specific error information.

[0082] If the task submitted by the user is legal, proceed to the next step.

[0083] Step 3: Predict user resource utilization efficiency

[0084] The resource acquisition module predicts the amount of resources required to complete the job based on the historical operation data of the job task through the BP neural network model.

[0085] Resource information includes the number of CPU cores, memory size, disk I / O, etc. The running time of the first billing job is estimated based on the resource amount. If the running time is lower than the threshold, a warning message is returned to the user. The warning message includes specific predicted running time and low utilization information, and gives recommended resource configuration information for this type of task.

[0086] Step 4: Determine the resource information required to complete the task

[0087] If the user is sure to continue running, the task is recorded as a status where the user has confirmed it, and the task is submitted to the task scheduling system to run the job task.

[0088] If the user modifies the job application resource information, return to step 2 to verify whether the job task application resources comply with the rules.

[0089] Step 5: Monitor the running status of the job

[0090] The system uses the monitoring module to track the running status of each job task in real time, including process status, resource usage, etc.

[0091] The monitoring module collects the operation data of the job every 1 minute and stores it in the database.

[0092] Step 6: Get job task running information

[0093] At the end of each billing cycle (for example, 24:00 every day), the system reads database records through the resource scheduling manager interface to collect comprehensive operation information for each job task.

[0094] The management module analyzes the collected data, determines the resource information required to complete the job task, and stores it in the database.

[0095] Step 7: Determine whether to execute the deduction

[0096] The deduction module calculates the deductible fees for this operation based on resource information and historical billing data.

[0097] The system determines whether to deduct fees based on the billing policy of the user account (such as pay-as-you-go billing, monthly billing, etc.).

[0098] Step 8: Generate bill

[0099] The bill generation module generates detailed bill information according to the billing cycle strategy (such as daily, weekly, and monthly billing) and the bill generation strategy.

[0100] The billing information includes the job task name, resource usage, deduction amount, billing date, etc.

[0101] Step 9: Balance reminder

[0102] The system monitors the user's account balance. When the balance is lower than the set threshold (such as 500 yuan), it sends a balance reminder to the user via email or text message.

[0103] The reminder information includes the current balance, reminder threshold and account recharge method.

[0104] Step 10: Scheduled payment

[0105] The payment module counts and pays out accounts according to the preset payment cycle (such as the 1st of each month).

[0106] Determine whether there are coupons in the user account. If there are coupons, deduct the coupon fee first and generate the final bill.

[0107] The system generates a bill and sends it to the user's email or pushes bill information through the WeChat interface. The user pays the bill within the specified time.

[0108] Step 11: Dynamic resource adjustment

[0109] Based on the job requirements and resource usage, the system trains the BP neural network model based on historical data to predict the resource utilization rate and job task running time after resource allocation. The system determines whether the resource utilization rate meets expectations through the running time. If it does not meet expectations, it provides recommended resource configuration content to achieve dynamic adjustment of resource allocation.

[0110] If the resource requirements of a job task increase, the system can automatically allocate more resources to it.

[0111] Step 12: Real-time billing updates

[0112] The deduction module updates the user's billing information in real time and calculates the fees based on the current resource usage.

[0113] The system updates the user's billing data at regular intervals (such as every hour) to ensure the accuracy of billing.

[0114] Step 13: User Interface Interaction

[0115] Users can view their account information, job status, and billing details through the web interface, command line interface, or API interface.

[0116] Users can adjust their billing strategies and resource requirements in real time, and the system will perform corresponding billing and resource allocation based on these adjustments.

[0117] Step 14: Exception handling and alarm

[0118] The system monitors abnormal conditions of job tasks, such as long running time, abnormal resource usage, etc.

[0119] When an abnormal situation occurs, the system will automatically alarm and notify the administrator for timely processing.

[0120] Implementation Effect

[0121] The billing method in this embodiment can realize real-time monitoring and analysis of job tasks, perform periodic billing according to the billing policy set by the user, and ensure the uniformity and accuracy of billing.

[0122] By decoupling the job task scheduling system and the billing system, the flexibility and maintainability of the system are improved.

[0123] Dynamic resource allocation and real-time billing update capabilities improve resource utilization and user satisfaction.

[0124] Balance reminder and scheduled payment functions improve user experience and system management efficiency.

[0125] Application Scenario

[0126] It is suitable for scientific research institutions, enterprise data centers and cloud computing service providers, especially those scenarios that require refined management and dynamic adjustment of high-performance computing resources.

[0127] In a cloud computing environment, this method can effectively manage resource usage, improve resource utilization efficiency, and provide users with flexible billing services.

[0128] The embodiment of the present invention further provides a charging device based on a computing management platform, comprising:

[0129] The communication module is used to receive job task requests and monitor the running status of job tasks;

[0130] The management module is used to obtain the operation information of the job task according to the operation status of the job task;

[0131] The resource acquisition module is used to determine the resource information required to complete the job task based on the operation information;

[0132] The deduction module is used to determine whether to perform deduction based on resource information and user-defined billing cycle policies;

[0133] The billing module is used to generate bills according to the billing cycle strategy and bill generation strategy

[0134] An embodiment of the present invention also provides an electronic device based on a computing management platform, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the billing method based on the computing management platform in the above embodiment is executed.

[0135] An embodiment of the present invention further provides a computer-readable storage medium storing computer-executable instructions. When the computer-readable storage medium is run on a computer, the computer executes the billing method based on the computing management platform in the above embodiment.

[0136] Finally, it should be noted that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art can still modify the technical solutions described in the aforementioned embodiments or replace some of the technical features therein by equivalents. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A billing method based on a computing management platform, comprising the following steps: Step A: Receive job task requests and monitor the resource information of the job tasks. The resource information includes: resource type CPU or GPU, resource partition, resource QOS, number of nodes, number of CPUs, memory size, number of GPUs, and storage size of a single node. Step B: According to the operation status of the job task, the BP neural network model is used to predict the operation information of the job task according to the operation parameters of the job task. The operation information includes: operation start time, end time, CPU time and wall time, as well as resource utilization rate during operation; Step C: Determine whether the user usage rate meets the set threshold based on the predicted operation information. If the operation usage rate is lower than the threshold, send a warning to the user and provide task parameter suggestions. After the user confirms, determine the resource information required to complete the job task; Step D: Determine whether to perform fee deduction based on resource information and user-defined billing cycle policy; Step E: Generate an original bill according to the billing cycle policy and the bill generation policy; Step F: Generate a final bill based on the preferential policy for the user set by the administrator.

2. A billing method based on a computing management platform according to claim 1, characterized in that: The step B also includes: obtaining a billing cycle from the management node, where the billing cycle is a billing time interval preset by the user; According to the running status of the job task, the attributes of the job task are analyzed in each billing cycle, and the parameter information of the job task is recorded.

3. A billing method based on a computing management platform according to claim 1, characterized in that: The step C also includes: Analyze the parameter information of the job task to determine the resource information required to complete the job task.

4. A billing method based on a computing management platform according to claim 3, characterized in that: The step C also includes: Using the BP neural network model to determine whether the resource utilization rate of the job task exceeds a set threshold according to the job task operation parameters, and obtaining a first determination result; The first judgment result includes that the operation task utilization rate is higher than the threshold value and that the operation task utilization rate is lower than the threshold value.

5. A billing method based on a computing management platform according to claim 4, characterized in that: If the first judgment result is that the job task is higher than the threshold, the task is submitted to the resource scheduling node to run the task. After the task is completed, the running information of the job task is obtained according to the running status of the job task. The actual running time is the running time of the second billing job. Obtaining a first billing job running time and a second billing job running time, wherein the first billing job running time is the job running time predicted by the BP neural network model of the job task, and the second billing job running time is the actual billing job running time of the job task; Determine whether the difference between the running time of the first billing job and the running time of the second billing job is greater than a threshold; If the difference between the running time of the first billing job and the running time of the second billing job is greater than the threshold, the running time of the second billing job is recorded in the database, accounting details are generated according to the running time of the second billing job, and the accounting sheet of the job task is updated; Update the BP neural network model according to the recorded billing job running time and correct the model prediction result weight; If the difference between the running time of the first billing job and the running time of the second billing job is less than the threshold, the running time of the second billing job is recorded in the database, accounting details are generated according to the running time of the second billing job, and the accounting sheet of the job task is updated; Update the job task's accounting book.

6. A billing method based on a computing management platform according to claim 1, characterized in that: The step D also includes: in response to the balance of the account from which the deduction is executed being less than a balance reminder threshold, sending a balance reminder message to the account, where the account includes an organization account or a user account.

7. A billing method based on a computing management platform according to claim 1, characterized in that: The step E also includes: Get the preset payment cycle, which is the time interval for payment statistics set by the user; According to the preset payment cycle, the account will collect the bill details of the preset payment cycle, and the account will be paid out regularly according to the bill details; Accounts include organizational accounts or user accounts.

8. A billing device based on a computing management platform, characterized in that: include: The communication module is used to receive job task requests and monitor the running status of job tasks; The management module is used to obtain the operation information of the job task according to the operation status of the job task; The resource acquisition module is used to determine the resource information required to complete the job task based on the operation information; The deduction module is used to determine whether to perform deduction based on resource information and user-defined billing cycle policies; The billing module is used to generate bills according to the billing cycle strategy and bill generation strategy.

9. An electronic device based on a computing management platform, characterized in that: It includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it executes the billing method based on the computing management platform as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: Computer executable instructions are stored, and when they are run on a computer, the computer executes the billing method based on the computing management platform as described in any one of claims 1 to 7.

Images