Data processing computing power scheduling method based on intelligent agent

Abstract

The invention relates to the technical field of computing power scheduling, and discloses a data processing computing power scheduling method based on an intelligent agent, and the method comprises the steps: obtaining basic data and real-time operation data of each computing power device in an operation cluster, and obtaining a real-time feature element of each computing power device according to the basic data and the real-time operation data; identifying each task based on an intelligent agent, and obtaining task feature elements and priorities; determining a processing sequence according to the queue time and priority of the current task and the allocated tasks; and according to the task feature element of each task and the real-time feature element of each computing power device, carrying out coincidence matching, and according to a coincidence matching result, allocating a corresponding computing power device to each task. According to the method, the real-time characteristic elements of each computing power device are acquired according to the basic data and the real-time operation data of each computing power device, so that the computing power task is accurately matched with the computing power device, and performance optimization and cost optimization of computing power scheduling are realized.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computing power scheduling, in particular to a data processing computing power scheduling method based on an intelligent agent. BACKGROUND

[0002] With the wide use of large-scale computing platforms (such as cloud computing, big data, and AI training platforms), the demand for computing power is also increasing. In order to ensure stable supply of computing power, many task requirements generally use computing power clusters to achieve computing power support. In this process, balanced scheduling of computing power can intelligently allocate computing tasks to the most suitable nodes, thereby avoiding the "uneven busy and idle" state of some nodes being overloaded while other nodes being idle, and thus optimizing the utilization of entire cluster resources.

[0003] In the prior art, the computing power scheduling scheme is determined according to the load state of the computing power device. When the load of the device is low, the computing power task is preferentially allocated to the computing power device. When the load of the device is high, the allocation priority of the computing power device is low. In this way, the load conditions of all devices are relatively consistent, thereby realizing the relative balance of all computing power devices.

[0004] Due to the differences between different task requirements and the performance gaps between different computing power devices, the existing computing power scheduling method cannot optimize the performance of computing power scheduling. In addition, due to the use cost gap between different computing power devices, the economic optimization of computing power scheduling cannot be guaranteed. Therefore, how to further improve the performance optimization and cost optimization of computing power scheduling is the fundamental problem to be solved by the present application. SUMMARY

[0005] In order to further improve the performance optimization and cost optimization of computing power scheduling, the present application provides a data processing computing power scheduling method based on an intelligent agent.

[0006] A data processing computing power scheduling method based on an intelligent agent, the method comprising: obtaining basic data and real-time running data of each computing power device in the computing cluster, and obtaining real-time characteristic elements of each computing power device according to the basic data and the real-time running data; identifying each task based on the intelligent agent, and obtaining task characteristic elements and priorities; determining a processing order according to the queue time and the priority of the current task and the allocated tasks, and performing coincidence matching according to the task characteristic elements of each task and the real-time characteristic elements of each computing power device, and allocating a corresponding computing power device to each task according to the coincidence matching result.

[0007] Further, the real-time characteristic elements include a transmission efficiency coefficient, an operation cost coefficient, an operation processing coefficient, and an image processing coefficient. The task characteristic elements include a transmission demand coefficient, a predicted operation duration, an operation processing demand coefficient, and an image processing demand coefficient. The matching process includes: The matching value of the task characteristic elements and the i-th computing power device is calculated by the formula The matching value of the task characteristic elements and the i-th computing power device is calculated by the formula ; The first m computing power devices are selected in descending order of the matching value The matching balance value of the task characteristic elements and the i-th computing power device is calculated by the formula The matching balance value of the task characteristic elements and the i-th computing power device is calculated by the formula The task is allocated to the computing power device with the largest matching balance value The task is allocated to the computing power device with the largest matching balance value Wherein, j=1, 2, 3. The transmission weight coefficient, the operation processing weight coefficient, and the image processing weight coefficient are respectively The transmission efficiency coefficient, the operation processing coefficient, and the image processing coefficient of the i-th computing power device are respectively The transmission demand coefficient, the operation processing demand coefficient, and the image processing demand coefficient are respectively The first definition function is When , otherwise , , The mean value of the three groups The reference value of the transmission efficiency coefficient, the operation processing coefficient, and the image processing coefficient is respectively The reference value of the transmission efficiency coefficient, the operation processing coefficient, and the image processing coefficient is respectively The unit time cost of the i-th computing power device is The predicted operation duration of the current task is The cost unit quantity is The cost adjustment coefficient is

[0008] Further, the transmission efficiency coefficient calculation process of each computing power device includes: The transmission efficiency coefficient of the i-th computing power device is calculated by the formula The transmission efficiency coefficient of the i-th computing power device is calculated by the formula ; Wherein, The average delay of the i-th computing power device is The delay impact quantity is controlled by the function The allowed throughput of the i-th computing power device is The running task quantity of the i-th computing power device is , The throughput of the k-th task of the i-th computing power device is​ for The contention and synchronization waiting of running tasks affect the throughput. is the throughput unit value, is the weight adjustment coefficient, x=1, 2, 3, 4; are the average storage write speed, average storage write speed, average memory write speed, and average memory write speed of the i-th computing power device, respectively. They are storage write speed benchmark value, storage write speed benchmark value, memory write speed benchmark value, and memory write speed benchmark value. For the second defined function, when hour, ,otherwise, , They are the impact weight coefficients of storage write, storage write, memory write, and memory write respectively.

[0009] Furthermore, the calculation process of the operation processing coefficient includes: By formula: Calculate the processing coefficient of the i-th computing power device ; in, is the computing power value of the i-th computing device, is the proportion of computing power occupied by the i-th computing device, for The computing power usage ratio corresponds to the computing power attenuation value.

[0010] Furthermore, the calculation process of the image processing coefficient includes: By formula: Calculate the image processing coefficient of the i-th computing power device ; in, is the graphics processing capability value of the i-th computing device, is the proportion of graphics processing computing power occupied by the i-th computing power device, for The graphics processing computing power usage ratio corresponds to the graphics processing capability attenuation value.

[0011] Furthermore, the process of determining the processing order includes: The current task is compared with the assigned task in priority, when the priority of the current task is not higher than the priority of the last task in the assigned task, the processing order of the current task is determined in time sequence, when the priority of the current task is higher than the priority of the last task in the assigned task, the processing order of the current task is determined according to the difference between the priorities and the queue time of the last task in the assigned task.

[0012] Further, the process of determining the processing order of the current task according to the difference between the priorities and the queue time of the last task in the assigned task comprises: The adjustment coefficient E is calculated by the formula: When E is greater than or equal to 0, the processing order of the current task is determined in time sequence, and when E is less than 0, the current task is adjusted before the last task in the assigned task. Wherein, is the queue time of the last task in the assigned task, is the correction coefficient, is the priority of the last task in the assigned task, and O is the priority of the current task, is the priority corresponding to the queue limit time.

[0013] Further, the method further comprises: Fault monitoring is performed on each computing power device, and when the monitoring result is abnormal, the computing power device is not scheduled.

[0014] In summary, the present application includes at least one of the following beneficial technical effects: The present application can judge the demand characteristics of each task, obtain task characteristic elements and priority, and adjust the priority of task processing more reasonably according to the priority and the queue time of the current task and the assigned task, by means of the intelligent agent identifying each task. In addition, the real-time characteristic elements of each computing power device are obtained through the basic data and real-time running data of each computing power device, so as to realize accurate matching of computing power tasks and computing power devices, and further realize performance optimization and cost optimization of computing power scheduling. BRIEF DESCRIPTION OF DRAWINGS

[0015] Figure 1 is a step flow chart of the data processing computing power scheduling method based on an intelligent agent. DETAILED DESCRIPTION

[0016] The embodiments of the present application will be described in detail below, and examples of the embodiments are shown in the accompanying drawings.

[0017] In the description of the present specification, the description of the terms "certain embodiments", "one embodiment", "some embodiments", "illustrative embodiments", "example", "specific example", or "some examples" means that the specific features, structures, materials or characteristics described in connection with the described embodiment or example are included in at least one embodiment or example of the present application. In the present specification, the illustrative description of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner.

[0018] The embodiment of the present application discloses a data processing computing power scheduling method based on an agent, please refer to Figure 1 The method comprises the following steps: acquiring the basic data and real-time running data of each computing power device in the operation cluster, and acquiring the real-time characteristic elements of each computing power device according to the basic data and real-time running data; identifying each task based on the agent, and acquiring the task characteristic elements and priority; determining the processing order according to the queue time and priority of the current task and the allocated task; and performing coincidence matching according to the task characteristic elements of each task and the real-time characteristic elements of each computing power device, and allocating the corresponding computing power device to each task according to the coincidence matching result. In this process, the agent identifies each task, which can judge the demand characteristics of each task, acquire the task characteristic elements and priority, and reasonably adjust the priority of task processing according to the priority and the queue time of the current task and the allocated task. In addition, the embodiment acquires the real-time characteristic elements of each computing power device according to the basic data and real-time running data of each computing power device, which can realize accurate matching of computing power tasks and computing power devices, and improve the efficiency and effect of task processing as a whole. Wherein, the agent is trained according to the task sample data in the experience data, the demand degree of each task in each characteristic element can be judged by marking, the task characteristic elements can be obtained by quantizing the value range of each real-time characteristic element, and the matching accuracy of computing power tasks and computing power devices can be improved through the one-by-one matching process of task characteristic elements and computing power device real-time characteristic elements, and the performance optimization and cost optimization of computing power scheduling can be realized.

[0019] In one embodiment, the real-time characteristic elements include transmission efficiency coefficient, operation cost coefficient, operation processing coefficient and image processing coefficient; the task characteristic elements include transmission demand coefficient, predicted operation time, operation processing demand coefficient and image processing demand coefficient; the process of coincidence matching includes: The matching value of the task characteristic elements and the i-th computing power device is calculated by the formula ; wherein j = 1, 2, 3; ; wherein j = 1, 2, 3; respectively are transmission weight coefficient, operation processing weight coefficient, graphics processing weight coefficient, are respectively selected and set according to the occupancy degree of different needs in empirical data; respectively are transmission efficiency coefficient, operation processing coefficient, image processing coefficient of the i th computing power device, respectively are transmission demand coefficient, operation processing demand coefficient, image processing demand coefficient, which are obtained according to the identification of the agent to the task; is the first definition function, when , , otherwise, , , are the mean values of the three groups , is a correction coefficient, which is set according to the value range and influence degree of in test data, respectively are reference values of transmission efficiency coefficient, operation processing coefficient and image processing coefficient, is used for numerical normalization processing of , so that the value of is set according to the critical value of in test value, through the matching value obtained process, when the accommodation degree of computing power device to the task in each feature is stronger (i.e. value is higher), when the occupancy of computing power device to each feature of the task is balanced (i.e. value is about small), the matching degree is higher (i.e. value is higher), so the top m computing power devices are selected in the order from large to small according to the matching value , the value of m is selected according to the order of magnitude of computing power in computing power cluster, the larger the order of magnitude, the larger the value of m can be set, and the top m computing power devices can better realize the processing process of the task, and then the matching balance value of the task feature element and the i th computing power device is calculated by formula , wherein, is the unit time cost of the i th computing power device, is the predicted operation time of the current task, which is obtained according to the prediction of the agent, is the cost unit quantity, which is determined according to the unit of , is the cost adjustment coefficient, which is set according to the test data and cost control degree fitting, so the matching balance value is larger, which means that when the task is processed by the i th computing power device, it can have better economy on the basis of ensuring reasonable scheduling of computing power, so each task is allocated to the matching balance value The maximum computing power device can further improve the performance optimization and cost optimization of computing power scheduling.

[0020] In one embodiment, a transmission efficiency coefficient calculation process of each computing power device is provided, including: The transmission efficiency coefficient of the i-th computing power device is calculated as ; wherein, is the average delay of the i-th computing power device, is a delay impact quantity reference function, which sets the corresponding impact quantity according to the range in which the different delays in the test data are located, fits the reference relationship, and then obtains the delay impact quantity reference function It should be noted that the larger the delay is, the smaller the value is, is the allowed throughput of the i-th computing power device, is the running task quantity of the i-th computing power device, , is the throughput of the k-th task of the i-th computing power device, is the running task quantity contention and synchronization waiting impact throughput, which is set according to the average throughput reduction of the computing power device under different running task quantities. Through this value, the accuracy of real-time throughput judgment of the computing power device can be improved, is a throughput unit value for removing the dimension of throughput, is a weight adjustment coefficient, which is set according to the test data fitting, so that , the transmission efficiency of the computing power device can be evaluated according to the delay and throughput of the computing power device, and x=1, 2, 3, 4; are the average storage write speed, the average storage write speed, the average memory write speed, and the average memory write speed of the i-th computing power device, respectively, are the storage write speed reference value, the storage write speed reference value, the memory write speed reference value, and the memory write speed reference value, respectively, are obtained according to the setting standard of the computing power device, is a second definition function, when , , otherwise, , are the storage write, storage write, memory write, and memory write impact weight coefficients, respectively, which are adaptively set according to different requirements of storage and memory in the test data, so that , the key factors related to transmission in the computing power device can be comprehensively considered to judge the transmission efficiency of the device.

[0021] In one embodiment, the calculation process of the operation processing coefficient includes: through the formula: The operation processing coefficient of the i-th computing power device is calculated ; wherein, is the operation capability value of the i-th computing power device, which is obtained according to the existing CPU performance scoring scheme, is the processing computing power occupancy ratio of the i-th computing power device, is The processing computing power occupancy ratio corresponds to the operation capability attenuation value, which is obtained by comparing the average of the historical running data of the CPU of the i-th computing power device, so that the real-time CPU performance state of the i-th computing power device can be judged through the operation processing coefficient .

[0022] In one embodiment, the calculation process of the image processing coefficient includes: through the formula: The image processing coefficient of the i-th computing power device is calculated ; wherein, is the graphics processing capability value of the i-th computing power device, which is obtained according to the existing GPU performance scoring scheme, is the graphics processing computing power occupancy ratio of the i-th computing power device, is The graphics processing computing power occupancy ratio corresponds to the graphics processing capability attenuation value, which is obtained by comparing the average of the historical running data of the GPU of the i-th computing power device, so that the real-time GPU performance state of the i-th computing power device can be judged through the image processing coefficient .

[0023] In one embodiment, a process for determining the processing order is provided, which includes: comparing the priority of the current task with the assigned tasks, when the priority of the current task is not higher than the priority of the last task in the assigned tasks, determining the processing order of the current task according to the time sequence; when the priority of the current task is higher than the priority of the last task in the assigned tasks, determining the processing order of the current task according to the difference between the priorities and the queue time of the last task in the assigned tasks, which specifically includes: through the formula: The adjustment coefficient E is calculated, wherein, is the queue time of the last task in the assigned tasks, is a correction coefficient, which is selected and set according to experience data, is the priority of the last task in the assigned tasks, and O is the priority of the current task, is the priority difference. The corresponding reference queue limit time needs to be explained. The priority number set in the embodiment is greater than 3 levels, and the corresponding reference queue limit time is set for different priorities. The higher the level, the shorter the corresponding reference queue limit time. Therefore, by adjusting the size of the coefficient E, the balance between the queue time and the priority can be determined. When E is greater than or equal to 0, the processing order of the current task is determined according to the time sequence. When E is less than 0, the current task is adjusted to be before the last task in the sorted task, thereby making the task processing order more reasonable.

[0024] In one embodiment, the data processing computing power scheduling method further comprises: monitoring the fault of each computing power device. The fault monitoring can be performed using a general scheme in the prior art. When the monitoring result is abnormal, the computing power device is not scheduled for tasks. Through this process, the influence of the faulty computing power device on the computing power scheduling process can be avoided.

[0025] Although the embodiments of the present application have been shown and described above, it should be understood that the above embodiments are exemplary and should not be construed as limiting the present application. Those skilled in the art can make changes, modifications, replacements and variations to the above embodiments within the scope of the present application.

Claims

1. A method for scheduling data processing computing power based on an agent, characterized in that, The method comprises: obtaining basic data and real-time running data of each computing power device in the computing power cluster, and obtaining real-time characteristic elements of each computing power device according to the basic data and the real-time running data; identifying each task based on an intelligent agent, and obtaining task characteristic elements and priorities; determining a processing sequence according to the queue time and the priority of the current task and the assigned tasks, and performing coincidence matching of the task characteristic elements of each task and the real-time characteristic elements of each computing power device, and assigning a corresponding computing power device to each task according to the coincidence matching result; the real-time characteristic elements comprise a transmission efficiency coefficient, an operation cost coefficient, an operation processing coefficient and an image processing coefficient; the task characteristic elements comprise a transmission demand coefficient, a predicted operation time length, an operation processing demand coefficient and an image processing demand coefficient; the coincidence matching process comprises: The matching value of the task characteristic element and the i-th computing power device is calculated by the formula ;​ According to the matching value Select the top m computing power devices in the order from large to small; calculate the matching balance value of the task characteristic element and the i-th computing power device through the formula , and each task is assigned to the matching balance value The largest computing power device;​ Where j = 1, 2, 3; They are transmission weight coefficient, operation processing weight coefficient, and graphic processing weight coefficient respectively; are the transmission efficiency coefficient, operation processing coefficient, and image processing coefficient of the i-th computing power device, respectively. They are transmission demand coefficient, operation processing demand coefficient, and image processing demand coefficient respectively; For the first defined function, when hour, ,otherwise, , , For three groups The mean of is the correction factor, They are the reference values ​​of transmission efficiency coefficient, operation processing coefficient and image processing coefficient respectively. is the unit time cost of the i-th computing device, is the predicted computing time of the current task, is the cost unit quantity, is the cost adjustment factor.

2. The agent-based data processing computing power scheduling method according to claim 1, characterized in that, the transmission efficiency coefficient calculation process of each computing power device comprises: by the formula: The transmission efficiency coefficient of the ith computing power device is calculated ; wherein, is the average delay of the ith computing power device, is the delay impact quantity reference function, is the allowed throughput of the ith computing power device, is the running task quantity of the ith computing power device, , is the throughput of the kth task of the ith computing power device, is the running task quantity contention and synchronization waiting impact throughput, is the throughput unit value, is the weight adjustment coefficient, x = 1, 2, 3, 4; are the average storage write speed, the average storage write speed, the average memory write speed, and the average memory write speed of the ith computing power device, respectively, are the storage write speed reference value, the storage write speed reference value, the memory write speed reference value, and the memory write speed reference value, respectively, is the second definition function, when , , otherwise, , are the storage write, storage write, memory write, and memory write impact weight coefficients, respectively.

3. The agent-based data processing computing power scheduling method of claim 2, wherein, the operation processing coefficient calculation process comprises: by the formula: calculating an operation processing coefficient of the ith computing power device ; wherein, is the operation capability value of the i-th computing power device, is the processing power occupancy ratio of the i-th computing power device, is the processing power occupancy ratio corresponds to the operation capability decay value.

4. The agent-based data processing computing power scheduling method according to claim 3, characterized in that, the image processing coefficient calculation process comprises: by the formula: calculating an image processing coefficient of the ith mining device ; wherein, is a graphics processing capability value of the i-th computing power device, is a graphics processing computing power occupation ratio of the i-th computing power device, is The graphics processing computing power occupation ratio corresponds to a graphics processing capability decay value.

5. The agent-based data processing computing power scheduling method according to claim 1, characterized in that, the processing sequence determination process comprises: comparing the priorities of the current task and the assigned tasks, and determining the processing sequence of the current task according to the time sequence when the priority of the current task is not higher than the priority of the last task in the assigned tasks, or determining the processing sequence of the current task according to the difference between the priorities and the queue time of the last task in the assigned tasks when the priority of the current task is higher than the priority of the last task in the assigned tasks.

6. The agent-based data processing computing power scheduling method according to claim 5, characterized in that, the process of determining the processing sequence of the current task according to the difference between the priorities and the queue time of the last task in the assigned tasks comprises: by the formula: obtaining an adjustment coefficient E, and determining the processing sequence of the current task according to the time sequence when E is greater than or equal to 0, or adjusting the current task to before the last task in the assigned tasks when E is less than 0. wherein, is the queue time of the last task in the assigned task, is the correction factor, is the priority of the last task in the assigned task, O is the current task priority, is the priority is the corresponding reference queue limit time.

7. The agent-based data processing computing power scheduling method of claim 1, wherein, the method further comprises: monitoring the failure of each computing power device, and not scheduling tasks for the computing power device when the monitoring result is abnormal.

Images