Task scheduling method and system based on electric power big data model

Abstract

The invention discloses a task scheduling method and system based on an electric power big data model. The method comprises the following steps: S1, collecting and preprocessing electric power system data by utilizing the electric power big data model, calculating a data integrity index, and integrating high-integrity data; s2, on the basis of the integrated data, analyzing features of a task to be scheduled, calculating average execution time of the task and an expected value of resource consumption, and determining an execution difficulty coefficient of the task; s3, evaluating a system resource condition, calculating a resource utilization rate and a residual amount, and allocating resources according to a task execution difficulty coefficient; s4, optimizing the execution sequence and time arrangement of the tasks in combination with the task characteristics and the resource allocation condition, and formulating a task scheduling scheme; s5, executing the task, monitoring the task progress and the resource use condition, calculating the task execution progress deviation and the resource consumption change rate, and adjusting the scheduling scheme according to the monitoring result; according to the method, the reasonability, accuracy and overall operation efficiency of task scheduling in the power system can be effectively improved.

Description

Technical Field

[0001] The present invention relates to technical fields such as power systems, big data technologies, task scheduling and optimization, and software engineering, and in particular to a task scheduling method and system based on a power big data model. Background Art

[0002] In today's power system, with the rapid development of smart grids and the explosive growth of power data, traditional task scheduling methods face many challenges. On the one hand, the tasks in the power system are becoming increasingly complex and diverse, including equipment monitoring, fault diagnosis, load forecasting, etc. These tasks have different requirements for computing resources and time. Traditional methods are difficult to accurately assess the needs and priorities of tasks, resulting in unreasonable resource allocation and affecting the efficiency of task execution.

[0003] On the other hand, the emergence of power big data provides new opportunities and challenges for task scheduling. Although big data contains rich information, how to effectively collect, store, process and analyze this data and apply it to task scheduling is an urgent problem to be solved. At present, there is a lack of a task scheduling method and system that can fully utilize the power big data model to achieve efficient resource allocation and task execution sequence optimization, ensuring the stable operation and reliable service of the power system. Summary of the invention

[0004] The task scheduling method and system based on the power big data model include:

[0005] S1. Collect and pre-process power system data using the power big data model, calculate the data integrity index, and integrate high-integrity data;

[0006] S2. Based on the integrated data, analyze the characteristics of the tasks to be scheduled, calculate the average execution time and expected resource consumption of the tasks, and determine the execution difficulty coefficient of the tasks;

[0007] S3, evaluate the system resource status, calculate resource utilization and remaining amount, and allocate resources according to the difficulty coefficient of task execution;

[0008] S4. Optimize the execution order and time arrangement of tasks and formulate task scheduling plans based on task characteristics and resource allocation;

[0009] S5. Execute tasks and monitor task progress and resource usage, calculate task execution progress deviation and resource consumption change rate, and adjust the scheduling plan based on the monitoring results.

[0010] The task scheduling method based on the power big data model as described above, wherein the power big data model is used to collect and preprocess power system data, calculate the data integrity index, and integrate high integrity data, includes the following sub-steps:

[0011] Use the power big data model to collect various data in the power system and pre-process the collected data;

[0012] Calculate the data integrity index and select high-integrity data for integration.

[0013] The task scheduling method based on the power big data model as described above, wherein, based on the integrated data, the characteristics of the tasks to be scheduled are analyzed, the average execution time and expected resource consumption of the tasks are calculated, and the execution difficulty coefficient of the tasks is determined, includes the following sub-steps:

[0014] Based on the integrated high-integrity data, in-depth analysis of the characteristics of the tasks to be scheduled;

[0015] By analyzing and modeling historical task execution data, the average execution time of tasks is calculated to provide a time reference for task scheduling;

[0016] Predict the expected resource consumption of the task based on the characteristics and resource requirements of the task;

[0017] The average execution time and expected resource consumption of the comprehensive task are used to determine the execution difficulty coefficient of the task, providing a basis for resource allocation.

[0018] The task scheduling method based on the power big data model as described above, wherein the system resource status is evaluated, resource utilization and remaining amount are calculated, and resources are allocated according to the task execution difficulty coefficient, includes the following sub-steps:

[0019] Monitor and evaluate various resources in the power system in real time and calculate resource utilization;

[0020] Calculate the remaining resources based on resource utilization and total resources;

[0021] Allocate resources reasonably according to the difficulty coefficient of task execution to ensure efficient use of resources.

[0022] The task scheduling method based on the power big data model as described above, wherein the execution order and time arrangement of tasks are optimized in combination with task characteristics and resource allocation, and a task scheduling plan is formulated, includes the following sub-steps:

[0023] Combine the characteristics of the task and the resource allocation, and comprehensively consider factors such as task priority, execution time, and resource requirements;

[0024] Use optimization algorithms to optimize the execution order and time schedule of tasks and formulate the best task scheduling plan;

[0025] Ensure that the task scheduling plan maximizes the utilization of system resources while meeting task requirements.

[0026] The task scheduling method based on the power big data model as described above, wherein the task is executed and the task progress and resource usage are monitored, the task execution progress deviation and resource consumption change rate are calculated, and the scheduling plan is adjusted according to the monitoring results, including the following sub-steps:

[0027] Execute tasks according to the task scheduling plan and monitor the progress of tasks and resource usage in real time;

[0028] Calculate the task execution progress deviation and promptly discover problems in the task execution process;

[0029] Calculate the resource consumption change rate and grasp the dynamic changes of resources;

[0030] According to the monitoring results, timely adjust the task scheduling plan to ensure the smooth execution of tasks and the rational use of resources.

[0031] The task scheduling system based on the power big data model includes:

[0032] Data acquisition and preprocessing module: Use the power big data model to collect and preprocess power system data, calculate the data integrity index, and integrate high-integrity data;

[0033] Data analysis module: Based on the integrated data, analyze the characteristics of the tasks to be scheduled, calculate the average execution time and expected resource consumption of the tasks, and determine the execution difficulty coefficient of the tasks;

[0034] Resource evaluation module: evaluates system resource status, calculates resource utilization and remaining amount, and allocates resources according to the difficulty coefficient of task execution;

[0035] Task scheduling module: Combines task characteristics and resource allocation to optimize the execution order and time schedule of tasks and formulate task scheduling plans;

[0036] Task execution and monitoring module: executes tasks and monitors task progress and resource usage, calculates task execution progress deviation and resource consumption change rate, and adjusts the scheduling plan based on the monitoring results.

[0037] A computer storage medium, characterized in that it comprises: at least one memory and at least one processor;

[0038] A memory for storing one or more program instructions;

[0039] A processor is used to run one or more program instructions to execute any of the task scheduling methods based on the power big data model described above.

[0040] The beneficial effects achieved by the present invention are as follows:

[0041] The task scheduling method and system based on the power big data model have brought many significant beneficial effects. First, it greatly improves the efficiency of task scheduling. With the help of the power big data model, it accurately analyzes the task characteristics and system resource status, optimizes the execution order and time arrangement, shortens the average task execution time, and reduces resource waste. Secondly, it realizes the reasonable allocation of resources. By calculating the data integrity index, the task execution difficulty coefficient, and evaluating the system resource status, it ensures the maximum resource utilization and avoids excessive or insufficient resource allocation. Furthermore, it enhances the stability and reliability of the system, monitors the task progress and resource usage in real time, calculates the progress deviation and resource consumption change rate, finds problems in time and adjusts the scheduling plan to reduce the probability of failure. At the same time, the method and system can adapt to the complex and changeable power system environment, and dynamically adjust the task scheduling plan to cope with the changing task requirements and resource conditions. Finally, it provides strong support for the development of smart grids, promotes the power industry to achieve intelligent control, optimized operation and reliable services, and promotes sustainable development. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the present invention. For ordinary technicians in this field, other drawings can also be obtained based on these drawings.

[0043] Figure 1 It is a flow chart of a task scheduling method based on a power big data model provided in an embodiment of the present application.

[0044] Figure 2 It is a schematic diagram of a task scheduling system based on a power big data model provided in an embodiment of the present application. DETAILED DESCRIPTION

[0045] The following is a clear and complete description of the technical solutions in the embodiments of the present invention in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0046] Embodiment 1

[0047] like Figure 1 As shown, the task scheduling method based on the power big data model in the embodiment of the present application includes:

[0048] Step S1: using the power big data model to collect and pre-process power system data, calculate the data integrity index, and integrate high integrity data, specifically including the following sub-steps:

[0049] Step S11: using the power big data model to collect various data in the power system and pre-process the collected data;

[0050] First, the power big data model is used to collect various data of the power system from multiple data sources such as smart meters, sensors, and SCADA systems, including electrical parameters such as voltage, current, power, and electricity, as well as information such as equipment status and environmental parameters. Then, data preprocessing is performed, including data cleaning, removing outliers and noise data by setting thresholds and statistical analysis; data conversion, converting data in different formats into a unified format for subsequent processing; and data normalization, so that data with different characteristics have the same scale.

[0051] Step S12: Calculate the data integrity index and select high-integrity data for integration;

[0052] When calculating the data integrity index, multiple factors need to be considered comprehensively, and weights can be set for each key attribute of the data, such as data completeness, accuracy, timeliness, and stability.

[0053] 1. Completeness: During the data collection phase, when obtaining data from various data sources, the amount of data N for each data attribute j Record, in the process of data preprocessing, after data cleaning and filling, determine the number of complete data M j , then the completeness rate calculation formula is Among them, C j is the completeness rate; M j is the amount of complete data; N j Data Attribute A j The total amount of data.

[0054] 2. Accuracy: For common data attributes in the power system, there are corresponding standard operating ranges and ideal values. j The data value x ij Then, calculate the mean of each data value and the standard data The absolute error value Using formula To calculate the accuracy, where A cj It is an accuracy indicator of data attributes, which is used to measure the closeness between the data value of the data attribute and the standard data value. The closer it is to 1, the more accurate the data is; the closer it is to 0, the lower the data accuracy is. j Data Attribute A j The total amount of data; x ij Is the data attribute A j The specific data points in the datasets included; It is the mean of the standard data values ​​corresponding to the data attribute, and is the average level of the ideal or standard data values ​​of the data attribute; A maxj It is the maximum acceptable absolute value of the error in the pre-selected data attributes.

[0055] 3. Timeliness: When collecting data, for each data value x ij Record acquisition timestamp t ij , when it is necessary to calculate the timeliness T j When, according to the formula Calculate, where T j is the timeliness index of the data attribute, N j Data Attribute A j The total amount of data; t ij is the timestamp of the ith data value in the data attribute; t now is the current time; T maxj is the maximum acceptable time difference of the data attribute.

[0056] 4. Stability: According to the pre-set maximum acceptable standard deviation, according to the formula Calculate stability, where S j is the stability index of data attributes; σ j It is the standard deviation of the data value sequence of the data attribute, which reflects the degree of dispersion of the data value of the data attribute relative to its mean. By calculating the standard deviation of the data value sequence, the fluctuation of the data can be evaluated; S maxj is the maximum acceptable standard deviation of the data attribute.

[0057] In summary, the integrity score of the data attribute is S Ij =α j C j +β j A cj +γ j T j +δ j S j , where S Ij is the integrity score of the data attribute; α j , β j , γ j , δ j The completeness rate C j 、AccuracyA cj , Timeliness T j And stability S j Finally, the calculation formula of data integrity index is: Where I is the data integrity index; m is the number of data attributes, that is, the total number of different data attributes involved in calculating data integrity; j is the index variable, and j ≥ 1; ω jis the weight of the jth data attribute; S Ij is the completeness score of the data attribute.

[0058] According to the set integrity index threshold, high-integrity data is screened out. These high-integrity data have higher reliability and availability. They can be integrated by merging data sets to ensure data consistency and coherence, providing an accurate and reliable data basis for subsequent task scheduling.

[0059] S2. Based on the integrated data, analyze the characteristics of the tasks to be scheduled, calculate the average execution time and expected resource consumption of the tasks, and determine the execution difficulty coefficient of the tasks, which specifically includes the following sub-steps:

[0060] Step S21, based on the integrated high-integrity data, deeply analyzing the characteristics of the tasks to be scheduled;

[0061] Conduct a comprehensive review and analysis of the integrated high-integrity data. By carefully studying the data, determine the specific type of task, such as data computing tasks, data transmission tasks, or monitoring tasks. At the same time, clarify the priority of tasks, and give more attention to urgent and important tasks. In addition, analyze the data processing volume, complexity, and time sensitivity of the task. For a large-scale data computing task, it is necessary to consider its algorithm complexity and data size in order to arrange computing resources reasonably. By deeply understanding the characteristics of the task, a more accurate and effective strategy can be formulated for task scheduling to ensure that the task can be executed efficiently and stably.

[0062] Step S22: Calculate the average execution time of tasks by analyzing and modeling historical task execution data to provide a time reference for task scheduling;

[0063] We collect a wide range of data from the execution of historical tasks, including the start time, end time, time spent on the execution process, and time distribution at different stages. Then, we use data analysis and modeling techniques to deeply process and analyze these historical data. We use the following formula to calculate the average execution time of a task:

[0064]

[0065] Where T is the average execution time of the task; n represents the total number of tasks to be scheduled; i is an index variable used to traverse each individual task; ω i is the task weight, reflecting the importance and priority of the task; f r (T i ) is the resource utilization adjustment factor, which reflects the impact of resource usage on task execution time; T i is the actual execution time of the task; Di is the dependency delay, which is the delay time of the task caused by the dependency between tasks; g(t i ) is the time decay function, reflecting the impact of task completion at different time points; h(X i ) is a machine learning-based method for predicting the execution time of tasks.

[0066] When faced with a new task to be scheduled, it can quickly estimate its approximate execution time based on the type and characteristics of the task.

[0067] Step S23: predicting the expected value of resource consumption of the task according to the characteristics and resource requirements of the task;

[0068] Combine the specific characteristics of the task and the demand for various resources to establish a corresponding prediction model. Refer to the historical resource consumption of the task for analog prediction, and use machine learning algorithms to make more accurate predictions of the task's resource consumption. For a data-intensive task that requires a large memory and high network bandwidth, by predicting its expected resource consumption value, sufficient resources can be allocated in advance to avoid insufficient resources during execution and prevent resource waste.

[0069] Step S24: Determine the execution difficulty coefficient of the task by integrating the average execution time of the task and the expected value of resource consumption, so as to provide a basis for resource allocation;

[0070] The execution difficulty coefficient of a task is determined by combining the average execution time and expected resource consumption of the task. First, the execution difficulty coefficient can clearly understand the complexity and resource requirements of each task in task scheduling. The average execution time reflects the requirements of the task in the time dimension, while the expected resource consumption reflects the occupation of system resources by the task. If the system is highly sensitive to time, the weight of the average execution time can be appropriately increased; if the efficiency of resource utilization is emphasized, the weight of the expected resource consumption can be increased. To ensure the accuracy of the calculation, the average execution time and the expected resource consumption value need to be standardized so that they have a unified dimension.

[0071] After obtaining the execution difficulty coefficient, sort the tasks according to their size. Tasks with high execution difficulty are given priority in resource allocation to ensure their smooth execution. At the same time, the calculation method of the execution difficulty coefficient is continuously optimized and adjusted according to actual conditions. Collect more historical data, analyze the execution status and resource consumption of different tasks, and adjust the weight ratio and standardization method. In addition, combined with other factors such as task priority and urgency, further improve the assessment of task execution difficulty to provide a basis for resource allocation.

[0072] Step S3: Evaluate the system resource status, calculate resource utilization and remaining amount, and allocate resources according to the task execution difficulty coefficient, which specifically includes the following sub-steps:

[0073] Step S31, real-time monitoring and evaluation of various resources in the power system and calculation of resource utilization;

[0074] To monitor and evaluate various resources in the power system in real time and calculate resource utilization, we must first identify the types of resources that need to be monitored, such as computing resources, storage resources, and network resources. Then, use the corresponding monitoring tools or system interfaces to collect real-time data. For computing resources, use the performance monitoring tools of the operating system; for storage resources, query the disk management tool to obtain the usage and available capacity information of the disk partition; for network resources, use network monitoring software or network device management interfaces to obtain bandwidth usage and network traffic data.

[0075] For computing resources:

[0076]

[0077] Among them, U a is the resource utilization rate that comprehensively considers CPU and memory usage; n is the number of different time periods or different task running instances monitored; U ci is the CPU usage rate in the i-th time period; ω ci is the weight of the CPU in the i-th time period, and the weight value can be assigned according to actual experience or system characteristics; U mi is the memory usage rate in the time period; ω mi is the weight of memory in the i-th time period; It is the average value of CPU usage in all time periods; The average memory usage for all time periods.

[0078] For storage resources:

[0079]

[0080] Among them, U b is the resource utilization rate that takes into account the disk read and write performance and the remaining space; C is the amount of resources used; T is the total amount of resources; m is the number of different sampling times for counting the disk read and write speed; i is an index variable whose value range is 1≤i≤m; r i is the disk reading speed at the i-th sampling time; is the average of all sampled reading speeds; ω i is the disk writing speed at the i-th sampling time; is the average of all sampled write speeds; t wait is the average disk I / O waiting time; ttotal is the total disk I / O time.

[0081] For web resources:

[0082]

[0083] Among them, U c B is the resource utilization rate that takes into account bandwidth, packet loss rate, and delay factors; used is the actual network bandwidth used; B total is the total network bandwidth; k is the number of different time periods for statistical network status; p j is the packet loss rate of the network in each j-th time period; is the average packet loss rate of all time periods; d j is the network delay duration in the jth period; is the average delay time of all time periods; It is the duration of the network being in a busy data transmission state during the jth period; is the total duration of the jth period.

[0084] Step S32, calculating the remaining amount of resources according to the resource utilization rate and the total amount of resources;

[0085] Determine the total amount of resources by checking the hardware specification manual or using operating system tools. In actual operation, it is necessary to consider complex situations such as dynamic changes in resources and the relationship between multiple resource types. The following formula is used to calculate the average remaining amount of resources:

[0086]

[0087] Among them, S avg is the average remaining amount of resources; T represents the total amount of resources; t1 and t2 represent the starting point and end point of a time interval; U(t) represents the function of resource utilization over time; Represents the definite integral of the resource utilization function U(t) over the time period [t1, t2].

[0088] Step S33: Allocate resources reasonably according to the execution difficulty coefficient of the task to ensure efficient use of resources.

[0089] Comprehensively evaluate and set the difficulty coefficient based on factors such as the type, complexity, amount of data required, and expected running time of the task. Simple data query tasks have a lower difficulty coefficient, while complex power system flow calculation and large-scale computing tasks have a higher difficulty coefficient. Establish a task difficulty assessment standard, and have professionals score and determine the coefficient based on the characteristics of the task. Next, analyze the matching of resource demand and remaining resources. For each task to be allocated resources, estimate the demand for various resources based on its execution difficulty coefficient, compare the remaining amount of various resources in the current system, and determine whether the task operation requirements can be met. If the remaining resources are insufficient, consider suspending low-priority tasks or dynamically expanding resources. Finally, formulate a resource allocation strategy to prioritize high-priority and high-difficulty tasks according to task priority to obtain sufficient resources. For tasks that have both computing-intensive and storage-intensive requirements, allocate CPU, memory and other resources in proportion from the remaining resources based on their degree of dependence on different resources.

[0090] Step S4: Optimize the execution order and time arrangement of tasks and formulate a task scheduling plan based on task characteristics and resource allocation.

[0091] By comprehensively considering the task characteristics and resource allocation, a scientific and reasonable task scheduling plan is formulated to improve the overall performance and efficiency of the system and ensure that each task can be completed smoothly and efficiently. The specific steps include the following:

[0092] Step S41, combining the characteristics of the task and the resource allocation, comprehensively considering factors such as the task priority, execution time, and resource requirements;

[0093] In-depth analysis of the characteristics of the task to determine whether the task is time-sensitive. If so, give it a higher priority to ensure that it is completed within the specified time; determine whether the task has dependencies. For tasks with dependencies, the execution time must be reasonably arranged after the dependent tasks are completed. Then, combined with the resource allocation situation, according to the specific resource requirements of different tasks, computing resource-intensive tasks focus on the allocation of CPU and memory, and tasks with large storage resource requirements focus on the availability of disk space. At the same time, comprehensively consider the priority of the task, and give priority to high-priority tasks in resource allocation to ensure that their execution is not overly affected by low-priority tasks. For tasks with a long execution time, their impact on the overall performance of the system must be fully considered when allocating resources to avoid long-term occupation of a large amount of resources that prevent other tasks from being executed in time. By comprehensively analyzing the characteristics of the task and the resource allocation situation, as well as comprehensively considering the priority, execution time, and resource demand factors of the task, a scientific and reasonable task execution plan can be formulated.

[0094] Step S42: Optimize the execution order and time arrangement of tasks using an optimization algorithm to formulate an optimal task scheduling plan;

[0095] Select the appropriate optimization algorithm based on the expected goal of task scheduling. Taking genetic algorithm as an example, first randomly generate a set of initial task scheduling schemes as the population, then calculate the fitness value of each scheme according to the set evaluation rules, and then use selection, crossover, mutation and other operations to generate a new population. Repeat this cycle and continuously iterate and update the population until the established stop condition is met. Finally, the optimal task scheduling scheme is determined to achieve scientific optimization steps for task execution order and time arrangement. For task scheduling problems, define the fitness function to evaluate the pros and cons of task scheduling schemes:

[0096]

[0097] Where f(x) is the fitness function; α, β, and γ are weight coefficients that are adjusted according to the specific optimization goal; n is the number of tasks; i is an index variable; t i represents the execution time of the i-th task; u i Indicates the amount of resources occupied by the i-th task; U total Indicates the total resources of the system; p i Indicates the priority of the i-th task.

[0098] Step S43: ensuring that the task scheduling scheme meets the task requirements while maximizing the utilization of system resources;

[0099] After developing a task scheduling plan, it is necessary to fully verify it. Check whether each task can be smoothly executed according to the established order and schedule, whether it meets its own requirements for time, resources, etc., whether time-sensitive tasks can be completed within the specified time, and whether tasks with large resource requirements can obtain sufficient resources. At the same time, calculate the actual utilization rate of system resources under the scheduling plan, compare the resource utilization rates under different plans, and maximize the system resource utilization rate as much as possible by continuously adjusting and optimizing the task order and time schedule in the scheduling plan, ensuring that the entire task scheduling not only meets the task requirements but also uses resources efficiently.

[0100] Step S5, executing tasks and monitoring task progress and resource usage, calculating task execution progress deviation and resource consumption change rate, and adjusting the scheduling plan according to the monitoring results, specifically including the following sub-steps:

[0101] Step S51, executing tasks according to the task scheduling plan, and monitoring the progress of tasks and resource usage in real time;

[0102] According to the established task scheduling plan, start the execution process of each task. During the execution of the task, build an effective monitoring system, which can obtain real-time data through various software and hardware interfaces related to task execution. The monitoring of task progress can be started from multiple dimensions, such as setting the key nodes of the task, and judging the degree of progress by detecting whether the task reaches these key nodes on time; it can also be measured by counting the proportion of the number of completed subtasks to the total number of tasks. For the monitoring of resource usage, real-time data collection is performed on computing resources, storage resources, and network resources, and these data are integrated and stored.

[0103] Step S52: Calculate the task execution progress deviation and promptly discover problems in the task execution process;

[0104] To clarify the original planned execution progress of the task, it is usually pre-set in the task scheduling plan, with time or task completion as the measurement indicator. Then, compare the difference between the actual task execution progress and the planned progress in real time, and use the following formula to calculate the progress deviation of the task execution:

[0105]

[0106] Where f(x) represents the task execution progress deviation; n represents the number of different factors that affect the task progress; i is an index variable whose value range is 1 to n; ω i represents the weight of the i-th factor; E i It represents the quantitative value of the actual completion of the i-th factor; P i It represents the quantitative value of the planned completion of the i-th factor.

[0107] Step S53: Calculate the resource consumption change rate to understand the dynamic change of resources;

[0108] Determine the initial usage and current usage of different resources in a specific time period. Taking CPU resources as an example, record the CPU usage at the beginning of the task as the initial usage, and then obtain the CPU usage at the current moment as the current usage. Use the following formula to calculate the resource consumption change rate:

[0109]

[0110] Where V represents the rate of change of resource consumption; m represents the number of different resource types; j is an index variable; ω j represents the weight of the jth resource type; T1 and T2 represent two different time points, which are used to calculate the consumption change of resources during this period; R j1 represents the usage of the jth resource at time point T1; R j2Represents the usage of the j-th resource at time point T2.

[0111] Step S54: timely adjust the task scheduling plan according to the monitoring results to ensure the smooth execution of tasks and the rational use of resources;

[0112] After calculating the task execution progress deviation and resource consumption change rate through the previous steps, adjust the scheduling plan based on these monitoring results. If the task execution progress deviation shows that the task is behind schedule, it is necessary to analyze whether it is caused by insufficient resources or other reasons such as technical difficulties in the task itself. If there are insufficient resources, such as the CPU usage rate is high for a long time, resulting in slow task processing, you can consider allocating more computing resources to the task; if there are technical difficulties in the task, it is necessary to deploy professional technicians to intervene and solve them, and adjust the scheduling time of subsequent tasks appropriately. In the case of abnormal resource consumption change rate, such as excessive growth in the consumption of a certain resource, it is necessary to check whether the task's use logic of the resource is reasonable, optimize the task algorithm to reduce resource consumption if necessary, or reallocate resources to other more urgent tasks to ensure the rational use of overall resources and ensure that all tasks can be completed smoothly as expected.

[0113] Embodiment 2

[0114] like Figure 2 As shown, the second embodiment of the present application provides a task scheduling system based on a power big data model, including:

[0115] Data collection and preprocessing module 21: Use the power big data model to collect and preprocess power system data, calculate the data integrity index, and integrate high integrity data, including the following submodules:

[0116] Data collection submodule 211: uses the power big data model to collect various data in the power system and pre-processes the collected data.

[0117] First, with the help of the powerful data acquisition capabilities of the power big data model, we collect power-related data covering different levels and types from many data sources in the power system. These data are the basic raw materials for subsequent analysis and application. However, the newly collected data often have various problems and need to be preprocessed. The preprocessing work includes data cleaning, data standardization, and data normalization. These preprocessing works can improve the quality of the data and ensure that the subsequent processing links can be carried out based on reliable data, providing a good data start for the entire task scheduling method based on the power big data model.

[0118] Data integrity assessment and integration submodule 212: Calculate the data integrity index and select high-integrity data for integration.

[0119] When calculating the data integrity index, it is necessary to consider multiple factors comprehensively, set weights for each key attribute of the data, and calculate based on the four factors of data completeness, accuracy, timeliness, and stability.

[0120] 1. Completeness: During the data collection phase, when obtaining data from various data sources, the amount of data N for each data attribute j Record, in the process of data preprocessing, after data cleaning and filling, determine the number of complete data M j , then the completeness rate calculation formula is Among them, C j is the completeness rate; M j is the amount of complete data; N j Data Attribute A j The total amount of data.

[0121] 2. Accuracy: For common data attributes in the power system, there are corresponding standard operating ranges and ideal values. j The data value x ij Then, calculate the mean of each data value and the standard data The absolute error value Using formula To calculate the accuracy, where A cj It is an accuracy indicator of data attributes, which is used to measure the closeness between the data value of the data attribute and the standard data value. The closer it is to 1, the more accurate the data is; the closer it is to 0, the lower the data accuracy is. j Data Attribute A j The total amount of data; x ij Is the data attribute A j The specific data points in the datasets included; It is the mean of the standard data values ​​corresponding to the data attribute, and is the average level of the ideal or standard data values ​​of the data attribute; A maxj It is the maximum acceptable absolute value of the error in the pre-selected data attributes.

[0122] 3. Timeliness: When collecting data, for each data value x ij Record acquisition timestamp t ij , when it is necessary to calculate the timeliness T j When, according to the formula Calculate, where T j is the timeliness index of the data attribute, N j Data Attribute A j The total amount of data; t ij is the timestamp of the ith data value in the data attribute; t now is the current time; T maxjis the maximum acceptable time difference of the data attribute.

[0123] 4. Stability: According to the pre-set maximum acceptable standard deviation, according to the formula Calculate stability, where S j is the stability index of data attributes; σ j It is the standard deviation of the data value sequence of the data attribute, which reflects the degree of dispersion of the data value of the data attribute relative to its mean. By calculating the standard deviation of the data value sequence, the fluctuation of the data can be evaluated; S maxj is the maximum acceptable standard deviation of the data attribute.

[0124] In summary, the integrity score of the data attribute is S Ij =α j C j +β j A cj +γ j T j +δ j S j , where S Ij is the integrity score of the data attribute; α j , β j , γ j , δ j The completeness rate C j 、AccuracyA cj , Timeliness T j And stability S j Finally, the calculation formula of data integrity index is: Where I is the data integrity index; m is the number of data attributes, that is, the total number of different data attributes involved in calculating data integrity; j is the index variable, and j ≥ 1; ω j is the weight of the jth data attribute; S Ij is the completeness score of the data attribute.

[0125] According to the calculated data integrity index, corresponding thresholds or screening criteria are set to select data with higher integrity. Then these high-integrity data are integrated. The integration process involves organizing scattered data together according to certain logical relationships to form a more systematic and usable data set, providing strong data support for the efficient operation of the entire power system.

[0126] Data analysis module 22: Based on the integrated data, analyze the characteristics of the tasks to be scheduled, calculate the average execution time and expected resource consumption of the tasks, and determine the execution difficulty coefficient of the tasks, including the following sub-modules:

[0127] Task feature analysis submodule 221: Based on the integrated high-integrity data, the features of the tasks to be scheduled are deeply analyzed; the tasks are of different types, such as maintenance of power equipment, further analysis of power data, or scheduling of power transmission, as well as the urgency and importance of the tasks. This is a key link in understanding the nature of the tasks in the entire task scheduling process.

[0128] Task execution time calculation submodule 222: By analyzing and modeling historical task execution data, the average execution time of the task is calculated to provide a time reference for task scheduling; this submodule focuses on the analysis of the time dimension. By reviewing the execution records of similar tasks in the past, using data analysis technology and appropriate modeling methods, the average execution time of the task is calculated. The average execution time can provide a more accurate time reference for the current task to be scheduled, helping the scheduling system to be more reasonable in arranging the task execution sequence and time window, and avoiding problems such as low efficiency of the power system caused by task accumulation or unreasonable time allocation. The calculation formula is:

[0129] The average execution time of a task is calculated using the following formula:

[0130]

[0131] Where T is the average execution time of the task; n represents the total number of tasks to be scheduled; i is an index variable used to traverse each individual task; ω i is the task weight, reflecting the importance and priority of the task; f r (T i ) is the resource utilization adjustment factor, which reflects the impact of resource usage on task execution time; T i is the actual execution time of the task; D i is the dependency delay, which is the delay time of the task caused by the dependency between tasks; g(t i ) is the time decay function, reflecting the impact of task completion at different time points; h(X i ) is a machine learning-based method for predicting the execution time of tasks.

[0132] When faced with a new task to be scheduled, it can quickly estimate its approximate execution time based on the type and characteristics of the task.

[0133] Resource consumption prediction submodule 223: predict the expected resource consumption value of the task according to the characteristics and resource requirements of the task; combine the previously analyzed task characteristics and the known resource requirements during the task execution process, and use appropriate prediction methods to estimate the expected resource consumption value of the task. Knowing how much resources the task will consume during the execution process provides an important basis for resource allocation, ensuring that resources can be reasonably allocated to each task, and avoiding resource waste or resource shortage.

[0134] Execution difficulty coefficient confirmation submodule 224: The execution difficulty coefficient of the task is determined by comprehensively considering the average execution time and expected resource consumption value of the task, providing a basis for resource allocation; the execution difficulty coefficient of the task is determined by comprehensively weighing the average execution time and expected resource consumption value of the task, and adopting a suitable calculation method. This provides a direct basis for subsequent resource allocation, helping the scheduling system to reasonably allocate resources according to the difficulty of the task, giving priority to tasks with lower difficulty and higher importance, and also making resource and time planning in advance for tasks with higher difficulty.

[0135] Resource evaluation module 23: evaluates the system resource status, calculates resource utilization and remaining amount, and allocates resources according to the task execution difficulty coefficient, including the following sub-modules:

[0136] Resource status assessment submodule 231: monitor and evaluate various resources in the power system in real time and calculate resource utilization; obtain resource usage in real time, and calculate resource, storage resource, and network resource usage information. Through this information, a specific calculation method is used to calculate resource utilization, so as to clearly understand the current resource busyness and provide the most basic resource status information for subsequent resource allocation decisions.

[0137] For computing resources:

[0138]

[0139] Among them, U a is the resource utilization rate that comprehensively considers CPU and memory usage; n is the number of different time periods or different task running instances monitored; U ci is the CPU usage rate in the i-th time period; ω ci is the weight of the CPU in the i-th time period, and the weight value can be assigned according to actual experience or system characteristics; U mi is the memory usage rate in the time period; ω mi is the weight of memory in the i-th time period; It is the average value of CPU usage in all time periods; The average memory usage for all time periods.

[0140] For storage resources:

[0141]

[0142] Among them, U b is the resource utilization rate that takes into account the disk read and write performance and the remaining space; C is the amount of resources used; T is the total amount of resources; m is the number of different sampling times for counting the disk read and write speed; i is an index variable whose value range is 1≤i≤m; r i is the disk reading speed at the i-th sampling time; is the average of all sampled reading speeds; ω i is the disk writing speed at the i-th sampling time; is the average of all sampled write speeds; t wait is the average disk I / O waiting time; t total is the total disk I / O time.

[0143] For web resources:

[0144]

[0145] Among them, U c B is the resource utilization rate that takes into account bandwidth, packet loss rate, and delay factors; used is the actual network bandwidth used; B total is the total network bandwidth; k is the number of different time periods for statistical network status; p j is the packet loss rate of the network in each j-th time period; is the average packet loss rate of all time periods; d j is the network delay duration in the jth period; is the average delay time of all time periods; It is the duration of the network being in a busy data transmission state during the jth period; is the total duration of the jth period.

[0146] Resource margin calculation submodule 232: Calculate resource margin according to resource utilization and total resource amount; determine total resource amount by checking hardware specification manual, using operating system tools, etc. In actual operation, complex situations such as dynamic changes of resources and the relationship between multiple resource types need to be considered, and the following formula is used to calculate average resource margin:

[0147]

[0148] Among them, S avg is the average remaining amount of resources; T represents the total amount of resources; t1 and t2 represent the starting point and end point of a time interval; U(t) represents the function of resource utilization over time; Represents the definite integral of the resource utilization function U(t) over the time period [t1, t2].

[0149] Based on the needs of different tasks, resources can be arranged more scientifically and reasonably to avoid blind allocation due to lack of understanding of remaining resources, which may lead to resource shortages affecting task execution or idle resources causing waste.

[0150] Resource allocation submodule 233: According to the execution difficulty coefficient of the task, reasonably allocate resources to ensure efficient use of resources; according to the previously determined task execution difficulty coefficient, combined with the calculated resource surplus and other information, reasonably allocate resources. Consider the difficulty of the task to decide how many resources to allocate to each task. For tasks with a lower execution difficulty coefficient, relatively fewer resources are allocated to avoid wasting resources; for tasks with a higher execution difficulty coefficient, sufficient resources are allocated according to their specific difficulty and importance to ensure that the tasks can be executed smoothly. This allocation method aims to maximize the utilization efficiency of resources and ensure the smooth implementation of various tasks in the power system.

[0151] Task scheduling module 24: Combine task characteristics and resource allocation to optimize the execution order and time arrangement of tasks and formulate task scheduling plans, including the following sub-modules:

[0152] Comprehensive consideration of factors submodule 241: Combined with the characteristics of the task and the resource allocation, the system comprehensively considers the priority, execution time, resource requirements and other factors of the task; the system integrates and analyzes the various attributes of the task and the actual situation of resource allocation. Task characteristics include task type, urgency, importance, etc., and resource allocation reflects the amount of resources currently available for each task in the system. At the same time, by comprehensively considering key factors such as task priority, execution time and resource requirements, the status and requirements of each task in the entire task scheduling system are clarified, providing basic information for subsequent optimization decisions.

[0153] Solution optimization submodule 242: Use optimization algorithms to optimize the execution order and time arrangement of tasks and formulate the best task scheduling solution; use optimization algorithms to intelligently optimize the execution order and time arrangement of tasks. For task scheduling problems, define fitness functions to evaluate the pros and cons of task scheduling solutions:

[0154]

[0155] Where f(x) is the fitness function; α, β, and γ are weight coefficients that are adjusted according to the specific optimization goal; n is the number of tasks; i is an index variable; t i represents the execution time of the i-th task; u i Indicates the amount of resources occupied by the i-th task; U totalIndicates the total resources of the system; p i Indicates the priority of the i-th task.

[0156] Based on the various factors considered above, we search and evaluate among numerous task scheduling schemes to find the optimal scheme that can maximize the utilization of system resources while meeting task requirements, thus ensuring that tasks can be executed efficiently and orderly.

[0157] Scheme verification and assurance submodule 243: Ensure that the task scheduling scheme meets the task requirements while maximizing the utilization of system resources; verify and adjust the formulated task scheduling scheme. On the one hand, ensure that the scheme can meet the specific requirements of each task, such as the execution time requirements and resource requirements of the task; on the other hand, reconfirm whether the scheme can maximize the utilization of system resources to avoid resource waste or obstruction of task execution. If problems are found in the scheme, appropriate adjustments and optimizations will be made to ensure the feasibility and effectiveness of the task scheduling scheme.

[0158] Task execution and monitoring module 25: executes tasks and monitors task progress and resource usage, calculates task execution progress deviation and resource consumption change rate, and adjusts the scheduling plan based on the monitoring results, including the following sub-modules:

[0159] Task execution and monitoring submodule 251: execute tasks according to the task scheduling plan, and monitor the progress of tasks and resource usage in real time; start task execution according to the established task scheduling plan, and obtain the progress of tasks and resource usage in real time through various monitoring means during task execution, providing real-time data support for subsequent evaluation and adjustment.

[0160] The progress deviation calculation submodule 252 calculates the task execution progress deviation and promptly discovers problems in the task execution process; the deviation is calculated by comparing the actual task execution progress with the planned progress, and the progress deviation of the task execution is calculated using the following formula:

[0161]

[0162] Where f(x) represents the task execution progress deviation; n represents the number of different factors that affect the task progress; i is an index variable whose value range is 1 to n; ω i represents the weight of the i-th factor; E i It represents the quantitative value of the actual completion of the i-th factor; P i It represents the quantitative value of the planned completion of the i-th factor.

[0163] If the deviation rate exceeds a certain reasonable range, delays, obstructions and other problems in the task execution process can be discovered in a timely manner.

[0164] Resource consumption monitoring submodule 253: Calculate resource consumption change rate to grasp the dynamic change of resources; by calculating the resource consumption change rate, understand whether the resource consumption is within the expected range, determine the initial usage and current usage of different resources in a specific time period, take CPU resources as an example, record the CPU usage at the beginning of the task as the initial usage, and then obtain the CPU usage at the current moment as the current usage. Use the following formula to calculate the resource consumption change rate:

[0165]

[0166] Where V represents the rate of change of resource consumption; m represents the number of different resource types; j is an index variable; ω j represents the weight of the jth resource type; T1 and T2 represent two different time points, which are used to calculate the consumption change of resources during this period; R j1 represents the usage of the jth resource at time point T1; R j2 Represents the usage of the j-th resource at time point T2.

[0167] Understand resource consumption trends and discover abnormal resource consumption in a timely manner.

[0168] Scheduling scheme adjustment submodule 254: According to the monitoring results, timely adjust the task scheduling scheme to ensure the smooth execution of the task and the rational use of resources; according to the task progress and resource usage provided by the task execution and monitoring submodule, as well as the problems found by the progress deviation calculation module and the resource consumption monitoring module, timely adjust the task scheduling scheme. If it is found that the task progress is lagging behind, reallocate resources or adjust the execution order of tasks to catch up with the progress; if the resources are consumed too quickly, optimize the task execution method or increase the resource supply to ensure that the task can be completed smoothly and the resources are reasonably used.

[0169] Corresponding to the above embodiment, an embodiment of the present invention provides a computer storage medium, including: at least one memory and at least one processor;

[0170] The memory is used to store one or more program instructions;

[0171] A processor, configured to run one or more program instructions to execute a task scheduling method based on a power big data model;

[0172] Corresponding to the above-mentioned embodiments, an embodiment of the present invention provides a computer-readable storage medium, wherein the computer storage medium contains one or more program instructions, and the one or more program instructions are used by a processor to execute a task scheduling method based on a power big data model.

[0173] The embodiments disclosed in the present invention provide a computer-readable storage medium, in which computer program instructions are stored. When the computer program instructions are executed on a computer, the computer executes the above-mentioned task scheduling method based on the power big data model.

[0174] In the embodiment of the present invention, the processor may be an integrated circuit chip having the signal processing capability. The processor may be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0175] The methods, steps and logic block diagrams disclosed in the embodiments of the present invention can be implemented or executed. The general processor can be a microprocessor or the processor can also be any conventional processor. The steps of the method disclosed in the embodiments of the present invention can be directly embodied as a hardware decoding processor for execution, or can be executed by a combination of hardware and software modules in the decoding processor. The software module can be located in a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, or a mature storage medium in the art. The processor reads the information in the storage medium and completes the steps of the above method in combination with its hardware.

[0176] The storage medium may be a memory, which may be, for example, a volatile memory or a nonvolatile memory, or may include both volatile and nonvolatile memory.

[0177] Among them, the non-volatile memory can be a read-only memory (ROM), a programmable ROM (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory.

[0178] The volatile memory may be a random access memory (RAM) which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate synchronous DRAM (DDRSDRAM), enhanced synchronous DRAM (ESDRAM), synchronous link DRAM (SLDRAM), and direct RAM bus DRAM (DRRAM).

[0179] The storage media described in the embodiments of the present invention are intended to include, but are not limited to, these and any other suitable types of memory.

[0180] Those skilled in the art will appreciate that in one or more of the above examples, the functions described in the present invention can be implemented using a combination of hardware and software. When software is used, the corresponding functions can be stored in a computer-readable medium or transmitted as one or more instructions or codes on a computer-readable medium. Computer-readable media include computer storage media and communication media, wherein communication media include any media that facilitates the transmission of computer programs from one place to another. Storage media can be any available media that can be accessed by general or special-purpose computers.

[0181] The specific implementation methods described above further illustrate the objectives, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above description is only a specific implementation method of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions and improvements made on the basis of the technical solution of the present invention should be included in the scope of protection of the present invention.

Claims

1. A task scheduling method based on a power big data model, characterized in that: include: S1. Collect and pre-process power system data using the power big data model, calculate the data integrity index, and integrate high-integrity data; S2. Based on the integrated data, analyze the characteristics of the tasks to be scheduled, calculate the average execution time and expected resource consumption of the tasks, and determine the execution difficulty coefficient of the tasks; S3, evaluate the system resource status, calculate resource utilization and remaining amount, and allocate resources according to the difficulty coefficient of task execution; S4. Optimize the execution order and time arrangement of tasks and formulate task scheduling plans based on task characteristics and resource allocation; S5. Execute tasks and monitor task progress and resource usage, calculate task execution progress deviation and resource consumption change rate, and adjust the scheduling plan based on the monitoring results.

2. The task scheduling method based on the power big data model according to claim 1 is characterized in that: The power big data model is used to collect and preprocess power system data, calculate the data integrity index, and integrate high-integrity data, including the following sub-steps: Use the power big data model to collect various data in the power system and pre-process the collected data; Calculate the data integrity index and select high-integrity data for integration.

3. The task scheduling method based on the power big data model according to claim 1 is characterized in that: Based on the integrated data, analyze the characteristics of the tasks to be scheduled and calculate the average execution time of the tasks, including the following sub-steps: Based on the integrated high-integrity data, in-depth analysis of the characteristics of the tasks to be scheduled; By analyzing and modeling historical task execution data, the average execution time of tasks is calculated to provide a time reference for task scheduling.

4. The task scheduling method based on the power big data model according to claim 3 is characterized in that: Calculate the expected value of resource consumption and determine the difficulty coefficient of task execution, including the following sub-steps: Predict the expected resource consumption of the task based on the characteristics and resource requirements of the task; The average execution time and expected resource consumption of the comprehensive task are used to determine the execution difficulty coefficient of the task, providing a basis for resource allocation.

5. The task scheduling method based on the power big data model according to claim 1 is characterized in that: Evaluate system resource status, calculate resource utilization and remaining amount, and allocate resources according to task execution difficulty coefficient, including the following sub-steps: Monitor and evaluate various resources in the power system in real time and calculate resource utilization; Calculate the remaining resources based on resource utilization and total resources; Allocate resources reasonably according to the difficulty coefficient of task execution to ensure efficient use of resources.

6. The task scheduling method based on the power big data model according to claim 1 is characterized in that: Combined with the task characteristics and resource allocation, optimize the execution order and time arrangement of tasks and formulate a task scheduling plan, including the following sub-steps: Combine the characteristics of the task and the resource allocation, and comprehensively consider factors such as task priority, execution time, and resource requirements; Use optimization algorithms to optimize the execution order and time schedule of tasks and formulate the best task scheduling plan; Ensure that the task scheduling plan maximizes the utilization of system resources while meeting task requirements.

7. The task scheduling method based on the power big data model according to claim 1 is characterized in that: Execute tasks and monitor task progress and resource usage, calculate task execution progress deviation and resource consumption change rate, and adjust the scheduling plan based on the monitoring results, including the following sub-steps: Execute tasks according to the task scheduling plan and monitor the progress of tasks and resource usage in real time; Calculate the task execution progress deviation and promptly identify problems in the task execution process; Calculate the resource consumption change rate and grasp the dynamic changes of resources; According to the monitoring results, timely adjust the task scheduling plan to ensure the smooth execution of tasks and the rational use of resources.

8. The task scheduling system based on the power big data model is characterized by: include: Data acquisition and preprocessing module: Use the power big data model to collect and preprocess power system data, calculate the data integrity index, and integrate high-integrity data; Data analysis module: Based on the integrated data, analyze the characteristics of the tasks to be scheduled, calculate the average execution time and expected resource consumption of the tasks, and determine the execution difficulty coefficient of the tasks; Resource evaluation module: evaluates system resource status, calculates resource utilization and remaining amount, and allocates resources according to the difficulty coefficient of task execution; Task scheduling module: optimizes the execution order and time arrangement of tasks and formulates task scheduling plans based on task characteristics and resource allocation; Task execution and monitoring module: executes tasks and monitors task progress and resource usage, calculates task execution progress deviation and resource consumption change rate, and adjusts the scheduling plan based on the monitoring results.

9. A computer storage medium, characterized in that: include: at least one memory and at least one processor; A memory for storing one or more program instructions; A processor, used to run one or more program instructions to execute the task scheduling method based on the power big data model as described in any one of claims 1-7.

Images