Data Processing Process Scheduling Method and System Based on Improved Jellyfish Search Algorithm

Abstract

The present invention discloses a data processing process scheduling method and system based on an improved jellyfish search algorithm. First, historical measurement data is collected and the computing power demand curve of the measurement data is statistically obtained. Then, based on a sliding window, the computing power demand curve is divided into multiple time windows, so that the computing power demand within each time window fluctuates smoothly. After constructing an optimization problem, iterative solution is carried out, and the obtained optimal solution represents the process distribution vector within each time window, ensuring that the computing resources scheduled within each time window meet the data processing delay requirements and reducing resource consumption. Moreover, the difference in process distribution between adjacent time windows is constrained to be less than a threshold, avoiding large fluctuations in process distribution between adjacent time windows and improving the scheduling efficiency. The entire process adaptively adjusts the size of each time window and the number of processes enabled in each time window, significantly reducing the subjective human influence and additional workload while ensuring the data processing efficiency.

Description

Technical Field

[0001] The present invention relates to the technical field of parallel processing of power measurement data, and in particular, to a data processing process scheduling method and system, an electronic device, and a computer-readable storage medium based on an improved jellyfish search algorithm. Background Art

[0002] The operation stability of the power system has a crucial impact on the economic operation of the entire society and people's lives. The data sharing platform can quickly and accurately reflect the real-time state of the power system and the operation status of each device by aggregating signal data such as voltage, current, and power obtained by various sensors and combining big data processing technology. In the operation of the power grid, with the rapid development of new energy sources such as distributed photovoltaic power generation and wind power generation and the continuous improvement and optimization of power grid infrastructure, the grid connection scale of the power grid is getting larger and larger, and more and more terminals are arranged in various links of the power system to ensure the safe and effective operation of the power system. Although the increase in the number of terminals and more fine-grained data collection provide better services for users, the steep increase in the amount of real-time data reported by the terminals brings new problems to the power system. Since the value of real-time data will quickly disappear over time, how to quickly process real-time data is one of the new problems faced by the power system, and data parallel processing can effectively accelerate the processing speed of measurement data and ensure the real-time and accuracy of the measurement data file association.

[0003] However, the current parallel processing method is not designed according to the characteristics of the measurement data reported by the device. Workers often only take the number of specified processes based on tests or experience to ensure that the data is processed in a timely manner. The measurement data has the following characteristics: one is unbounded streaming data, and the other is that the devices corresponding to the measurement data are disordered. The data reporting frequencies of different devices are inconsistent and the data arrival times at the system are inconsistent. Therefore, the number of processes specified manually does not match the amount of measurement data uploaded in real time. An excessive number of processes will consume too much system resources, while a too small number of processes will cause the real-time data to not be processed in a timely manner because workers cannot make timely adjustments, resulting in data backlog and reducing the value of real-time data. Therefore, the existing data parallel processing method still cannot meet the real-time requirements of measurement data, resulting in a large gap in the data processing load of different processes. As a result, there is surplus computing power in some processes, while some processes cannot process the data within the tolerable time, ultimately affecting the overall real-time performance of data processing, and the need for manual setting of the number of processes brings additional workload to data processing. Therefore, there is an urgent need to design a process adaptive scheduling method based on the data characteristics of measurement data, so that different amounts of measurement data can match the corresponding computing resources, which can effectively improve the processing efficiency of measurement data while avoiding excessive additional resource consumption. Summary of the Invention

[0004] The present invention provides a data processing process scheduling method and system, an electronic device, and a computer-readable storage medium based on an improved jellyfish search algorithm to solve the technical problem that the existing data parallel processing method cannot meet the real-time requirements of measurement data processing.

[0005] According to one aspect of the present invention, there is provided a data processing process scheduling method based on an improved jellyfish search algorithm, including the following:

[0006] Collect historical measurement data and statistically obtain the computing power demand curve of the measurement data;

[0007] Based on a sliding window, divide the computing power demand curve into multiple time windows, and after division, the computing power demand within each time window fluctuates smoothly.

[0008] Construct an optimization problem with minimizing resource overhead as the optimization objective, the processing delay of each time window being lower than the tolerance delay and the number of processes allowed in each host as mandatory constraint conditions, and the difference in process distribution between adjacent time windows being less than a threshold as a non-mandatory constraint condition;

[0009] Iteratively solve the optimization problem, and the obtained optimal solution represents the process distribution vector within each time window;

[0010] Based on the process distribution vector within each time window, schedule the number of processes on each host within the corresponding time period.

[0011] Further, the process of collecting historical measurement data and statistically obtaining the computing power demand curve of the measurement data is specifically as follows:

[0012] Collect historical measurement data for multiple days, divide a day into several moments, then classify and statistically obtain the data volume of each type of measurement data at the same moment in multiple days, calculate the upper limit value of the data volume of each type of measurement data at each moment based on the three-sigma criterion, and calculate the computing power demand of all types of measurement data at each moment, so as to obtain the computing power demand curve of the historical measurement data.

[0013] Further, the process of dividing the computing power demand curve into multiple time windows based on a sliding window is specifically as follows:

[0014] Set the size of the sliding window, and calculate the average computing power demand within the sliding window at initialization and use it as the initial value; where, at initialization, the coverage range of the current time window is the coverage range of the current sliding window;

[0015] Use a sliding window to slide non-overlappingly on the computing power demand curve and calculate the new value of the average computing power demand within the sliding window after the sliding. Based on the new value of the average computing power demand and the initial value, determine whether the time window division condition is met. If so, the time range covered by the sliding window after the sliding is included in the current time window. If not, proceed to the next step.

[0016] Divide the sliding window after sliding based on the binary search algorithm, find the time point coordinates that meet the time window division conditions, and then divide the sliding time window into a left window and a right window based on the time point coordinates. The time range covered by the left window is included in the current time window, and the average computing power requirement in the right window is calculated and used as the updated value of the average computing power requirement in the sliding window. The time range covered by the right window is set as the initial coverage range of the next time window.

[0017] The computing power demand curve is traversed by sliding the sliding window, thereby dividing the computing power demand curve into multiple time windows. After the division, the computing power demand in each time window fluctuates smoothly.

[0018] Furthermore, if the rate of change of the new value of the mean computing power requirement is less than a preset threshold, or the time range covered by the current time window is less than the preset threshold, the time range covered by the sliding window after sliding is included in the current time window.

[0019] Furthermore, if the time range covered by the current time window is less than the preset threshold, the computing power demand variance in the sliding window after sliding is calculated. If the computing power demand variance is less than the preset threshold, the computing power demand mean in the sliding window is updated from the initial value to the new value, otherwise no update is performed.

[0020] Furthermore, the optimization problem is expressed as:

[0021]

[0022] Among them, e h T represents the resource overhead per unit time of data processing in host h. i represents the time required for the system to process data in the i-th time window, represents the number of processes enabled by host h in the i-th time window, H represents the number of hosts in the system, W represents the number of time windows, represents the number of processes enabled by host h in the i+1th time window, |w i | represents the length of the i-th time window, represents the delay tolerance, P hLet \(h_{max}\) denote the maximum number of parallel processes allowed for the \(h\)-th host, and \(\theta\) denote the threshold for the difference in process distribution between adjacent time windows. Constraints \(C1\) and \(C2\) are mandatory constraints, and constraint \(C3\) is a non-mandatory constraint.

[0023] Furthermore, the process of iteratively solving the optimization problem is specifically as follows:

[0024] Initialize the jellyfish swarm. Among them, the position of jellyfish \(i\) in the jellyfish swarm is represented as a matrix \(mat\) of dimension \(H\times W\). i Each column in the matrix represents the process distribution vector within a time window.

[0025] Evaluate the fitness of the jellyfish in the jellyfish swarm and select the current optimal jellyfish position.

[0026] Update the position of each jellyfish in the jellyfish swarm. Among them, when the movement mode of the jellyfish is the active movement mode, the standard jellyfish search algorithm formula is used for position update. When the movement mode of the jellyfish is the passive movement mode, the position is updated based on the following formula:

[0027] mat i [x,dire i =γ×mat i [x,dire i +(1 - γ)×fs(P(x=ν))

[0028] Among them, \(mat i [x,dire i represents the value at the coordinate \((x,dire i ) in the position matrix of jellyfish \(i\). The left side of the formula is the updated value, and the right side is the value before update. \(\gamma\) represents the weight parameter, \(\gamma\in[0,1]\). The function \(fs(P(x = \nu))\) represents a value randomly selected from the set \(\{0,1,2,\cdots,P h \}\) based on the probability model \(P(x = \nu)\). \(dire i represents the one-dimensional search direction of jellyfish \(i\) during position update.

[0029] Perform rounding processing on the position elements of the updated jellyfish.

[0030] Continuously repeat the above process until the maximum number of iterations is reached, and then obtain the final optimal jellyfish position. This optimal jellyfish position represents the process distribution vector within each time window.

[0031] In addition, the present invention also provides a data processing process scheduling system based on an improved jellyfish search algorithm. Using the method described above, it includes:

[0032] A data characteristic analysis module, which is used to collect historical measurement data and statistically obtain the computing power demand curve of the measurement data.

[0033] A time window division module, configured to divide the computing power demand curve into multiple time windows based on a sliding window, and the computing power demand within each time window after division fluctuates smoothly.

[0034] An optimization problem construction module, configured to construct an optimization problem with minimizing resource overhead as the optimization goal, the processing delay of each time window being lower than the tolerance delay and the number of processes allowed in each host as mandatory constraint conditions, and the difference in process distribution between adjacent time windows being less than a threshold as a non-mandatory constraint condition.

[0035] An iterative solution module, configured to iteratively solve the optimization problem, and the obtained optimal solution represents the process distribution vector within each time window.

[0036] A process scheduling module, configured to schedule the number of processes on each host within the corresponding time period based on the process distribution vector within each time window.

[0037] In addition, the present invention further provides an electronic device, including a processor and a memory. A computer program is stored in the memory, and the processor is configured to execute the steps of the method described above by calling the computer program stored in the memory.

[0038] In addition, the present invention further provides a computer-readable storage medium, configured to store a computer program for scheduling data processing processes based on an improved jellyfish search algorithm. The computer program executes the steps of the method described above when running on a computer.

[0039] The present invention has the following effects:

[0040] The data processing process scheduling method based on the improved jellyfish search algorithm of the present invention first collects historical measurement data and statistically obtains the computing power demand curve of the measurement data. Since the device data acquisition frequency of the sensing layer is usually fixed, the difference between the computing power demand curve obtained based on historical data analysis and the real-time computing power demand curve is very small. Especially when the scale of historical data is large, the difference between the two will be even smaller. Then, based on a sliding window, the computing power demand curve is divided into multiple time windows, making the computing power demand within each time window fluctuate smoothly. On the one hand, this can avoid frequent process switching, and on the other hand, it ensures that the amount of data processed within the same time window is basically stable. Next, with minimizing resource overhead as the optimization goal, the processing delay of each time window being lower than the tolerance delay and the number of processes allowed in each host as mandatory constraint conditions, and the difference in process distribution between adjacent time windows being less than a threshold as a non-mandatory constraint condition, an optimization problem is constructed and iteratively solved. The obtained optimal solution represents the process distribution vector within each time window, ensuring that the computing resources scheduled within each time window meet the delay requirements of data processing and reducing resource consumption. Moreover, since the difference in process distribution between adjacent time windows is restricted to be less than the threshold, large fluctuations in process distribution between adjacent time windows can be avoided, greatly improving the scheduling efficiency. In addition, the entire process does not require manual specification of the size of each time window and the number of processes enabled in each time window, but adaptively regulates the size of each time window and the number of processes enabled in each time window based on the temporal distribution characteristics of the measurement data, significantly reducing the subjective human influence and additional workload while ensuring data processing efficiency.

[0041] In addition, the data processing process scheduling system based on the improved jellyfish search algorithm of the present invention also has the above advantages.

[0042] In addition to the purposes, features, and advantages described above, the present invention has other purposes, features, and advantages. The following will refer to the drawings for a further detailed description of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] The drawings forming a part of this application are used to provide a further understanding of the present invention. The schematic embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0044] Figure 1 is a schematic flowchart of the data processing process scheduling method based on the improved jellyfish search algorithm of the preferred embodiment of the present invention.

[0045] Figure 2 is a schematic diagram of the principle of the data processing process scheduling method based on the improved jellyfish search algorithm of the preferred embodiment of the present invention.

[0046] Figure 3 isFigure 1 Schematic diagram of the sub - process in step S2

[0047] Figure 4 Schematic diagram for comparing the time - window division results of different computing - power demand curves when λ1 = 0.1 in the preferred embodiment of the present invention

[0048] Figure 5 Schematic diagram for comparing the time - window division results of the same computing - power demand curve when λ1 = 0.1 and λ1 = 0.5 respectively in the preferred embodiment of the present invention

[0049] Figure 6 Is Figure 1 Schematic diagram of the sub - process in step S4

[0050] Figure 7 Schematic diagram of the module structure of a data - processing process scheduling system based on an improved jellyfish search algorithm in another embodiment of the present invention Specific embodiments

[0051] The embodiments of the present invention will be described in detail below with reference to the accompanying drawings. However, the present invention can be implemented in many different ways defined and covered by the following.

[0052] It can be understood that, as Figure 1 And Figure 2 shown, the preferred embodiment of the present invention provides a data - processing process scheduling method based on an improved jellyfish search algorithm, including the following:[[]]

[0053] Step S1: Collect historical measurement data and statistically obtain the computing - power demand curve of the measurement data;

[0054] Step S2: Divide the computing - power demand curve into multiple time windows based on a sliding window, and the computing - power demand within each time window after division fluctuates smoothly;

[0055] Step S3: Construct an optimization problem with minimizing resource overhead as the optimization goal, the processing delay of each time window being lower than the tolerance delay and the number of processes allowed in each host as mandatory constraint conditions, and the difference in process distribution between adjacent time windows being less than a threshold as a non - mandatory constraint condition;

[0056] Step S4: Iteratively solve the optimization problem, and the obtained optimal solution represents the process distribution vector within each time window;

[0057] Step S5: Schedule the number of processes on each host within the corresponding time period based on the process distribution vector within each time window.

[0058] It can be understood that for the data processing process scheduling method based on the improved jellyfish search algorithm in this embodiment, historical measurement data is first collected and the computing power demand curve of the measurement data is statistically obtained. Since the data acquisition frequency of the devices in the sensing layer is usually fixed, the difference between the computing power demand curve obtained based on historical data analysis and the real-time computing power demand curve is very small. Especially when the scale of the historical data volume is large, the difference between the two will be even smaller. Then, the computing power demand curve is divided into multiple time windows based on a sliding window, so that the computing power demand within each time window fluctuates smoothly. On the one hand, it can avoid frequent process switching, and on the other hand, it ensures that the data processing volume within the same time window is basically stable. Next, with minimizing resource overhead as the optimization goal, the processing delay of each time window being lower than the tolerance delay and the number of processes allowed in each host as mandatory constraint conditions, and the difference in process distribution between adjacent time windows being less than a threshold as a non-mandatory constraint condition, an optimization problem is constructed and iteratively solved. The optimal solution obtained represents the process distribution vector within each time window, ensuring that the computing resources scheduled within each time window meet the delay requirements of data processing and reducing resource consumption. Moreover, since the difference in process distribution between adjacent time windows is constrained to be less than a threshold, large fluctuations in process distribution between adjacent time windows can be avoided, greatly improving the scheduling efficiency. In addition, throughout the process, there is no need to manually specify the size of each time window and the number of processes enabled in each time window, but rather adaptively adjust the size of each time window and the number of processes enabled in each time window based on the temporal distribution characteristics of the measurement data, significantly reducing the subjective human influence and extra workload while ensuring data processing efficiency.

[0059] It can be understood that in the present invention, a real-time measurement center collects and processes the measurement data sensed by a large number of terminals. Assume that the real-time measurement center (i.e., the system) has H hosts for data processing, and the maximum number of parallel processes allowed for the h-th host is P h , then under this constraint, assuming that the computing power of processes under the same host is the same, let C h represent the computing power of each process in host h. In addition, assume that the measurement data is divided into K categories, and the computing power required for the k-th category of measurement data is c k . In addition, assume that each host has a load balancing function, that is, the host can evenly distribute the load to each process. Therefore, the present invention does not consider the load balancing problem and only considers how to optimize the number of processes based on the characteristics of the measurement data to quickly process the real-time measurement data.

[0060] It can be understood that in the step S1, the process of collecting historical measurement data and statistically obtaining the computing power demand curve of the measurement data is specifically as follows:

[0061] Collect historical measurement data for multiple days, divide a day into several moments, then classify and count the historical measurement data to obtain the data volume of each type of measurement data at the same moment in multiple days. Calculate the upper limit value of the data volume of each type of measurement data at each moment based on the three-sigma criterion, and calculate the computing power requirements of all types of measurement data at each moment, so as to obtain the computing power requirement curve of the historical measurement data.

[0062] Specifically, collect a sufficient amount of historical measurement data for several days, divide a day into several moments with a sufficiently fine granularity, then classify the measurement data, and count the number of data of each type of measurement data at the same moment in several days. Assume that in several days, the data volume of the k-th type of measurement data at moment t follows a normal distribution, and calculate the mean value and variance Then take as the statistical eigenvalue. Based on the three-sigma criterion, it can be known that the probability that the data volume of the k-th type of measurement data at moment t is less than is less than 0.27%, which means characterizes the upper limit of the data volume of the k-th type of measurement data at moment t and covers almost all scenarios. Thus, comprehensively calculate the computing power requirement V t of all measurement data at moment t as:

[0063]

[0064] Repeat the above steps to calculate the computing power requirements of the measurement data at each moment in a day, so as to obtain the computing power requirement curve of the measurement data, that is, the distribution curve of the computing power requirement of the measurement data changing with time.

[0065] It can be understood that since the device data acquisition frequency of the sensing layer is usually fixed, the difference between the computing power requirement curve obtained based on historical data analysis and the real-time computing power requirement curve is very small. Especially when the scale of historical data volume is large, the difference between the two is even smaller. Therefore, the computing power requirement curve obtained by analyzing the massive historical measurement data in this invention can accurately reflect the real-time computing power requirement curve of the measurement data.

[0066] It can be understood that as Figure 3 shown, in step S2, the process of dividing the computing power requirement curve into multiple time windows based on the sliding window is specifically as follows:

[0067] Step S21: Set the size of the sliding window, and calculate the average value of the computing power requirements within the sliding window at initialization, and use it as the initial value; where, at initialization, the coverage range of the current time window is the coverage range of the current sliding window;

[0068] Step S22: Use a sliding window to slide non - overlapping on the computing power demand curve, calculate the new value of the average computing power demand within the sliding window after sliding, and determine whether the time window division condition is met based on the new value and the initial value of the average computing power demand. If it is met, the time range covered by the sliding window after sliding is included in the current time window; if not, proceed to the next step;

[0069] Step S23: Based on the binary search algorithm, divide the sliding window after sliding, find the time - point coordinates that meet the time window division condition, and divide the sliding window after sliding into a left window and a right window based on these time - point coordinates. Include the time range covered by the left window in the current time window, calculate the average computing power demand within the right window, use it as the updated value of the average computing power demand within the sliding window, and set the time range covered by the right window as the initial coverage range of the next time window. It can be understood that in subsequent traversals, the next time window will repeat the above process as the current time window;

[0070] Step S24: Use the sliding window to slide and traverse the computing power demand curve, thereby dividing the computing power demand curve into multiple time windows. After division, the computing power demand within each time window fluctuates smoothly.

[0071] Specifically, set the length of the sliding window ε to be fixed as |ε|, and calculate the average computing power demand within the sliding window at initialization Use it as the initial value. It can be understood that the time range covered by the sliding window at initialization is the initial range of the current time window. Optionally, the set point = {0} can also be initialized as the window division pointer set. Then, the left and right boundaries of the time range covered by the current time window at initialization are left = 0 and right = |ε| - 1 respectively.

[0072] Then, use the sliding window to slide non - overlapping on the computing power demand curve, that is, the sliding step size is equal to the length of the sliding window, and calculate the new value of the average computing power demand within the sliding window after sliding And based on the new value And the initial value Determine whether the time window division condition is met. If it is met, the time range covered by the sliding window after sliding is included in the current time window; if not, proceed to the next step.

[0073] Specifically, if the change rate of the new value of the average computing power demand is less than the preset threshold, that is It means that the computing power requirement change before and after sliding is small. Then, the time range covered by the sliding window after sliding is also included in the current time window. At this time, the right boundary of the current time window is updated to right = right + |ε|. Or, during the sliding process, if the coverage range of the current time window is less than the preset threshold, that is, the difference between the right boundary right and the left boundary left of the time window is less than It means that the scale of the time window is too small. To avoid frequent switching of processes in a short time, the time range covered by the sliding window after sliding is still included in the current time window. At this time, the right boundary of the time window is updated to right = right + |ε|. Therefore, as the sliding window slides, the coverage range of the current time window will increase. Optionally, if the coverage range of the current time window is less than the preset threshold and the variance of the computing power requirement within the sliding window after sliding is less than the preset threshold λ2, it means that the computing power requirement within the sliding window after sliding fluctuates smoothly. Then, the average value of the computing power requirement within the sliding window is updated from the initial value to a new value Otherwise, no update is performed and it remains the initial value If none of the above conditions are met, then proceed to step S23 for processing. It can be understood that in this invention, the sliding window is used to divide the time window. For example, if the size of the sliding window is 6, then during initialization, the current time window is 0 - 6; the coverage range after the sliding window slides once is 7 - 12. If the change rate of the new value is less than λ1, then 7 - 12 is directly included in the current time window, and the coverage range of the current time window is 0 - 12; after sliding again, the coverage range of the sliding window is 13 - 18. If the change rate of the new value is greater than the threshold λ1, but since the minimum length of the set time window is 20, and at this time 12 is obviously less than 20, so the sliding window is also directly added to the current time window, that is, the coverage range of the current time window is 0 - 18.

[0074] It can be understood that in step S23, the sliding window after sliding is divided based on the binary search algorithm to find the one that meets The time point coordinate L within the most recent sliding window. Then, based on this time point coordinate L, the slid time window is divided into a left window and a right window. The split left window is added to the current time window, and the right boundary of the current time window can be updated to right + L, and the window division pointer set is updated to point = point ∪ {(right + L)}. The coverage range of the right window is the initial coverage range of the next time window, that is, the left and right boundaries of the next time window are respectively: left = right + L, right = left + |ε| - L, and the average computing power demand within the right window is calculated and used as the updated value of the average computing power demand within the sliding window, that is, is updated to the average computing power demand within the split right window, that is, the right window at this time forms a new current time window.

[0075] It can be understood that in step S25, the sliding window is used to slide and traverse the computing power demand curve, and the end node is added to the point set. After continuous iteration, a day is divided into W time windows, and let w i represent the i-th time window, and |w i | represents the size of the time window, with the unit of seconds. Then the computing power demand within the i-th time window can be expressed as:

[0076]

[0077] where α i represents the computing power demand of the measurement data within the i-th time window.

[0078] It can be understood that the present invention proposes a time window division algorithm based on a sliding window, which can divide a day into multiple time windows based on the statistical characteristics of measurement data, making the computing power demand within each time window fluctuate smoothly. On the one hand, it can avoid frequent process switching, and on the other hand, it can ensure that the data processing volume within the same time window is basically stable. Among them, the result of time window division of the computing power demand curve using the time window division algorithm of the present invention is as Figures 4 to 5 shown.

[0079] Optionally, before step S21, the following content is further included:

[0080] A sliding window with a window size of ε and a step size of 1 is used to smooth the computing power demand curve, that is, the average computing power demand within the sliding window is calculated each time it slides, and then an average computing power demand curve is generated based on the average computing power demand calculated each time it slides.

[0081] It can be understood that in step S3, after dividing the computing power demand curve into time windows, it is necessary to determine the process distribution within each time window to ensure that the measurement data is processed in a timely manner. Let represent the process distribution vector of the system in the i-th time window, where represents the number of processes enabled by host h in the i-th time window, that is, it is necessary to solve the above process distribution vector. Since the host has a load balancing function, that is, the computing power demand corresponding to the host is evenly distributed to multiple processes, so that the time spent by each process in processing data is basically the same. Therefore, in the i-th time window, the time required for the system to process data can be expressed as:

[0082]

[0083] where C h represents the computing power of each process in host h, and α i represents the computing power demand of the measurement data in the i-th time window. Thus, the optimization objective is defined as minimizing the resource overhead, which can be expressed as:

[0084]

[0085] where e h represents the resource overhead per unit time for the process in host h to process data. In addition, the constraint conditions of the above optimization problem are:

[0086]

[0087] where H represents the number of hosts in the system, W represents the number of time windows, represents the number of processes enabled by host h in the i + 1-th time window, |w i | represents the length of the i-th time window, represents the tolerated delay, P h represents the maximum allowable number of parallel processes of the h-th host, and θ represents the threshold of the difference in process distribution between adjacent time windows. Constraints C1 and C2 are mandatory constraints, and constraint C3 is a non-mandatory constraint. Among them, constraint C1 means that the time delay processed in each time window should be lower than the tolerated delay Constraint C2 represents the value range of the number of processes in host h, and constraint C3 means that the difference in the process distribution vectors between adjacent time windows should be less than the threshold θ. The smaller the difference, the better the solution.

[0088] Therefore, in step S3, the optimization problem is expressed as:

[0089]

[0090] C1:

[0091] C2:

[0092] C3:

[0093] It can be understood that in the step S4, an improved jellyfish search algorithm is used to iteratively solve the above optimization problem. Specifically, as Figure 6 shown, the process of iteratively solving the optimization problem is specifically as follows:

[0094] Step S41: Initialize the jellyfish swarm. Among them, the position of jellyfish i in the jellyfish swarm is represented as a matrix mat of H×W dimensions i , and each column in the matrix represents the process distribution vector within a time window;

[0095] Step S42: Evaluate the fitness of the jellyfish in the jellyfish swarm and select the current optimal jellyfish position;

[0096] Step S43: Update the position of each jellyfish in the jellyfish swarm. Among them, when the movement mode of the jellyfish is the active movement mode, the standard jellyfish search algorithm formula is used for position update, and when the movement mode of the jellyfish is the passive movement mode, the position is updated based on the following formula:

[0097] mat i [x,dire i =γ×mat i [x,dire i +(1 - γ)×fs(P(x=ν))

[0098] where, mat i [x,dire i represents the value at the coordinate (x,dire i ) in the position matrix of jellyfish i. The left side of the formula is the updated value, and the right side is the value before update. γ represents the weight parameter, γ∈[0,1]. The function fs(P(x=ν)) represents a value randomly selected from the set {0,1,2,...,P h} based on the probability model P(x=ν). dire i represents the one-dimensional search direction of jellyfish i during position update;

[0099] Step S44: Perform rounding processing on the position elements of the updated jellyfish;

[0100] Step S45: Continuously repeat the above steps S42 to S44 until the maximum number of iterations is reached, and then obtain the optimal jellyfish position. Among them, the optimal jellyfish position represents the process distribution vector within each time window.

[0101] Specifically, first, initialize the population size to N p of the jellyfish swarm, where the position of jellyfish i is represented as a matrix mat of dimension H×W i , and each column in the matrix represents the process distribution vector in a time window. Based on the mathematical model of the above optimization problem, it can be seen that the value range of each element in the matrix is {0, 1, 2,..., P h}, so the initialization method of the Logistic chaotic mapping of the standard jellyfish search algorithm can be directly used to initialize the jellyfish positions, and the current iteration number is represented as τ = 0. The initialization process of the Logistic chaotic mapping belongs to the prior art and will not be elaborated here.

[0102] Then, evaluate the fitness of the jellyfish in the jellyfish swarm and select the current optimal jellyfish position. Since there is no coding, the position matrix mat of the jellyfish i is substituted into the definition formula of the optimization objective f in the above step S3 to obtain the fitness of this jellyfish. Since the elements in the position matrix mat i naturally satisfy the constraint C2 and the constraint C3 is a non-mandatory constraint, it is defined that as long as the position of the jellyfish satisfies the constraint C1, the solution represented by the position of this jellyfish is a feasible solution. And select the current optimal position X in the jellyfish swarm according to the following definition * :

[0103] An infeasible solution is always inferior to a feasible solution. For feasible solution jellyfish i and feasible solution jellyfish j: (1) If the fitness of the position of jellyfish i is greater than the fitness of the position of jellyfish j, then jellyfish i is better than jellyfish j; (2) When the fitness of the position of jellyfish i is equal to the fitness of the position of jellyfish j, if the value of the position of jellyfish i corresponding to the constraint C3 is less than the value of the position of jellyfish j corresponding to the constraint C3, then jellyfish i is better than jellyfish j; (3) If neither of the above is satisfied, then jellyfish i and jellyfish j are equivalent.

[0104] Next, update the position of each jellyfish in the jellyfish swarm. First, randomly generate a random number rc τ , if the random number rc τ is not less than 0.5, it is considered that the jellyfish moves with the ocean current, and the position of the jellyfish is updated based on the following formula:

[0105] mat i [x, y] = mat i [x, y] + rand1 × (X * [x, y] - β × rand2 × μ[x, y]) where mat i [x, y] represents the value at the coordinate (x, y) in the position matrix of jellyfish i. The left side of the equation represents the updated value, and the right side represents the value before update. X *[x, y] represents the value at the coordinate (x, y) in the optimal position matrix of jellyfish i, and μ[x, y] represents the value at the coordinate (x, y) in the mean matrix of jellyfish i. rand1 and rand2 represent random numbers from 0 to 1, and β represents the distribution coefficient, which is a constant.

[0106] If the random number rc τ is less than 0.5, it is considered that the jellyfish flows along with the individuals around it or the jellyfish group. At this time, another random number rand3 is generated. If rand3 > 1 - rc τ , the movement pattern of the jellyfish is type A, that is, the passive movement pattern. If rand3 ≤ 1 - rc τ , the movement pattern of the jellyfish is type B, that is, the active movement pattern. In the standard jellyfish search algorithm, type A movement is a random movement in the entire search space. Although the blind random search will increase the diversity of the population, it will slow down the convergence speed of the algorithm. Therefore, in this invention, the search algorithm for type A movement is improved, while type B movement adopts the method of the standard jellyfish search algorithm, and the search algorithm formula for type B movement will not be elaborated here. Among them, the specific improvement content of the type A movement search algorithm is as follows:

[0107] First, record the historical optimal position and historical average position of the jellyfish. For example, let the historical optimal position of jellyfish i be represented as The historical average position is The variance of the historical position is According to the principle of normal distribution, the numbers closer to the mean are more likely to be selected. This means that the space closer to the mean is more likely to be searched. Therefore, in the subsequent search, the areas that have not been searched should be focused on, that is, the areas far from the mean. Therefore, based on the normal distribution, a probability function model is established:

[0108]

[0109] Among them, represents the probability density function of the normal distribution with a mean of and a variance of . Obviously, the value of the area closer to the mean is smaller, and the value of the part farther from the mean is larger. Then, substitute v ∈ {0, 1, 2,..., P h} into the probability function and perform normalization to obtain the probability P(x = v) of taking v from {0, 1, 2,..., P h} as:

[0110]

[0111] Secondly, in this invention, the position matrix mat of jellyfish ii It can be regarded as a vector of W dimensions, and the value of each dimension is a vector of dimension H. Therefore, the search space has W dimensions. If a random search is performed on all dimensions, the computational complexity is relatively high, and the optimal solution cannot be quickly found. Therefore, the present invention proposes a single-dimension-based search method in combination with the above probability model P(x = v), which is specifically as follows:

[0112] Calculate among the W dimensions and the dimension closest to . Take the vector of the w-th dimension of , denoted as a1, and take the vector of the w-th dimension of , denoted as a2, and measure the similarity between a1 and a2 through cosine similarity. After calculating the similarity for all W dimensions, randomly select a dimension as the search direction based on the similarity, denoted as dire i . The principle of this single-dimension search method is that the more similar the average position is to the optimal position, the higher the possibility of improving the superiority of the jellyfish position by adjusting this dimension, which is conducive to achieving rapid convergence of the iteration.

[0113] Therefore, combining the above two aspects, the calculation formula of the improved type A motion search algorithm in the present invention is:

[0114] mat i [x, dire i = γ × mat i [x, dire i + (1 - γ) × fs(P(x = ν))

[0115] where mat i [x, dire i represents the value of the coordinate (x, dire i ) in the position matrix of jellyfish i. The left side of the equation is the updated value, and the right side is the value before update. γ represents the weight parameter, which is a constant, γ ∈ [0, 1]. The function fs(P(x = ν)) represents a value randomly selected from the set {0, 1, 2,..., P h} based on the probability model P(x = ν), and dire i represents the single-dimension search direction of jellyfish i during position update.

[0116] Then, regularize the position elements of the updated jellyfish. Among them, each element in the new position undergoes the following two steps: First is integerization, that is, mat i [x, y] = int(mat i [x, y]); second is boundary processing. If mat i [x, y] exceeds the upper bound P h , then mati [x, y] = mat i [x, y] - P h , if mat i [x, y] is lower than the lower bound 0, then mat i [x, y] = mat i [x, y] + P h , if it is still out of bounds after one boundary processing, repeat the above operation until the value satisfies the boundary range. Additionally, if there is a time window in the solution represented by the position of the jellyfish that does not satisfy the constraint C1, then sequentially increase the number of processes in the host with the least number of processes, with a step size of 1, until the time window satisfies the constraint C1, or all elements reach the upper bound of the value.

[0117] Then, continuously repeat the above steps S42 to S44 until the maximum number of iterations is reached, and then return the current optimal position X of the jellyfish * , and the optimal position X * The solution represented by is the process distribution vector within each time window.

[0118] It can be understood that in the step S5, the main process will schedule the number of processes on each host within the corresponding time period based on the process distribution vector within each time window, ensuring that the computing resources scheduled within each time window meet the delay requirements of data processing and reducing resource consumption. When the current time window is about to end, the main process will initiate a process transition operation and schedule the process distribution of each host based on the process distribution vector of the next time window.

[0119] In addition, as Figure 7 shown, another embodiment of the present invention also provides a data processing process scheduling system based on an improved jellyfish search algorithm, preferably adopting the method as described above, including:

[0120] A data characteristic analysis module, used to collect historical measurement data and statistically obtain the computing power demand curve of the measurement data;

[0121] A time window division module, used to divide the computing power demand curve into multiple time windows based on a sliding window, and the computing power demand within each time window after division fluctuates smoothly;

[0122] An optimization problem construction module, used to construct an optimization problem with minimizing resource overhead as the optimization goal, with the processing delay of each time window being lower than the tolerance delay and the number of processes allowed in each host as mandatory constraint conditions, and with the difference in process distribution between adjacent time windows being less than a threshold as a non-mandatory constraint condition;

[0123] An iterative solution module, used to iteratively solve the optimization problem, and the obtained optimal solution represents the process distribution vector within each time window;

[0124] A process scheduling module, which is used to schedule the number of processes on each host within the corresponding time period based on the process distribution vector within each time window.

[0125] It can be understood that for the data processing process scheduling system based on the improved jellyfish search algorithm in this embodiment, historical measurement data is first collected and the computing power demand curve of the measurement data is obtained through statistics. Since the data collection frequency of the devices in the sensing layer is usually fixed, the difference between the computing power demand curve obtained based on historical data analysis and the real-time computing power demand curve is very small. Especially when the scale of historical data is large, the difference between the two will be even smaller. Then, the computing power demand curve is divided into multiple time windows based on a sliding window, so that the computing power demand within each time window fluctuates smoothly. On the one hand, it can avoid frequent process switching, and on the other hand, it can ensure that the data processing volume within the same time window is basically stable. Next, with minimizing resource overhead as the optimization goal, the processing delay of each time window being lower than the tolerance delay and the number of processes allowed in each host as mandatory constraint conditions, and the difference in process distribution between adjacent time windows being less than a threshold as a non-mandatory constraint condition, an optimization problem is constructed and iteratively solved. The optimal solution obtained represents the process distribution vector within each time window, ensuring that the computing resources scheduled within each time window meet the delay requirements of data processing and reducing resource consumption. Moreover, since the difference in process distribution between adjacent time windows is restricted to be less than a threshold, large fluctuations in process distribution between adjacent time windows can be avoided, greatly improving the scheduling efficiency. In addition, the size of each time window and the number of processes enabled in each time window do not need to be specified manually throughout the process. Instead, the size of each time window and the number of processes enabled in each time window are adaptively regulated based on the temporal distribution characteristics of the measurement data, significantly reducing the subjective human influence and additional workload while ensuring data processing efficiency.

[0126] It can be understood that each module in this system embodiment corresponds to each step in the above method embodiment. Therefore, the specific working principles of each module will not be elaborated here, and reference can be made to each step in the above method embodiment.

[0127] In addition, another embodiment of the present invention further provides an electronic device, including a processor and a memory. A computer program is stored in the memory, and the processor is used to execute the steps of the above-mentioned method by calling the computer program stored in the memory.

[0128] In addition, another embodiment of the present invention further provides a computer-readable storage medium, which is used to store a computer program for data processing process scheduling based on the improved jellyfish search algorithm. When the computer program runs on a computer, it executes the steps of the above-mentioned method.

[0129] The forms of common computer-readable storage media include: floppy disks, flexible disks, hard disks, magnetic tapes, any other magnetic media, CD-ROMs, any other optical media, punch cards, paper tapes, any other physical media with a pattern of holes, random access memories (RAMs), programmable read-only memories (PROMs), erasable programmable read-only memories (EPROMs), flash erasable programmable read-only memories (FLASH-EPROMs), any other memory chips or cartridges, or any other media readable by a computer. Instructions can further be transmitted or received by a transmission medium. The term transmission medium can include any tangible or intangible medium that can be used to store, encode, or carry instructions for execution by a machine, and includes digital or analog communication signals or other intangible media that facilitate the communication of the above instructions. Transmission media includes coaxial cables, copper wires, and optical fibers, which include the wires of a bus used to transmit a computer data signal.

[0130] The above are only the preferred embodiments of the present invention and are not intended to limit the present invention. For those skilled in the art, various modifications and variations can be made to the present invention. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principle of the present invention shall be included within the protection scope of the present invention.

[0131] Those skilled in the art should understand that the embodiments of the present application can be provided as methods, systems, or computer program products. Therefore, the present application can take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present application can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk memories, CD-ROMs, optical memories, etc.) containing computer-usable program code. The solutions in the embodiments of the present application can be implemented using various computer languages. For example, object-oriented programming languages such as Java and interpreted scripting languages such as JavaScript.

[0132] The present application is described with reference to the flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or block in the flowchart and / or block diagram, and the combination of processes and / or blocks in the flowchart and / or block diagram can be implemented by computer program instructions. These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing devices to generate a machine, such that the instructions executed by the processor of the computer or other programmable data processing devices generate for implementation in the process Figure 1one process or multiple processes and / or blocks Figure 1 a device for the functions specified in one block or multiple blocks.

[0133] These computer program instructions can also be stored in a computer-readable memory that can guide a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured article including an instruction device, and the instruction device implements the process Figure 1 one process or multiple processes and / or blocks Figure 1 the functions specified in one block or multiple blocks.

[0134] These computer program instructions can also be loaded onto a computer or other programmable data processing device, so that a series of operation steps are executed on the computer or other programmable device to produce a computer-implemented process. Thus, the instructions executed on the computer or other programmable device provide steps for implementing the functions specified in Figure 1 one process or multiple processes and / or blocks Figure 1 one block or multiple blocks.

[0135] Although the preferred embodiments of the present application have been described, those skilled in the art can make additional changes and modifications once they learn the basic creative concepts. Therefore, the appended claims are intended to be interpreted to include the preferred embodiments as well as all changes and modifications that fall within the scope of the present application.

[0136] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalent technologies, the present application also intends to include these modifications and variations.

Claims

1. A data processing process scheduling method based on an improved jellyfish search algorithm, characterized in that: Includes the following: Collect historical measurement data and calculate the computing power demand curve of the measurement data; The computing power demand curve is divided into multiple time windows based on a sliding window, and the computing power demand in each time window fluctuates smoothly after division; The optimization problem is constructed with minimizing resource overhead as the optimization goal, the processing delay of each time window being lower than the tolerable delay and the number of processes allowed on each host as mandatory constraints, and the difference in process distribution between adjacent time windows being smaller than a threshold as a non-mandatory constraint. The optimization problem is solved iteratively, and the optimal solution obtained represents the process distribution vector within each time window; Schedule the number of processes on each host within the corresponding time segment based on the process distribution vector within each time window; The optimization problem is expressed as: Among them, e h T represents the resource overhead per unit time of data processing in host h. i represents the time required for the system to process data in the i-th time window, represents the number of processes enabled by host h in the i-th time window, H represents the number of hosts in the system, represents the process distribution vector of the system in the i-th time window, W represents the number of time windows, represents the number of processes enabled by host h in the i+1th time window, |w i | represents the length of the i-th time window, represents the delay tolerance, P h represents the maximum number of parallel processes allowed on the hth host, θ represents the threshold for the difference in process distribution between adjacent time windows, constraints C1 and C2 are mandatory constraints, and constraint C3 is a non-mandatory constraint; The process of iteratively solving the optimization problem is specifically as follows: Initialize the jellyfish swarm, where the position of jellyfish i in the jellyfish swarm is represented by the H×W dimensional matrix mat i ,Each column in the matrix represents the process distribution vector within a time window; Evaluate the fitness of jellyfish in the jellyfish group and select the current optimal jellyfish position; The position of each jellyfish in the jellyfish group is updated. When the jellyfish's motion mode is active, the standard jellyfish search algorithm formula is used to update the position. When the jellyfish's motion mode is passive, the position is updated based on the following formula: mat i [x,say i ]=γ×mat i [x,say i ]+(1-γ)×fs(P(x=ν)) Among them, mat i [x,dire i ] indicates that the coordinates of the position matrix of jellyfish i are (x, dire i ), where the left side is the updated value and the right side is the value before the update, γ represents the weight parameter, γ∈[0,1], and the function fs(P(x=ν)) represents the probability model P(x=ν) from the set {0,1,2,...,P h A randomly selected value in}, dire i represents the one-dimensional search direction of jellyfish i when the position is updated; Normalize the updated position elements of the jellyfish; The above process is repeated until the maximum number of iterations is reached, and the final optimal jellyfish position is obtained. The optimal jellyfish position represents the process distribution vector in each time window.

2. The data processing process scheduling method based on the improved jellyfish search algorithm according to claim 1, characterized in that: The process of collecting historical measurement data and obtaining the computing power demand curve of the measurement data is specifically as follows: Collect historical measurement data from multiple days, divide a day into several moments, then classify the historical measurement data and statistically obtain the data volume of each type of measurement data at the same moment over multiple days. Based on the three sigma principle, calculate the upper limit of the data volume of each type of measurement data at each moment, and calculate the computing power requirements of all types of measurement data at each moment, thereby obtaining the computing power demand curve of historical measurement data.

3. The data processing process scheduling method based on the improved jellyfish search algorithm according to claim 1, characterized in that: The process of dividing the computing power demand curve into multiple time windows based on the sliding window is specifically as follows: Set the size of the sliding window and calculate the average computing power required within the sliding window during initialization, which is used as the initial value. During initialization, the coverage of the current time window is the coverage of the current sliding window. Use a sliding window to slide non-overlappingly on the computing power demand curve and calculate the new value of the average computing power demand within the sliding window after the sliding. Based on the new value of the average computing power demand and the initial value, determine whether the time window division condition is met. If so, the time range covered by the sliding window after the sliding is included in the current time window. If not, proceed to the next step. Divide the sliding window after sliding based on the binary search algorithm, find the time point coordinates that meet the time window division conditions, and then divide the sliding time window into a left window and a right window based on the time point coordinates. The time range covered by the left window is included in the current time window, and the average computing power requirement in the right window is calculated and used as the updated value of the average computing power requirement in the sliding window. The time range covered by the right window is set as the initial coverage range of the next time window. The computing power demand curve is traversed by sliding the sliding window, thereby dividing the computing power demand curve into multiple time windows. After the division, the computing power demand in each time window fluctuates smoothly.

4. The data processing process scheduling method based on the improved jellyfish search algorithm according to claim 3, characterized in that: If the rate of change of the new value of the mean computing power requirement is less than the preset threshold, or the time range covered by the current time window is less than the preset threshold, the time range covered by the sliding window after sliding is included in the current time window.

5. The data processing process scheduling method based on the improved jellyfish search algorithm according to claim 4, characterized in that: If the time range covered by the current time window is less than the preset threshold, the computing power demand variance in the sliding window after sliding is calculated. If the computing power demand variance is less than the preset threshold, the computing power demand mean in the sliding window is updated from the initial value to the new value, otherwise no update is performed.

6. A data processing process scheduling system based on an improved jellyfish search algorithm, using the method according to any one of claims 1 to 5, characterized in that: include: The data characteristics analysis module is used to collect historical measurement data and obtain the computing power demand curve of the measurement data; The time window division module is used to divide the computing power demand curve into multiple time windows based on a sliding window, so that the computing power demand in each time window fluctuates smoothly after division; An optimization problem construction module is used to construct an optimization problem with minimizing resource overhead as the optimization objective, with the processing delay of each time window being lower than the tolerable delay and the number of processes allowed on each host as mandatory constraints, and with the difference in process distribution between adjacent time windows being smaller than a threshold as a non-mandatory constraint. Iterative solution module, used to iteratively solve the optimization problem, and the optimal solution obtained represents the process distribution vector in each time window; The process scheduling module is used to schedule the number of processes on each host within the corresponding time segment based on the process distribution vector within each time window.

7. An electronic device, characterized in that: The method comprises a processor and a memory, wherein a computer program is stored in the memory, and the processor is configured to execute the steps of the method according to any one of claims 1 to 5 by calling the computer program stored in the memory.

8. A computer-readable storage medium for storing a computer program for scheduling data processing processes based on an improved jellyfish search algorithm, characterized in that: When the computer program is run on a computer, the steps of the method according to any one of claims 1 to 5 are executed.

Images