Intelligent scheduling system integrating cooperative work of multiple external self-developed plug-ins

Abstract

The invention discloses an intelligent scheduling system integrating cooperative work of multiple external self-developed plug-ins. The intelligent scheduling system comprises a data information acquisition unit, a data information preprocessing unit, a storage database, a decision analysis unit, a task scheduling distribution unit and an execution scheduling unit. According to the method, the load and the resource of the sub-node server are analyzed according to the load model and the scheduling model, and the calculation task is allocated according to the load prediction and resource prediction conditions of the sub-node server, so that the overall working efficiency is improved, the waiting time and the processing time are reduced, and the task is completed more quickly; task scheduling allocation is carried out based on the resource utilization rate, tasks are allocated according to the resource utilization rate, obtained through resource prediction, of the sub-node servers and scheduling optimization of data information tasks, the priority of the tasks with large loads is high, it is ensured that the tasks with the high priority are executed firstly, and resource use is balanced as much as possible.

Description

Technical Field

[0001] The present invention belongs to the field of artificial intelligence technology, and more specifically, particularly relates to an intelligent scheduling system integrating multiple external self-developed plug-ins to work in collaboration. Background Art

[0002] With the development and application of artificial intelligence technology, intelligent scheduling architecture, as an emerging technical solution, is gradually becoming an important tool for various industries to optimize resource allocation and improve efficiency.

[0003] Intelligent scheduling architecture refers to a resource scheduling system built based on artificial intelligence algorithms and data analysis technology, which aims to achieve intelligent management and optimal allocation of resources. Through real-time monitoring and analysis of information such as task requirements and resource status, the intelligent scheduling architecture can automatically adjust task allocation plans to improve resource utilization and work efficiency.

[0004] Principles of Intelligent Scheduling Architecture

[0005] 1. Data collection and processing: The intelligent scheduling architecture collects and organizes various types of data, including task requirements, resource status, environmental conditions, etc.

[0006] 2. Intelligent decision-making algorithm: Based on the collected data, the intelligent scheduling architecture will use various intelligent algorithms, such as machine learning and deep learning, to conduct decision analysis and generate the optimal scheduling plan.

[0007] 3. Task allocation and optimization: Based on the scheduling plan generated by the algorithm, the intelligent scheduling architecture will automatically assign tasks to appropriate resources and continuously optimize the scheduling plan to improve efficiency and response speed.

[0008] However, there are some problems with the existing technology: the existing intelligent scheduling system executes all external plug-in functions, and judges, filters and summarizes the results returned by all functional plug-ins based on the returned results. This method is inefficient and consumes a lot of resources when executing a small function. It has high resource consumption, long waiting time and poor user experience. Therefore, we propose an intelligent scheduling system that integrates multiple external self-developed plug-ins to work together. Summary of the invention

[0009] In view of the problems existing in the prior art, the purpose of the present invention is to provide an intelligent scheduling system that integrates multiple external self-developed plug-ins to work together. By combining a machine learning model, it can predict future resource usage and perform scheduling in advance, thereby making more efficient use of resources and reducing task waiting time.

[0010] To achieve the above-mentioned object, the present invention provides the following technical solutions: an intelligent scheduling system integrating multiple external self-developed plug-ins to work together, including a data information acquisition unit, a data information preprocessing unit, a storage database, a decision analysis unit, a task scheduling allocation unit and an execution scheduling unit;

[0011] The data information collection unit is used for the central node to collect data information from various data sources at regular intervals. The data information collection unit includes task attribute data obtained through the API interface, server monitoring data obtained through sensors, and historical data obtained through the database;

[0012] The data information preprocessing unit is used for the central node to preprocess the acquired data, and the data information preprocessing unit includes data cleaning, data standardization and feature extraction for the data information;

[0013] The storage database is used to store the preprocessed data information, and the central node stores the features of the preprocessed data information in the database;

[0014] The decision analysis unit is used for decision customization of task allocation. The central node analyzes the load and resources of the sub-node server through the load model and scheduling model, and allocates computing tasks according to the load prediction and resource prediction of the sub-node server;

[0015] The task scheduling and allocation unit performs task scheduling and allocation based on resource utilization, allocates tasks according to the resource utilization of the sub-node server obtained by resource prediction, and the scheduling optimization of data information tasks, and the tasks with larger computing loads have higher priorities, and ensures that the tasks with higher priorities are executed first, and tries to balance the use of resources;

[0016] The execution scheduling sends execution commands to each subnode server according to the scheduling results; monitors the execution status of the task in real time to ensure the normal execution of the task; and records the status of the task execution for subsequent analysis and optimization.

[0017] Specifically, the data cleaning includes filtering the data, detecting outliers, and replacing and filling out outliers and missing values;

[0018] The calculation of the filtering process is as follows:

[0019] For each data information x i , use windows of different scales to perform median filtering, and then perform weighted fusion on the filtering results of different scales;

[0020]

[0021] Among them, y iRepresents the output data information after filtering; S represents the number of scales, that is, the number of different scales used when performing median filtering operations, and each scale corresponds to a filtering window of different sizes; w s Represents the weight of scale s, which is used to perform weighted averaging of filtering results of different scales; It represents the result of median filtering the data information at scale s. Specifically, It is all the data information within the window range at scale s, that is, the neighborhood data information at this scale; window s Represents the neighborhood window corresponding to scale s. At different scales, the size of the neighborhood window is different.

[0022] Specifically, the calculation of the outlier detection is as follows:

[0023] The Z-score of the data information is obtained by calculating the difference between the data information and the mean and dividing it by the standard deviation;

[0024] Among them, X is the data information; μ is the mean of the data information; σ is the standard deviation of the data information;

[0025] Furthermore, if the outlier is determined as Z>3, the data is determined to be an outlier; if Z<3, the data is determined to be a normal value and the outlier is removed.

[0026] And the vacancies removed by outliers and missing values ​​are filled by the mean, and the mean is calculated as follows:

[0027] Among them, x i is the i-th value in the data information; n is the total number of data information; Represents the sum of all data information.

[0028] Specifically, the calculation of the data standardization is as follows:

[0029] Minimum-maximum normalization linearly transforms the data to the specified range, and the range is [0,1], the formula is as follows:

[0030] Where: x min and x max are the minimum and maximum values ​​of the data respectively; X' is the standardized value.

[0031] Specifically, the feature extraction mainly includes numerical feature extraction and text feature extraction:

[0032] The numerical features extract statistical features through mean, standard deviation and median to measure the central tendency and dispersion of data; describe the extreme values ​​of data through maximum value, minimum value and range; and describe the distribution form of data through skewness and kurtosis;

[0033] Aggregate features summarize time series or categorical features through sums and averages;

[0034] The difference feature is used to calculate the data changes at adjacent moments;

[0035] The text feature extraction converts the text into a fixed-length vector through the bag-of-words model, in which each dimension represents the frequency of occurrence of a word in the text; and calculates the importance of each word in the document based on word frequency and text frequency through TF-IDF.

[0036] Specifically, the calculation steps of the load model are as follows:

[0037] Data preparation: Collect historical load data of subnode servers, including CPU usage, memory usage, and task execution time; divide the historical load data into training sets and test sets;

[0038] Model selection: Select the neural network prediction model;

[0039] Model training: Use the training set data to train the model; evaluate the performance of the model on the test set and adjust the model parameters to optimize performance;

[0040] Model application: Use the trained model to predict the system load in the future, and schedule in advance based on the prediction results to avoid resource bottlenecks.

[0041] Specifically, the calculation formula of the load model is as follows:

[0042] Combine CPU usage, memory usage, and task execution time to predict the computing pressure of the task;

[0043] Propagation from input layer to hidden layer:

[0044] Among them, a k is the input of the hidden layer neuron, w th is the weight from the input layer to the hidden layer, x i is the output of the input layer neuron, b k is the bias of the hidden layer neurons, n is the number of input layer neurons;

[0045] Propagation from hidden layer to output layer:

[0046] Among them, y iis the output of the output layer neuron, w hj is the weight from the hidden layer to the output layer, σ is the activation function, m is the number of neurons in the hidden layer, and b j is the bias of the output layer;

[0047] Through the output of the output layer, the available resource size of the child node server is obtained.

[0048] Specifically, the resource allocation of the decision analysis unit calculates the optimal resource allocation scheme according to the resource status of the current sub-node server and the resource size required by the task. The steps of resource allocation are as follows:

[0049] The resource status of the node server is evaluated by calculating a score based on the CPU, memory, disk, and network usage of the child node server. The lower the score, the more idle the resources are.

[0050] When allocating task requirements, the subnode server reserves some resources for emergencies to ensure system stability;

[0051] Task priority sorting is sorted according to the priority of the tasks, and high-priority tasks are executed first;

[0052] If there are task dependencies, the predecessor tasks need to be scheduled first.

[0053] Specifically, the resource allocation scheme is based on scheduling the resource utilization of the sub-node servers, and preferentially schedules tasks to the sub-node servers with the lowest resource utilization, and the steps of task allocation are as follows:

[0054] Initialization: Create an empty scheduling result list schedule_results and create an empty allocated task set allocated_tasks;

[0055] Sorting sub-node servers: sort all sub-node servers according to resource utilization, and select the server with the lowest utilization as the candidate server;

[0056] Sorting tasks: Sort all tasks to be scheduled according to their priority. Tasks with higher priority are placed first. If there are task dependencies, the predecessor tasks need to be scheduled first.

[0057] Scheduling tasks one by one: traverse the sorted task list and try to assign them to candidate servers one by one; for each task, check whether the resources of the candidate server are sufficient;

[0058] If the resources are sufficient, the task is assigned to the corresponding subnode server and the resource status of the subnode server is updated; the task is added to the assigned task set and recorded in schedule_results;

[0059] If there are insufficient resources, skip the task and try the next one;

[0060] Repeat the above steps to continue assigning tasks: Continue to traverse the remaining tasks until all tasks have been attempted to be assigned or all child node servers are fully loaded;

[0061] Return result: Returns the final schedule_results, which contains the allocation result of each task.

[0062] Specifically, the steps of task scheduling optimization of the task scheduling allocation unit are as follows:

[0063] Collect historical task scheduling data, including task priority, resource requirements, and execution time;

[0064] Divide the historical task scheduling data into training set and test set;

[0065] Select a reinforcement learning model;

[0066] Train the model using the training set data; evaluate the model's performance on the test set and adjust model parameters to optimize performance;

[0067] Use the trained model to select the optimal scheduling strategy based on the current status and historical experience.

[0068] Technical effects and advantages of the present invention:

[0069] The present invention optimizes resource allocation in a multi-task or multi-resource environment through an intelligent scheduling algorithm, that is, the load and resources of the sub-node server are analyzed according to the load model and the scheduling model, and computing tasks are allocated according to the load prediction and resource prediction of the sub-node server, thereby improving overall work efficiency, reducing waiting time and processing time, and completing tasks faster;

[0070] By reasonably arranging resource usage to avoid waste, and optimizing scheduling strategies to reduce unnecessary energy consumption and other resource consumption, that is, scheduling and allocating tasks based on resource utilization, allocating tasks based on the resource utilization of the sub-node servers obtained by resource prediction, and scheduling optimization of data information tasks, and giving higher priority to tasks with larger computing loads, and ensuring that high-priority tasks are executed first, and try to balance the use of resources;

[0071] When faced with emergencies or changes in demand, the intelligent scheduling system can quickly adapt and reschedule tasks to improve the system's resilience; improve service quality through intelligent scheduling, allowing users to enjoy faster and more reliable services, that is, monitor the execution status of tasks in real time to ensure normal execution of tasks; record the status of task execution for subsequent analysis and optimization.

[0072] Further features and advantages of the present invention will become apparent from the following detailed description of exemplary embodiments of the present invention with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0073] Figure 1 It is a schematic diagram of the system structure provided by the present invention;

[0074] Figure 2 It is a schematic diagram of the calculation flow of the load model provided by the present invention;

[0075] Figure 3 It is a schematic diagram of the steps of resource allocation provided by the present invention;

[0076] Figure 4 It is a flowchart of the steps of task scheduling optimization provided by the present invention. DETAILED DESCRIPTION

[0077] In order to make the purpose, technical scheme and advantages of the present invention clearer, the present invention is further described in detail below in conjunction with specific embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0078] like Figures 1 to 4 As shown, an embodiment of the present invention provides an intelligent scheduling system that integrates multiple external self-developed plug-ins to work together, including a data information acquisition unit, a data information preprocessing unit, a storage database, a decision analysis unit, a task scheduling allocation unit, and an execution scheduling unit;

[0079] The data information collection unit is used for the central node to collect data information from various data sources at regular intervals. The data information collection unit includes task attribute data obtained through the API interface, server monitoring data obtained through sensors, and historical data obtained through the database;

[0080] The data information preprocessing unit is used for the central node to preprocess the acquired data, and the data information preprocessing unit includes data cleaning, data standardization and feature extraction for the data information;

[0081] The storage database is used to store the preprocessed data information, and the central node stores the features of the preprocessed data information in the database;

[0082] The decision analysis unit is used for decision customization of task allocation. The central node analyzes the load and resources of the sub-node server through the load model and scheduling model, and allocates computing tasks according to the load prediction and resource prediction of the sub-node server;

[0083] The task scheduling and allocation unit performs task scheduling and allocation based on resource utilization, allocates tasks according to the resource utilization of the sub-node server obtained by resource prediction, and the scheduling optimization of data information tasks, and the tasks with larger computing loads have higher priorities, and ensures that the tasks with higher priorities are executed first, and tries to balance the use of resources;

[0084] The execution scheduling sends execution commands to each subnode server according to the scheduling results; monitors the execution status of the task in real time to ensure the normal execution of the task; and records the status of the task execution for subsequent analysis and optimization.

[0085] In this embodiment, preferably, the data cleaning includes filtering the data, detecting outliers, and replacing and filling out outliers and missing values;

[0086] The calculation of the filtering process is as follows:

[0087] For each data information x i , use windows of different scales to perform median filtering, and then perform weighted fusion on the filtering results of different scales;

[0088]

[0089] Among them, y i Represents the output data information after filtering; S represents the number of scales, that is, the number of different scales used when performing median filtering operations, and each scale corresponds to a filtering window of different sizes; w s Represents the weight of scale s, which is used to perform weighted averaging of filtering results of different scales; It represents the result of median filtering the data information at scale s. Specifically, It is all the data information within the window range at scale s, that is, the neighborhood data information at this scale; window s Represents the neighborhood window corresponding to scale s. At different scales, the size of the neighborhood window is different;

[0090] It should be noted that the use of multi-scale median filtering technology can process signals containing noise of different scales. In multi-scale median filtering, the signal is filtered at different scales, and then the filtering results of different scales are fused, which can effectively filter out the noise in the data information and improve the accuracy of the data information.

[0091] In this embodiment, preferably, the calculation of the outlier detection is as follows:

[0092] The Z-score of the data information is obtained by calculating the difference between the data information and the mean and dividing it by the standard deviation;

[0093] Among them, X is the data information; μ is the mean of the data information; σ is the standard deviation of the data information;

[0094] Furthermore, if the outlier is determined as Z>3, the data is determined to be an outlier; if Z<3, the data is determined to be a normal value and the outlier is removed.

[0095] And the vacancies removed by outliers and missing values ​​are filled by the mean, and the mean is calculated as follows:

[0096] Among them, x i is the i-th value in the data information; n is the total number of data information; Represents the sum of all data information;

[0097] It should be noted that outliers are detected by the mean and standard deviation, which makes it easier to fill in outliers and missing values ​​based on the mean.

[0098] In this embodiment, preferably, the calculation of the data standardization is as follows:

[0099] Minimum-maximum normalization linearly transforms the data to the specified range, and the range is [0,1], the formula is as follows:

[0100] Where: x min and x max are the minimum and maximum values ​​of the data information respectively; X' is the standardized value;

[0101] It should be noted that converting data of different dimensions or units into dimensionless data gives it a uniform scale and distribution, and the result range is fixed, usually between [0,1], which is suitable for some models that require a fixed input range.

[0102] In this embodiment, preferably, the feature extraction mainly includes numerical feature extraction and text feature extraction:

[0103] The numerical features extract statistical features through mean, standard deviation and median to measure the central tendency and dispersion of data; describe the extreme values ​​of data through maximum value, minimum value and range; and describe the distribution form of data through skewness and kurtosis;

[0104] Aggregate features summarize time series or categorical features through sums and averages;

[0105] The difference feature is used to calculate the data changes at adjacent moments;

[0106] The text feature extraction converts the text into a fixed-length vector through the bag-of-words model, where each dimension represents the frequency of occurrence of a word in the text; the importance of each word in the document is calculated based on word frequency and text frequency through TF-IDF;

[0107] It should be noted that meaningful features for model prediction or classification tasks are extracted from the original data to improve the performance of the model, reduce computational complexity, and even provide necessary input for certain models.

[0108] In this embodiment, preferably, the calculation steps of the load model are as follows:

[0109] Data preparation: Collect historical load data of subnode servers, including CPU usage, memory usage, and task execution time; divide the historical load data into training sets and test sets;

[0110] Model selection: Select the neural network prediction model;

[0111] Model training: Use the training set data to train the model; evaluate the performance of the model on the test set and adjust the model parameters to optimize performance;

[0112] Model application: Use the trained model to predict the system load in the future, and schedule in advance based on the prediction results to avoid resource bottlenecks;

[0113] It should be noted that the load resources of the sub-node server are predicted, and analysis and processing are performed based on the CPU usage, memory usage, and task execution time of the sub-node server, so as to facilitate the prediction of the load resources of the sub-node server and avoid resource bottlenecks.

[0114] In this embodiment, preferably, the calculation formula of the load model is as follows:

[0115] Combine CPU usage, memory usage, and task execution time to predict the computing pressure of the task;

[0116] Propagation from input layer to hidden layer:

[0117] Among them, a k is the input of the hidden layer neuron, w th is the weight from the input layer to the hidden layer, x i is the output of the input layer neuron, b kis the bias of the hidden layer neurons, n is the number of input layer neurons;

[0118] Propagation from hidden layer to output layer:

[0119] Among them, y i is the output of the output layer neuron, w hj is the weight from the hidden layer to the output layer, σ is the activation function, m is the number of neurons in the hidden layer, and b j is the bias of the output layer;

[0120] Through the output of the output layer, the available resource size of the child node server is obtained;

[0121] It should be noted that the prediction of the load resources of the sub-node servers is realized through the neural network, so as to obtain the available resource size of the sub-node servers and to allocate tasks to the sub-node servers.

[0122] In this embodiment, preferably, the resource allocation of the decision analysis unit calculates the optimal resource allocation scheme according to the resource status of the current subnode server and the resource size required by the task. The steps of resource allocation are as follows:

[0123] The resource status of the node server is evaluated by calculating a score based on the CPU, memory, disk, and network usage of the child node server. The lower the score, the more idle the resources are.

[0124] When allocating task requirements, the subnode server reserves some resources for emergencies to ensure system stability;

[0125] Task priority sorting is sorted according to the priority of the tasks, and high-priority tasks are executed first;

[0126] If there are task dependencies, the predecessor tasks need to be scheduled first;

[0127] It should be noted that by making optimal resource allocation for the task requirement resource size through the resource status of the sub-node server, the accurate allocation of tasks can be effectively improved, which facilitates the rapid allocation of tasks and improves the utilization rate of the sub-node server.

[0128] In this embodiment, preferably, the resource allocation scheme is based on the scheduling of resource utilization of the sub-node servers, and the tasks are preferentially scheduled to the sub-node servers with the lowest resource utilization, and the steps of task allocation are as follows:

[0129] Initialization: Create an empty scheduling result list schedule_results and create an empty allocated task set allocated_tasks;

[0130] Sorting sub-node servers: sort all sub-node servers according to resource utilization, and select the server with the lowest utilization as the candidate server;

[0131] Sorting tasks: Sort all tasks to be scheduled according to their priority. Tasks with higher priority are placed first. If there are task dependencies, the predecessor tasks need to be scheduled first.

[0132] Scheduling tasks one by one: traverse the sorted task list and try to assign them to candidate servers one by one; for each task, check whether the resources of the candidate server are sufficient;

[0133] If the resources are sufficient, the task is assigned to the corresponding subnode server and the resource status of the subnode server is updated; the task is added to the assigned task set and recorded in schedule_results;

[0134] If there are insufficient resources, skip the task and try the next one;

[0135] Repeat the above steps to continue assigning tasks: Continue to traverse the remaining tasks until all tasks have been attempted to be assigned or all child node servers are fully loaded;

[0136] Return result: Return the final schedule_results, which contains the allocation result of each task;

[0137] It should be noted that based on the scheduling of resource utilization of the subnode servers, the tasks are preferentially scheduled to the subnode servers with the lowest resource utilization, and the task allocation results are returned to facilitate task allocation management.

[0138] In this embodiment, preferably, the task scheduling optimization steps of the task scheduling allocation unit are as follows:

[0139] Collect historical task scheduling data, including task priority, resource requirements, and execution time;

[0140] Divide the historical task scheduling data into training set and test set;

[0141] Select a reinforcement learning model;

[0142] Train the model using the training set data; evaluate the model's performance on the test set and adjust model parameters to optimize performance;

[0143] Use the trained model to select the optimal scheduling strategy based on the current status and historical experience;

[0144] It should be noted that the task priority, resource requirements, and execution time of historical tasks are used to implement model calculation processing for tasks, so as to select the optimal scheduling strategy based on the current status and historical experience.

[0145] The steps of this application process:

[0146] Step 1: The data information collection unit of the central node obtains data information from the data source: the central node collects data information from various data sources at regular intervals, including task attribute data obtained through the API interface, server monitoring data obtained through sensors, and historical data obtained through the database;

[0147] Step 2: The data information preprocessing unit performs data preprocessing on the collected data information: The central node preprocesses the acquired data, including data cleaning, data standardization and feature extraction of the data information;

[0148] Step 3: Storing the preprocessed data information in a storage database: The central node stores the features of the preprocessed data information in a database;

[0149] Step 4: The decision analysis unit customizes the decision of task allocation: the central node analyzes the load and resources of the sub-node server through the load model and scheduling model, and allocates computing tasks according to the load prediction and resource prediction of the sub-node server;

[0150] Step 5: The task scheduling and allocation unit performs task scheduling and allocation based on resource utilization: tasks are allocated based on the resource utilization of the sub-node servers obtained by resource prediction and the scheduling optimization of data information tasks, and tasks with larger computing loads are given higher priority, and tasks with higher priority are ensured to be executed first, and the use of resources is balanced as much as possible;

[0151] Step 6: The execution scheduling unit performs execution scheduling: sends execution commands to each sub-node server according to the scheduling results; monitors the execution status of the task in real time to ensure the normal execution of the task; records the status of the task execution for subsequent analysis and optimization

[0152] Finally, it should be noted that the above is only a preferred embodiment of the present invention and is not intended to limit the present invention. Although the present invention has been described in detail with reference to the aforementioned embodiments, it is still possible for those skilled in the art to modify the technical solutions described in the aforementioned embodiments or to make equivalent substitutions for some of the technical features therein. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. An intelligent scheduling system integrating multiple external self-developed plug-ins to work together, characterized in that: It includes a data information acquisition unit, a data information preprocessing unit, a storage database, a decision analysis unit, a task scheduling and allocation unit, and an execution scheduling unit; The data information collection unit is used for the central node to collect data information from various data sources at regular intervals. The data information collection unit includes task attribute data obtained through the API interface, server monitoring data obtained through sensors, and historical data obtained through the database; The data information preprocessing unit is used for the central node to preprocess the acquired data, and the data information preprocessing unit includes data cleaning, data standardization and feature extraction for the data information; The storage database is used to store the preprocessed data information, and the central node stores the features of the preprocessed data information in the database; The decision analysis unit is used for decision customization of task allocation. The central node analyzes the load and resources of the sub-node server through the load model and scheduling model, and allocates computing tasks according to the load prediction and resource prediction of the sub-node server; The task scheduling and allocation unit performs task scheduling and allocation based on resource utilization, allocates tasks according to the resource utilization of the sub-node server obtained by resource prediction, and the scheduling optimization of data information tasks, and the tasks with larger computing loads have higher priorities, and ensures that the tasks with higher priorities are executed first, and tries to balance the use of resources; The execution scheduling sends execution commands to each subnode server according to the scheduling results; monitors the execution status of the task in real time to ensure the normal execution of the task; Record the status of task execution for subsequent analysis and optimization.

2. According to claim 1, an intelligent scheduling system integrating multiple external self-developed plug-ins to work together is characterized by: The data cleaning includes filtering the data, detecting outliers, and replacing and filling out outliers and missing values; The calculation of the filtering process is as follows: For each data information x i , use windows of different scales to perform median filtering, and then perform weighted fusion on the filtering results of different scales; Among them, y i Represents the output data information after filtering; S represents the number of scales, that is, the number of different scales used when performing median filtering operations, and each scale corresponds to a filtering window of different sizes; w s Represents the weight of scale s, which is used to perform weighted averaging of filtering results of different scales; It represents the result of median filtering the data information at scale s. Specifically, It is all the data information within the window range at scale s, that is, the neighborhood data information at this scale; window s Represents the neighborhood window corresponding to scale s. At different scales, the size of the neighborhood window is different.

3. According to claim 2, an intelligent scheduling system integrating multiple external self-developed plug-ins to work together is characterized by: The calculation of the outlier detection is as follows: The Z-score of the data information is obtained by calculating the difference between the data information and the mean and dividing it by the standard deviation; Among them, X is the data information; μ is the mean of the data information; σ is the standard deviation of the data information; Furthermore, if the outlier is determined as Z>3, the data is determined to be an outlier; if Z<3, the data is determined to be a normal value and the outlier is removed. And the vacancies removed by outliers and missing values ​​are filled by the mean, and the mean is calculated as follows: Among them, x i is the i-th value in the data information; n is the total number of data information; Represents the sum of all data information.

4. According to claim 1, an intelligent scheduling system integrating multiple external self-developed plug-ins to work together is characterized by: The data normalization is calculated as follows: Minimum-maximum normalization linearly transforms the data to the specified range, and the range is [0,1], the formula is as follows: Where: x min and x max are the minimum and maximum values ​​of the data respectively; X' is the standardized value.

5. According to claim 1, an intelligent scheduling system integrating multiple external self-developed plug-ins to work together is characterized by: The feature extraction mainly includes numerical feature extraction and text feature extraction: The numerical features extract statistical features through mean, standard deviation and median to measure the central tendency and dispersion of data; describe the extreme values ​​of data through maximum value, minimum value and range; and describe the distribution form of data through skewness and kurtosis; Aggregate features summarize time series or categorical features through sums and averages; The difference feature is used to calculate the data changes at adjacent moments; The text feature extraction converts the text into a fixed-length vector through the bag-of-words model, in which each dimension represents the frequency of occurrence of a word in the text; and calculates the importance of each word in the document based on word frequency and text frequency through TF-IDF.

6. According to claim 1, an intelligent scheduling system integrating multiple external self-developed plug-ins to work together is characterized by: The calculation steps of the load model are as follows: Data preparation: Collect historical load data of subnode servers, including CPU usage, memory usage, and task execution time; divide the historical load data into training sets and test sets; Model selection: Select the neural network prediction model; Model training: Use the training set data to train the model; evaluate the performance of the model on the test set and adjust the model parameters to optimize performance; Model application: Use the trained model to predict the system load in the future, and schedule in advance based on the prediction results to avoid resource bottlenecks.

7. The intelligent scheduling system integrating multiple external self-developed plug-ins for collaborative operation according to claim 6 is characterized in that: The calculation formula of the load model is as follows: Combine CPU usage, memory usage, and task execution time to predict the computing pressure of the task; Propagation from input layer to hidden layer: Among them, a k is the input of the hidden layer neuron, w th is the weight from the input layer to the hidden layer, x i is the output of the input layer neuron, b k is the bias of the hidden layer neurons, n is the number of input layer neurons; Propagation from hidden layer to output layer: Among them, y i is the output of the output layer neuron, w hj is the weight from the hidden layer to the output layer, σ is the activation function, m is the number of neurons in the hidden layer, and b j is the bias of the output layer; Through the output of the output layer, the available resource size of the child node server is obtained.

8. The intelligent scheduling system integrating multiple external self-developed plug-ins to work together according to claim 1 is characterized by: The resource allocation of the decision analysis unit calculates the optimal resource allocation scheme according to the resource status of the current sub-node server and the resource size required by the task. The steps of resource allocation are as follows: The resource status of the node server is evaluated by calculating a score based on the CPU, memory, disk, and network usage of the child node server. The lower the score, the more idle the resources are. When allocating task requirements, the subnode server reserves some resources for emergencies to ensure system stability; Task priority sorting is sorted according to the priority of the tasks, and high-priority tasks are executed first; If there are task dependencies, the predecessor tasks need to be scheduled first.

9. The intelligent scheduling system integrating multiple external self-developed plug-ins for collaborative operation according to claim 8 is characterized in that: The resource allocation scheme is based on scheduling the resource utilization of the sub-node servers, and preferentially schedules tasks to the sub-node servers with the lowest resource utilization, and the steps of task allocation are as follows: Initialization: Create an empty scheduling result list schedule_results and create an empty allocated task set allocated_tasks; Sorting sub-node servers: sort all sub-node servers according to resource utilization, and select the server with the lowest utilization as the candidate server; Sorting tasks: Sort all tasks to be scheduled according to their priority. Tasks with higher priority are placed first. If there are task dependencies, the predecessor tasks need to be scheduled first. Scheduling tasks one by one: traverse the sorted task list and try to assign them to candidate servers one by one; for each task, check whether the resources of the candidate server are sufficient; If the resources are sufficient, the task is assigned to the corresponding subnode server and the resource status of the subnode server is updated; the task is added to the assigned task set and recorded in schedule_results; If there are insufficient resources, skip the task and try the next one; Repeat the above steps to continue assigning tasks: Continue to traverse the remaining tasks until all tasks have been attempted to be assigned or all child node servers are fully loaded; Return result: Returns the final schedule_results, which contains the allocation result of each task.

10. The intelligent scheduling system integrating multiple external self-developed plug-ins for collaborative work according to claim 1 is characterized in that: The steps of task scheduling optimization of the task scheduling allocation unit are as follows: Collect historical task scheduling data, including task priority, resource requirements, and execution time; Divide the historical task scheduling data into training set and test set; Select a reinforcement learning model; Train the model using the training set data; evaluate the model's performance on the test set and adjust model parameters to optimize performance; Use the trained model to select the optimal scheduling strategy based on the current status and historical experience.

Images