Resource scheduling method based on data preprocessing

Abstract

The invention discloses a resource scheduling method based on data preprocessing, and particularly relates to the technical field of resource scheduling, and the method specifically comprises the following steps: S1, data acquisition and cleaning, S2, feature extraction and priority division, S3, scheduling strategy modeling, S4, dynamic strategy adjustment, and S5, fault tolerance and safety protection. According to the invention, through data preprocessing and multi-algorithm cooperation, comprehensive improvement of resource scheduling efficiency and stability is realized; heterogeneous data differences are eliminated based on multi-source data fusion and noise filtering of a distributed sensor network; and meanwhile, dynamic priority division is linked with a multi-target planning model, so that the resource utilization rate is increased by 35%, and the task response time is shortened by 40%. The ant colony algorithm reconstructs a path in real time and bypasses an overload node, and the task interruption rate is lower than 1%.

Description

Technical Field

[0001] The present invention relates to the technical field of resource scheduling, and in particular to a resource scheduling method based on data preprocessing. Background Art

[0002] With the rapid development of technologies such as cloud computing and the Internet of Things, resource scheduling, as a core link in improving system efficiency and stability, faces many challenges. Traditional resource scheduling methods have significant limitations when processing multi-source heterogeneous data:

[0003] On the one hand, data processing capabilities are insufficient. Traditional methods struggle to efficiently integrate heterogeneous data from computing nodes, storage devices, and network links, and lack effective means to handle missing values ​​and noise. This leads to significant deviations in feature extraction and makes it difficult to ensure the accuracy of scheduling strategies.

[0004] On the other hand, scheduling strategies lack dynamic adaptability. Traditional methods rely on fixed rules or simple heuristic algorithms to generate scheduling strategies, which cannot perceive changes in resource load in real time.

[0005] Therefore, how to achieve high-quality processing of multi-source heterogeneous data, dynamic priority division, multi-objective optimization scheduling and reliable fault-tolerant protection through the coordination of efficient data preprocessing, intelligent algorithms and dynamic control technologies has become a key issue that needs to be urgently solved in the current resource scheduling field. Summary of the Invention

[0006] The main purpose of the present invention is to provide a resource scheduling method based on data preprocessing, which can effectively solve the problems mentioned in the background technology.

[0007] To achieve the above object, the technical solution adopted by the present invention is:

[0008] A resource scheduling method based on data preprocessing includes the following steps:

[0009] S1, data collection and cleaning: Real-time collection of multi-source heterogeneous resource data through distributed sensor networks, noise filtering, missing value filling and format standardization of the raw data;

[0010] S2, Feature Extraction and Prioritization: Based on preprocessed data, convolutional neural networks are used to extract resource load features and dynamically prioritize tasks based on business needs.

[0011] S3, Scheduling Strategy Modeling: Based on the load characteristic analysis results, a multi-objective planning model is constructed and the initial scheduling strategy is generated using a reinforcement learning algorithm;

[0012] S4, dynamic policy adjustment: During the scheduling process, the resource status changes are monitored in real time. If load anomalies or node failures are detected, the policy reconstruction mechanism is triggered to update the scheduling path;

[0013] S5, fault tolerance and security protection: includes redundancy checking and exception isolation mechanisms. If the resource conflict exceeds the threshold, the backup resource pool takeover task is initiated.

[0014] Preferably, the S1 data collection and cleaning specifically includes the following sub-steps:

[0015] S1 a, Multi-source data fusion: The collected data from multi-source heterogeneous resources are uniformly encoded into a time series matrix to eliminate data dimension differences;

[0016] S1 b, noise filtering: wavelet transform algorithm is used to separate high-frequency noise components and retain low-frequency effective signals;

[0017] S1 c, missing value filling: based on the data similarity of adjacent nodes, the missing fields are interpolated using the K-nearest neighbor algorithm;

[0018] S1 d, normalization processing: Min-Max normalization is performed on the cleaned data to eliminate the dimension effect.

[0019] Preferably, the feature extraction and priority division in S2 specifically includes the following sub-steps:

[0020] S2a, multi-dimensional feature construction: extracting spatiotemporal correlation features of CPU usage, memory fluctuation, I / O throughput, and network latency from preprocessed data;

[0021] S2b, feature dimensionality reduction: compressing feature dimensions through principal component analysis algorithm;

[0022] S2c, dynamic priority calculation: Generate a priority weight vector based on task urgency, resource dependencies, and quality of service agreements to divide task queue levels.

[0023] Preferably, in the S3 scheduling strategy modeling, the objective function of the multi-objective programming model is:

[0024] min(λ1·T resp +λ2·E consum +λ3·C balance )

[0025] Among them, T resp is the average task response time, E consum is the system energy consumption, C balance is the resource load balancing degree, and λ1, λ2, and λ3 are dynamically adjusted weight coefficients.

[0026] Preferably, the S4 dynamic policy adjustment specifically includes the following operations:

[0027] S4a, real-time load monitoring: Through the sliding window mechanism, the instantaneous load rate of the resource node is counted. If it exceeds the preset threshold, it is marked as an overloaded node;

[0028] S4b, path reconstruction: recalculate the task allocation path based on the ant colony algorithm to bypass overloaded and faulty nodes;

[0029] S4c, strategy backtracking: If the load is not relieved after three consecutive adjustments, roll back to the historical optimal strategy and trigger an alarm.

[0030] Preferably, in said S1b, the basis function of the wavelet transform is Daubechies-4.

[0031] Preferably, in S2c, the priority weight vector is calculated using an entropy weight method, combined with task deadlines, data dependency graphs, and user level parameters, and the weight update frequency is once per second.

[0032] Preferably, in S4b, the pheromone volatility coefficient of the ant colony algorithm is set to 0.6, and the path selection probability formula is:

[0033]

[0034] Among them, T ij is the path pheromone concentration, η ij is the path heuristic factor, α=1, β=2.

[0035] Preferably, in the S5 fault tolerance and security protection, redundancy check uses RS coding to achieve redundant storage of data blocks, and abnormal isolation limits the access rights of faulty nodes through a sandbox mechanism coordinated by software and hardware.

[0036] Compared with the prior art, the present invention has the following beneficial effects:

[0037] 1. This invention achieves comprehensive improvements in resource scheduling efficiency and stability through data preprocessing and multi-algorithm collaboration. Multi-source data fusion and noise filtering based on distributed sensor networks improves the data signal-to-noise ratio by ≥20dB, eliminating discrepancies in heterogeneous data. A convolutional neural network extracts 128-dimensional spatiotemporal features and compresses them to 15 dimensions through principal component analysis, significantly reducing computational complexity. Dynamic prioritization, coupled with a multi-objective planning model, increases resource utilization by 35% and shortens task response time by 40%. An ant colony algorithm reconstructs paths in real time, bypassing overloaded nodes and reducing task interruption rates to less than 1%. RS-coded redundant storage and sandbox isolation mechanisms collaborate to ensure data recovery success.

[0038] 2. The present invention links feature extraction S2 with scheduling strategy modeling S3. The 128-dimensional features extracted by S2 provide input for the multi-objective planning model of S3, enabling the model to simultaneously optimize response time, energy consumption and load balancing, thereby achieving multi-objective optimization capabilities. Strategy modeling S3 and dynamic adjustment S4 form a closed-loop feedback loop. The initial strategy generated by S3 adjusts the path in real time in S4 through the ant colony algorithm to bypass overloaded nodes. If the adjustment fails three times in a row, the strategy backtracking mechanism of S4c rolls back to the historical optimal strategy to avoid system crash. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 It is a schematic diagram of the overall structure of the present invention. DETAILED DESCRIPTION

[0040] In order to make the technical means, creative features, objectives and effects achieved by the present invention easier to understand, the present invention is further described below in conjunction with specific implementation methods.

[0041] Example 1, as Figure 1 As shown, a resource scheduling method based on data preprocessing includes the following steps:

[0042] S1, Data Collection and Cleaning: Real-time data collection from multiple sources of heterogeneous resources is performed through a distributed sensor network. The raw data is then filtered for noise, filled with missing values, and formatted in a standardized manner to eliminate noise, missing values, and inconsistencies in the raw data, providing high-quality input for subsequent analysis.

[0043] Specific S1 data collection and cleaning includes the following sub-steps:

[0044] S1 a, Multi-source data fusion: The collected data from multi-source heterogeneous resources (such as logs, sensor readings, network traffic) are uniformly encoded into a time series matrix to eliminate data dimension differences;

[0045] The above-mentioned multi-source heterogeneous resource data is heterogeneous data from computing nodes, storage devices and network links.

[0046] For example, align the CPU usage (Hz) and network latency (ms) of different sampling frequencies into a matrix with a unified timestamp;

[0047] The formula for constructing the timing matrix is:

[0048] M t×n =[x t1 ,x t2 ,...,x tn ]

[0049] Where t is the time window length, n is the data dimension, x ti is the value of the i-th dimension data at time t;

[0050] This eliminates the heterogeneity of multi-source data and facilitates subsequent unified processing;

[0051] S1 b, noise filtering: wavelet transform algorithm is used to separate high-frequency noise components and retain low-frequency effective signals;

[0052] In S1 b, Daubechies-4 is selected as the basis function of wavelet transform, the number of decomposition layers is 3, and the signal-to-noise ratio is improved by ≥20 dB;

[0053] The above wavelet decomposition and reconstruction involve the formula:

[0054]

[0055] Among them, d j,k is the j-th layer detail coefficient (high-frequency noise), α 3,k : Layer 3 approximation coefficient (valid signal).

[0056] S1 c, Missing value filling: Based on the data similarity of adjacent nodes, the missing fields are interpolated using the K-nearest neighbor algorithm. For example, if the CPU data of node A is missing, the average value of the five neighboring nodes with the closest load pattern is selected to fill the missing fields;

[0057] S1 d, normalization processing: Min-Max normalization is performed on the cleaned data to eliminate the dimension effect, thereby achieving unified data distribution and improving the stability of subsequent feature extraction.

[0058] S2, Feature Extraction and Prioritization: Based on preprocessed data, convolutional neural networks are used to extract resource load characteristics and dynamically prioritize tasks based on business needs. This allows key features to be extracted from preprocessed data and dynamically prioritized to optimize scheduling efficiency.

[0059] Specifically, feature extraction and priority division in S2 include the following sub-steps:

[0060] S2a, multi-dimensional feature construction: Extracts spatiotemporal correlation features of CPU usage, memory fluctuations, I / O throughput, and network latency from preprocessed data to generate a 128-dimensional feature vector that captures the dynamic changes in resources, improving the adaptability of scheduling strategies.

[0061] S2b, feature dimensionality reduction: compress feature dimensions using the principal component analysis (PCA) algorithm, retaining principal components with a variance contribution rate ≥ 95%;

[0062] The formula involved in principal component analysis is:

[0063] Z=X·W

[0064] Among them, X represents the original feature matrix, W represents the feature vector matrix, and Z represents the features after dimensionality reduction.

[0065] This reduces the feature dimension from 128 to 15, reducing computational complexity.

[0066] S2c, dynamic priority calculation: Generates priority weight vectors based on task urgency, resource dependencies, and quality of service agreements, and divides task queue levels;

[0067] In S2c, the priority weight vector is calculated using the entropy weight method, which combines the task deadline, data dependency graph, and user level parameters. The weight update frequency is once per second;

[0068] The formula involved in the above entropy weight method weight calculation is:

[0069]

[0070] Among them, E j It represents the entropy value of the j-th indicator, reflecting the degree of data dispersion.

[0071] S3, Scheduling Strategy Modeling: Based on the load characteristic analysis results, a multi-objective planning model is constructed with response time, resource utilization, and energy consumption as optimization targets, and a reinforcement learning algorithm is used to generate the initial scheduling strategy;

[0072] In the S3 scheduling strategy modeling, the objective function of the multi-objective planning model is:

[0073] min(λ1·T resp +λ2·E consum +λ3·C balance )

[0074] Among them, T resp is the average task response time, E consum is the system energy consumption, C balance is the resource load balancing degree, and λ1, λ2, and λ3 are dynamically adjusted weight coefficients.

[0075] The specific reinforcement learning algorithm uses a deep Q network (DQN) to generate the initial scheduling strategy, and optimizes the model through reward functions (such as task completion rate), thereby shortening the initial strategy generation time to 50ms and improving the ability to adapt to dynamic environments.

[0076] S4, dynamic policy adjustment: During the scheduling process, the resource status changes are monitored in real time. If load anomalies or node failures are detected, the policy reconstruction mechanism is triggered to update the scheduling path;

[0077] The specific S4 dynamic policy adjustment includes the following operations:

[0078] S4a, real-time load monitoring: uses a sliding window mechanism to count the instantaneous load rate of resource nodes. If it exceeds a preset threshold, it is marked as an overloaded node, further improving the accuracy of overload node identification;

[0079] S4b, path reconstruction: recalculate the task allocation path based on the ant colony algorithm to bypass overloaded and faulty nodes;

[0080] In S4b, the pheromone volatility coefficient of the ant colony algorithm is set to 0.6, and the path selection probability formula is:

[0081]

[0082] Among them, T ij is the path pheromone concentration, η ij is the path heuristic factor, α=1, β=2.

[0083] S4c, strategy backtracking: If the load is not relieved after three consecutive adjustments, roll back to the historical optimal strategy (such as the strategy with the highest load balancing degree) and trigger an alarm to avoid cascading failures.

[0084] S5, fault tolerance and security protection: including redundancy check and exception isolation mechanism to ensure data integrity and system stability during the scheduling process. If the resource conflict exceeds the threshold, the backup resource pool takeover task is initiated.

[0085] In S5 fault tolerance and security protection, redundancy check uses RS coding to achieve redundant storage of data blocks, and exception isolation uses a sandbox mechanism that collaborates with software and hardware to limit access rights to faulty nodes.

[0086] The above redundancy check involves the RS code generating polynomial, and the specific formula involved is:

[0087] G(x)=(x-α 0 )(x-α 1 )...(x-α n-k-1 )

[0088] Among them, α is the Galois field generator, n is the total number of blocks, and k is the number of data blocks.

[0089] Through the collaboration of data preprocessing S1 and feature extraction S2, through S1's noise filtering, missing value filling and normalization processing, the features extracted by S2 are ensured to have a high signal-to-noise ratio and consistency, avoiding feature deviations caused by data defects, thereby achieving high-quality data input;

[0090] The normalization processing of S1 d unifies the data dimension, enabling the principal component analysis (PCA) of S2b to efficiently compress the feature dimension (from 128 dimensions to 15 dimensions), reducing the complexity of subsequent model training and improving computational efficiency.

[0091] By linking feature extraction S2 with scheduling strategy modeling S3, the 128-dimensional features extracted by S2 (such as the spatiotemporal correlation of CPU and network delays) provide input to S3's multi-objective planning model, enabling the model to simultaneously optimize response time, energy consumption, and load balancing, achieving multi-objective optimization capabilities.

[0092] S2c's entropy weight method updates the priority weights every second, and combined with S3's reinforcement learning (DQN), it generates scheduling strategies that adapt to dynamic environments in real time.

[0093] Strategy modeling S3 and dynamic adjustment S4 form a closed-loop feedback loop. The initial strategy generated by S3 is used in S4 to adjust the path in real time using an ant colony algorithm (pheromone volatility coefficient = 0.6) to bypass overloaded nodes. If the adjustment fails three times in a row, the strategy backtracking mechanism in S4c rolls back to the historical optimal strategy to avoid system crashes.

[0094] S3's multi-objective model optimizes resource allocation, and S4's dynamic adjustment further ensures that resource utilization remains high despite load fluctuations.

[0095] Each step is globally protected with fault-tolerant protection S5. S5's RS-coded redundant storage ensures that even if some nodes fail, data can still be recovered from redundant blocks. S5's sandbox mechanism isolates faulty nodes and works in conjunction with S4's path reconstruction to prevent the spread of anomalies.

[0096] During the operation of this embodiment, data preprocessing and multi-algorithm collaboration achieved comprehensive improvements in resource scheduling efficiency and stability. Multi-source data fusion and noise filtering based on a distributed sensor network increased the data signal-to-noise ratio by ≥20dB, eliminating differences in heterogeneous data. A convolutional neural network was combined to extract 128-dimensional spatiotemporal features, which were compressed to 15 dimensions through principal component analysis, significantly reducing computational complexity. Dynamic prioritization, coupled with a multi-objective planning model, increased resource utilization by 35% and shortened task response time by 40%. An ant colony algorithm reconstructed paths in real time, bypassing overloaded nodes and achieving a task interruption rate of less than 1%. RS-coded redundant storage and sandbox isolation mechanisms collaborated to ensure a data recovery success rate of ≥99.99%, resulting in a system availability of 99.9%.

[0097] The basic principles, main features, and advantages of the present invention are shown and described above. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The above embodiments and descriptions are merely illustrative of the principles of the present invention. Various changes and modifications may be made to the present invention without departing from the spirit and scope of the present invention. Such changes and modifications are intended to fall within the scope of the present invention. The scope of protection claimed in the present invention is defined by the appended claims and their equivalents.

Claims

1. A resource scheduling method based on data preprocessing, characterized in that: The following steps are involved: S1, data collection and cleaning: Real-time collection of multi-source heterogeneous resource data through distributed sensor networks, noise filtering, missing value filling and format standardization of the raw data; S2, Feature Extraction and Prioritization: Based on preprocessed data, convolutional neural networks are used to extract resource load features and dynamically prioritize tasks based on business needs. S3, Scheduling Strategy Modeling: Based on the load characteristic analysis results, a multi-objective planning model is constructed and the initial scheduling strategy is generated using a reinforcement learning algorithm; S4, dynamic policy adjustment: During the scheduling process, the resource status changes are monitored in real time. If load anomalies or node failures are detected, the policy reconstruction mechanism is triggered to update the scheduling path; S5, fault tolerance and security protection: includes redundancy checking and exception isolation mechanisms. If the resource conflict exceeds the threshold, the backup resource pool takeover task is initiated.

2. The resource scheduling method based on data preprocessing according to claim 1, characterized in that: The S1 data collection and cleaning specifically includes the following sub-steps: S1 a, Multi-source data fusion: The collected data from multi-source heterogeneous resources are uniformly encoded into a time series matrix to eliminate data dimension differences; S1 b, noise filtering: wavelet transform algorithm is used to separate high-frequency noise components and retain low-frequency effective signals; S1 c, missing value filling: based on the data similarity of adjacent nodes, the missing fields are interpolated using the K-nearest neighbor algorithm; S1 d, normalization processing: Min-Max normalization is performed on the cleaned data to eliminate the dimension effect.

3. The resource scheduling method based on data preprocessing according to claim 1, characterized in that: The feature extraction and priority division in S2 specifically includes the following sub-steps: S2a, multi-dimensional feature construction: extracting spatiotemporal correlation features of CPU usage, memory fluctuation, I / O throughput, and network latency from preprocessed data; S2b, feature dimensionality reduction: compressing feature dimensions through principal component analysis algorithm; S2c, dynamic priority calculation: Generate a priority weight vector based on task urgency, resource dependencies, and quality of service agreements to divide task queue levels.

4. The resource scheduling method based on data preprocessing according to claim 1, characterized in that: In the S3 scheduling strategy modeling, the objective function of the multi-objective programming model is: min(λ1·T resp +λ2·E consum +λ3·C balance ) Among them, T resp is the average task response time, E consum is the system energy consumption, C balance is the resource load balancing degree, and λ1, λ2, and λ3 are dynamically adjusted weight coefficients.

5. The resource scheduling method based on data preprocessing according to claim 1, characterized in that: The S4 dynamic policy adjustment specifically includes the following operations: S4a, real-time load monitoring: Through the sliding window mechanism, the instantaneous load rate of the resource node is counted. If it exceeds the preset threshold, it is marked as an overloaded node; S4b, path reconstruction: recalculate the task allocation path based on the ant colony algorithm to bypass overloaded and faulty nodes; S4c, strategy backtracking: If the load is not relieved after three consecutive adjustments, roll back to the historical optimal strategy and trigger an alarm.

6. The resource scheduling method based on data preprocessing according to claim 2, characterized in that: In the above-mentioned S1 b, Daubechies-4 is selected as the basis function of wavelet transform.

7. The resource scheduling method based on data preprocessing according to claim 3, characterized in that: In S2c, the priority weight vector is calculated using an entropy weight method, combined with task deadlines, data dependency graphs, and user level parameters, and the weight update frequency is once per second.

8. The resource scheduling method based on data preprocessing according to claim 5, characterized in that: In S4b, the pheromone volatility coefficient of the ant colony algorithm is set to 0.6, and the path selection probability formula is: Among them, T ij is the path pheromone concentration, η ij is the path heuristic factor, α=1, β=2.

9. The resource scheduling method based on data preprocessing according to claim 1, characterized in that: In the S5 fault tolerance and security protection, redundancy check uses RS coding to achieve redundant storage of data blocks, and abnormal isolation limits the access rights of faulty nodes through a sandbox mechanism that collaborates with software and hardware.

Images