An intelligent computing power and storage scheduling method and system of a multi-service system

Abstract

The application relates to an intelligent computing power and storage scheduling method and system of a multi-service system, and belongs to the technical field of intelligent computing resource scheduling and storage management. The method comprises the following steps: collecting real-time monitoring data of each node in the multi-service system and performing feature extraction; predicting a computing power demand value and a storage demand hierarchical strategy in a future preset time window; dynamically correcting the prediction result to generate a dynamic resource demand table and a cross-service priority weight matrix; generating a task allocation scheme through a hybrid scheduling algorithm; generating a storage data migration instruction through a storage scheduling engine; performing computing node expansion, task migration and storage data redistribution operations; updating weight parameters of a time sequence prediction model and a scheduling strategy rule library based on a latest state snapshot of a resource pool after execution and a scheduling failure case library, obtaining an updated strategy version and applying the updated strategy version to a next scheduling period. The application can realize efficient scheduling of resources in a multi-service system environment.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of intelligent computing resource scheduling and storage management, and particularly relates to an intelligent computing power and storage scheduling method and system for a multi-service system. BACKGROUND

[0002] With the continuous development of information technology, technologies such as cloud computing, big data, high-performance computing, and virtualization have gradually become the core components in modern computing architecture. In particular, in a multi-service system, as computing tasks become increasingly complex, business demands also become more and more diverse. How to effectively manage system resources has become the key to improving overall performance and efficiency. Resource scheduling and management, as a basic component in these systems, directly affects the execution efficiency of tasks, the stability of the system, and the utilization rate of resources.

[0003] A multi-service system usually runs multiple types of tasks simultaneously. These tasks not only have different requirements in computing power, but also show great differences in the use of storage resources. Computing-intensive tasks often require high computing power, while storage-intensive tasks have high requirements for storage I / O throughput. In addition to the differences in task types, different tasks also have different priorities and resource consumption patterns, which makes the resource scheduling problem more complex. The system needs to be able to perceive these changes in real time and make corresponding adjustments to ensure that high-priority tasks are processed in a timely manner, while avoiding low-priority tasks from occupying too many resources.

[0004] Currently, in common resource scheduling methods, many systems rely on static configuration or rule-based scheduling strategies. For example, computing nodes may be assigned to fixed tasks, and storage resources may be allocated according to pre-set rules. However, these methods often fail to effectively cope with complex business demands and dynamic changes in resource usage. When the system load changes, or the resource requirements of tasks exceed expectations, these static scheduling methods can lead to resource overload or waste, thereby affecting the overall performance and efficiency of the system. For example, excessive load on computing nodes can cause task execution delays, while uneven allocation of storage resources can lead to excessive use or idling of some storage media, wasting valuable hardware resources.

[0005] Therefore, how to accurately predict the future computing power and storage requirements of the system, and thus achieve efficient resource scheduling in a multi-service system environment, is a problem that needs to be solved at present. SUMMARY

[0006] In order to achieve efficient resource scheduling in a multi-service system environment, the present application provides an intelligent computing power and storage scheduling method and system for a multi-service system.

[0007] In a first aspect, the application provides an intelligent computing power and storage scheduling method for a multi-service system, which adopts the following technical solution:

[0008] An intelligent computing power and storage scheduling method for a multi-service system, the method comprising:

[0009] Collecting real-time monitoring data of each node in the multi-service system;

[0010] Extracting features from the real-time monitoring data to generate a structured feature vector containing service type labels, task priority, and data access patterns, thereby obtaining a real-time service feature dataset;

[0011] Based on the real-time service feature dataset and historical load logs, predicting computing power demand values and storage demand hierarchical strategies within a future preset time window through a time series prediction model;

[0012] According to the computing power demand values and storage demand hierarchical strategies, dynamically correcting the prediction results by combining a reinforcement learning algorithm to generate a dynamic resource demand table and a cross-service priority weight matrix;

[0013] Based on the preset rules in the service type initialization scheduling strategy rule library;

[0014] Based on the preset rules and resource pool topology information, generating a task allocation scheme through a hybrid scheduling algorithm according to the dynamic resource demand table and the cross-service priority weight matrix;

[0015] Based on the storage demand hierarchical strategy, generating storage data migration instructions through a storage scheduling engine;

[0016] Sending the task allocation scheme and storage data migration instructions to a virtualized resource orchestration module to perform computing node expansion, task migration, and storage data redistribution operations;

[0017] Collecting resource pool state data after execution to generate a resource pool latest state snapshot;

[0018] Based on the resource pool latest state snapshot and a scheduling failure case library, updating the weight parameters of the time series prediction model and the scheduling strategy rule library to obtain an updated strategy version and apply it to the next scheduling cycle.

[0019] By adopting the technical solution, intelligent scheduling of multi-service system resources is realized. By deeply combining real-time monitoring data and prediction algorithms, the system can dynamically adjust resource allocation, improve resource utilization, reduce system bottlenecks, and at the same time, ensure the efficiency of task execution and the stability of the system. In addition, through adaptive optimization strategy, the system can continuously improve the scheduling decision, so that the scheduling effect after a long time running is more accurate and efficient, thereby further improving the performance and reliability of the entire multi-service system.

[0020] Optionally, the step of generating a dynamic resource demand table and a cross-service priority weight matrix by dynamically correcting the prediction result in combination with a reinforcement learning algorithm according to the computing power demand value and the storage demand hierarchical strategy comprises:

[0021] obtaining real-time resource pool state data and receiving a computing power demand value and a storage demand hierarchical strategy in a future preset time window output by a time series prediction model;

[0022] combining the computing power demand value, the storage demand hierarchical strategy and the real-time resource pool state data into a composite state vector, and performing normalization processing to generate a standardized state matrix;

[0023] defining an action space parameter according to a business type, and inputting the action space parameter into a pre-configured reinforcement learning agent; wherein the action space parameter comprises a resource allocation proportion adjustment instruction and a storage migration trigger instruction;

[0024] calculating the resource allocation proportion adjustment instruction and the storage migration trigger instruction based on the standardized state matrix and historical scheduling records through the reinforcement learning agent; wherein the reinforcement learning agent evaluates the action value according to a preset reward function, and updates the policy network parameter;

[0025] generating a dynamic resource demand table according to the resource allocation proportion adjustment instruction in combination with a preset business feature library;

[0026] constructing a cross-service priority weight matrix based on the dynamic resource demand table and the storage migration trigger instruction.

[0027] By adopting the technical solution, the dynamic resource scheduling and priority decision method based on reinforcement learning can automatically and intelligently handle the resource allocation problem in a multi-service system. By combining real-time resource pool state, time series prediction data and reinforcement learning algorithm, the system can flexibly adapt to changes in business load and make efficient and accurate resource scheduling decisions.

[0028] Optionally, after the step of constructing a cross-service priority weight matrix based on the dynamic resource demand table and the storage migration trigger instruction, the method further comprises:

[0029] input the dynamic resource demand table and the cross-service priority weight matrix into a digital twin simulation environment to generate a simulation scheduling result;

[0030] According to the error rate of the actual scheduling result and the simulation scheduling result, trigger the incremental learning module to update the policy network parameters of the reinforcement learning agent.

[0031] By adopting the above technical solution, the dynamic resource demand table and the cross-service priority weight matrix are input into the digital twin simulation environment for simulation, and the incremental learning module is triggered for policy optimization according to the error rate of the actual scheduling result and the simulation scheduling result, further strengthening the intelligent scheduling capability of the system in a multi-service environment. The use of digital twin simulation enables the system to test and verify various scheduling strategies in a virtual environment, thereby avoiding unforeseen errors in the production environment. Through the incremental learning module, the system can continuously optimize the decision-making strategy according to new data during actual operation, ensuring that resource scheduling can always meet the efficiency and flexibility requirements in various complex service scenarios. The overall technical solution can significantly improve resource utilization, reduce task delay, and improve system stability and overall performance, adapting to the constantly changing multi-service environment.

[0032] Optionally, initializing the preset rules in the scheduling strategy rule library based on the service type includes:

[0033] When the service type is a real-time task, configure pre-emptive scheduling rules and storage bandwidth reservation thresholds; or, when the service type is an offline task, configure elastic resource pool allocation rules and cold data storage migration strategies.

[0034] By adopting the above technical solution, customized scheduling rules are formulated for each task type based on the different needs of real-time tasks and offline tasks. Real-time tasks use pre-emptive scheduling rules and bandwidth reservation strategies to ensure their priority execution under high load; offline tasks optimize resource utilization through elastic resource pools and cold data migration strategies. The scheduling strategy library of the system is not only initialized according to business needs at the beginning, but also dynamically adjusted according to feedback information in each subsequent scheduling period, ensuring that the strategy always meets the current resource and business needs.

[0035] Optionally, based on the preset rules and resource pool topology information, according to the dynamic resource demand table and the cross-service priority weight matrix, the task allocation scheme is generated by a hybrid scheduling algorithm, including:

[0036] Obtain the dynamic resource demand table, the cross-service priority weight matrix, the resource pool topology information, and the preset rules;

[0037] Perform time window segmentation and normalization processing on the dynamic resource demand table to generate a standardized resource demand time slice;

[0038] construct a weighted directed graph based on the resource pool topology information, and generate a topology graph structure file;

[0039] classify and weight the standardized resource demand time slices according to a preset rule;

[0040] combine the cross-service priority weight matrix and the topology graph structure file to calculate a matching degree score of tasks and nodes, generate a candidate node score table and a rule conflict marker list;

[0041] generate a final task allocation scheme through a heuristic algorithm based on the candidate node score table and the rule conflict marker list.

[0042] By adopting the above technical solutions, the system can efficiently allocate tasks and resources under the condition of dynamic resource demand and service priority changes, optimize system performance, and ultimately realize accurate task scheduling and resource utilization, ensuring efficient coordination and dynamic adaptation of multi-dimensional resources.

[0043] Optionally, the step of generating a final task allocation scheme through a heuristic algorithm based on the candidate node score table and the rule conflict marker list includes:

[0044] obtaining a candidate node score table, a rule conflict marker list, and resource pool topology information;

[0045] filtering assignable node white lists for each task according to the candidate node score table, and excluding illegal node combinations in the rule conflict marker list;

[0046] constructing an initial population set based on the assignable node white lists; wherein each population individual adopts chromosome coding to represent a task allocation scheme;

[0047] constructing a multi-objective fitness function based on the candidate node score table and communication delay parameters and node load data in the resource pool topology information;

[0048] dynamically adjusting weight coefficients of each optimization objective in the multi-objective fitness function according to real-time resource pool state data;

[0049] performing a tournament selection operation on the initial population set to generate a parent population;

[0050] performing a topology-aware crossover operation on parent chromosomes in the parent population based on a communication path table in the resource pool topology information to generate an offspring population;

[0051] dynamically adjusting a mutation rate according to a population diversity evaluation result, and performing a probability mutation on offspring chromosomes in the offspring population based on the candidate node score table;

[0052] Performing local search optimization on chromosomes with fitness scores higher than a preset threshold value;

[0053] Filtering individuals with optimal fitness from the parent population and the offspring population to form an elite population;

[0054] When the convergence condition is met, decoding the chromosome with the highest fitness in the elite population as the final task allocation scheme.

[0055] By adopting the above technical solutions, the global optimization of the task allocation scheme is realized based on the combination of heuristic algorithms and genetic algorithms, the efficiency and quality of task allocation are improved, and finally the optimal task allocation scheme can be generated, ensuring efficient use of system resources, reducing communication costs, and achieving load balancing optimization.

[0056] Optionally, updating the weight parameters of the time series prediction model and the scheduling strategy rule library based on the latest state snapshot of the resource pool and the scheduling failure case library to obtain an updated strategy version includes:

[0057] According to the latest state snapshot of the resource pool, the node load rate, the storage distribution state, and the task execution progress are obtained;

[0058] The node load rate is dynamically normalized to generate a standardized load rate matrix;

[0059] The storage distribution state is topologically modeled to generate a storage resource distribution heat map;

[0060] The task execution progress is time series aligned to generate a task progress time series table;

[0061] Based on the standardized load rate matrix and the task progress time series table, the time series prediction model is incrementally trained to update the model weight parameters;

[0062] According to the storage resource distribution heat map, the storage constraint weight in the loss function of the time series prediction model is dynamically adjusted;

[0063] Combining the task progress time series table and the scheduling failure case library, the conflict rules in the scheduling strategy rule library that are contradictory to the task execution progress are detected;

[0064] The conflict rules are conditionally threshold recalibrated and dynamically sorted by execution priority to generate an optimized scheduling strategy rule library;

[0065] Based on the storage resource distribution heat map and the task execution progress, the hot and cold data migration path optimization of the storage demand hierarchical strategy is performed to obtain an updated storage hierarchical configuration file;

[0066] The updated time series prediction model, the optimized scheduling strategy rule library and the storage hierarchy configuration file are packaged as an updated strategy version.

[0067] By adopting the technical solution, the continuous optimization of the scheduling strategy is ensured, the load, storage and task scheduling strategy of the system are adjusted in real time in a data-driven manner, the system resource utilization is improved, the task execution delay is reduced, and the stable operation of the system is ensured.

[0068] In a second aspect, the application provides an intelligent computing power and storage scheduling system for a multi-service system, which adopts the following technical solution:

[0069] An intelligent computing power and storage scheduling system for a multi-service system, the system comprising:

[0070] A data acquisition module for acquiring real-time monitoring data of each node in the multi-service system;

[0071] A feature extraction module for extracting features from the real-time monitoring data to generate a structured feature vector containing service type labels, task priorities and data access patterns, thereby obtaining a real-time service feature dataset;

[0072] A prediction module for predicting computing power demand values and storage demand hierarchical strategies within a future preset time window based on the real-time service feature dataset and historical load logs through a time series prediction model;

[0073] A dynamic correction module for dynamically correcting the prediction results based on the computing power demand values and storage demand hierarchical strategies in combination with a reinforcement learning algorithm to generate a dynamic resource demand table and a cross-service priority weight matrix;

[0074] A rule initialization module for initializing preset rules in a scheduling strategy rule library based on service types;

[0075] A task allocation module for generating a task allocation scheme through a hybrid scheduling algorithm based on the dynamic resource demand table and the cross-service priority weight matrix according to the preset rules and resource pool topology information;

[0076] A storage data migration module for generating storage data migration instructions through a storage scheduling engine based on the storage demand hierarchical strategy;

[0077] An execution module for sending the task allocation scheme and the storage data migration instructions to a virtualized resource orchestration module to perform computing node expansion, task migration and storage data redistribution operations;

[0078] The state data collection module is configured to collect resource pool state data after execution, and generate a resource pool latest state snapshot; the resource pool latest state snapshot comprises node load rate, storage distribution state, and task execution progress;

[0079] The policy version updating module is configured to update the weight parameters of the time series prediction model and the scheduling policy rule library based on the resource pool latest state snapshot and the scheduling failure case library, obtain an updated policy version, and apply the updated policy version to a next scheduling period.

[0080] In a third aspect, the present application provides a computer device, which adopts the technical scheme as follows:

[0081] The computer device comprises a memory, a processor, and a computer program stored in the memory, and the processor executes the computer program to implement the steps of the method according to the first aspect.

[0082] In a fourth aspect, the present application provides a computer readable storage medium, which adopts the technical scheme as follows:

[0083] The computer readable storage medium stores a computer program capable of being loaded and executed by a processor to implement any one of the methods according to the first aspect.

[0084] In summary, the present application has at least one of the following beneficial technical effects: through real-time monitoring data and feature extraction, the system can dynamically predict the demand for computing power and storage, thereby generating accurate resource demand table and priority matrix. The prediction results are corrected by reinforcement learning, the task allocation and storage data migration strategy are optimized, and through the hybrid scheduling algorithm and storage scheduling engine, the system ensures that the tasks and data can be intelligently allocated and migrated to improve the resource utilization. Finally, based on the real-time feedback data, the prediction model and the scheduling strategy are updated to form a continuous optimization scheduling closed loop, which significantly improves the scheduling efficiency, resource utilization and stability of the system. BRIEF DESCRIPTION OF DRAWINGS

[0085] Figure 1 FIG. 1 is a first flowchart of an intelligent computing power and storage scheduling method of a multi-service system according to one embodiment of the present application.

[0086] Figure 2 FIG. 2 is a second flowchart of an intelligent computing power and storage scheduling method of a multi-service system according to one embodiment of the present application.

[0087] Figure 3 FIG. 3 is a third flowchart of an intelligent computing power and storage scheduling method of a multi-service system according to one embodiment of the present application.

[0088] Figure 4FIG. 6 is a sixth flowchart of an intelligent computing and storage scheduling method of a multi-service system according to an embodiment of the present application.

[0089] Figure 5 FIG. 5 is a fifth flowchart of an intelligent computing and storage scheduling method of a multi-service system according to an embodiment of the present application.

[0090] Figure 6 FIG. 6 is a sixth flowchart of an intelligent computing and storage scheduling method of a multi-service system according to an embodiment of the present application. DETAILED DESCRIPTION

[0091] For the purpose, technical solutions and advantages of the present application to be clearer, the following will further describe the present application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and not to limit the present application. Figures 1-6

[0092] The embodiments of the present application disclose an intelligent computing and storage scheduling method of a multi-service system.

[0093] Referring to Figure 1 , an intelligent computing and storage scheduling method of a multi-service system, the specific method comprises:

[0094] Step S101, collecting real-time monitoring data of each node in the multi-service system;

[0095] Among them, in the multi-service system, each node (such as a computing node, a storage node) will usually generate a large amount of monitoring data in real time, which includes CPU utilization, GPU utilization, storage I / O throughput, network delay and task queue state, etc. These real-time monitoring data are intuitive reflection of system running state, which can provide key basis for subsequent computing power demand and storage demand prediction.

[0096] For example, high CPU utilization may mean that the computing capacity of the node is close to saturation, and measures need to be taken for load balancing; while network delay may affect the efficiency of task execution, thereby affecting the performance of the entire system. Real-time monitoring data collection is usually achieved through monitoring software or hardware tools installed on each node of the system, which can ensure continuous acquisition of system running data. In this way, the system can respond to load changes in real time, thereby effectively avoiding resource bottlenecks and task execution delays.

[0097] It can be understood that after collecting these data, the system can realize comprehensive monitoring of the real-time state of each node, ensuring that the subsequent scheduling algorithm can make reasonable decisions based on the current load and demand. Through real-time data collection, the system can flexibly cope with changing business demands and ensure optimal allocation of resources. ​

[0098] Step S102, feature extraction is performed on the real-time monitoring data to generate a structured feature vector containing business type labels, task priority and data access patterns, and a real-time business feature dataset is obtained;

[0099] The feature extraction process is to analyze and process the collected real-time monitoring data to construct a data structure that can reflect the system working state and task characteristics. Feature extraction is not limited to extracting common indicators such as CPU or GPU utilization from raw monitoring data, but also generates specific feature labels for each task according to business requirements. These labels may include task priority, business type, data access pattern, etc. In this process, the priority of the task (such as high, medium, low) and the data access pattern (such as frequent access, occasional access, cold data, etc.) become the core content of feature extraction. These features can help the system understand the resource requirements and running state of each task in more detail.

[0100] In the embodiments of the present application, the structured feature vector generated by feature extraction of the monitoring data can clearly describe the specific requirements of each task or node. The real-time business feature dataset composed of these feature vectors will provide accurate input data for the subsequent prediction model. For example, in a system, the feature vector of a certain task may include the following information: CPU utilization 90%, GPU utilization 30%, task priority high, and data access pattern frequent. This feature vector can accurately support subsequent resource scheduling.

[0101] Step S103, based on the real-time business feature dataset and the historical load log, the time series prediction model is used to predict the computing power demand value and the storage demand hierarchical strategy in the future preset time window;

[0102] In one of the embodiments of the present application, the time series prediction model can use a long short-term memory network (LSTM) as a deep learning model specifically for processing time series data. Based on the historical load log and the real-time business feature dataset, LSTM can predict the computing power demand and storage demand of the system in the future preset time window. The core idea of the time series prediction model is to capture the potential temporal dependence in the data through learning from historical data, so as to predict future demand. This process requires the use of information containing task type, priority, resource consumption, etc. to help the model make predictions. For example, if the computing demand of certain tasks appears periodic fluctuations in the past few hours, the LSTM model will predict the demand of similar tasks in the future according to this rule.

[0103] Specifically, in predicting storage requirements, the system analyzes data access frequency and classifies storage requirements into tiers such as hot data, warm data, and cold data. Hot data is accessed frequently and requires high-performance storage; cold data is accessed less frequently and can be stored on cheaper storage media. According to these tiering strategies, the system allocates resources based on the performance characteristics of storage media (such as SSD, HDD, object storage). By dividing data into different tiers, the system can optimize the use of storage resources and avoid over-allocation.

[0104] As an implementation of the time series prediction model, a deep learning framework can be used to build an LSTM model, which typically consists of an input layer, an LSTM layer, a fully connected layer, and an output layer. In the training phase, the Adam optimizer (learning rate = 0.001) and the MSE loss function are used, and the early stopping mechanism (patience = 10 rounds) is introduced to prevent over-training. When predicting, a sliding window recursion is used to generate multi-step prediction results, and MinMaxScaler inverse transformation is used to restore the actual dimension. This architecture achieves a prediction accuracy of RMSE ≤ 5% on the test set, and supports dynamic loading of new data for incremental training to adapt to power consumption mode drift.

[0105] Step S104, according to the computing power demand value and the storage demand tiering strategy, combining the reinforcement learning algorithm to dynamically correct the prediction results, and generating a dynamic resource demand table and a cross-business priority weight matrix;

[0106] The obtained prediction results are only estimates of future computing power demand and storage demand, but these estimates need to be further optimized and corrected. Reinforcement learning is an algorithm that optimizes decision-making processes through rewards and penalties. In this step, the reinforcement learning algorithm will combine the computing power demand and storage demand tiering strategy to dynamically adjust resource demand for different tasks and businesses. For example, some high-priority tasks may require more computing power resources, while other low-priority tasks can be delayed when resources are tight.

[0107] Specifically, the dynamic resource demand table will be updated in real time based on the priority of different tasks, resource demand, historical scheduling results, and other information, and a cross-business priority weight matrix will be calculated. This matrix helps the system decide which tasks should be given priority to resources to ensure that high-priority tasks are not disturbed by low-priority tasks. At the same time, the cross-business priority weight matrix can handle the competition between multiple tasks, and by reasonably allocating computing resources and storage resources, the overall efficiency of the system can be maximized.

[0108] Step S105, based on the type of business, initialize the preset rules in the scheduling strategy rule library;

[0109] The scheduling strategy rule library is initialized according to the business type, and corresponding scheduling rules are set for different types of businesses. The scheduling strategy rule library contains multiple preset rules, which are set according to historical experience and business requirements.

[0110] For example, for real-time tasks or offline tasks, the system can process according to different preset rules. Real-time tasks can be prioritized through the configuration of preemptive scheduling rules, and still be guaranteed when system resources are insufficient, avoiding delays caused by insufficient bandwidth and other problems; offline tasks can reduce the pressure on storage resources through the migration strategy of cold data storage, improving the overall resource utilization of the system. Through these rules, the system can maintain flexibility when processing multiple tasks and ensure that each task is executed on the most suitable resource node, thereby improving the processing capacity and resource utilization of the system.

[0111] Step S106, based on the preset rules and resource pool topology information, according to the dynamic resource demand table and the cross-business priority weight matrix, a task allocation scheme is generated through a hybrid scheduling algorithm;

[0112] The core of the hybrid scheduling algorithm is to combine the advantages of different scheduling algorithms, considering multiple factors such as computing power demand, storage demand, and resource pool topology structure, to develop the optimal task allocation scheme. The resource pool topology structure includes the physical location mapping of computing nodes and storage nodes, network bandwidth, and communication overhead between nodes. The system uses the hybrid scheduling algorithm to reasonably arrange the allocation of tasks, ensuring that tasks can be executed on the best nodes, while also avoiding excessive communication delay and resource contention.

[0113] Specifically, the hybrid scheduling algorithm optimizes task allocation by considering multiple factors such as task priority, resource availability, network delay, etc., improving resource utilization and reducing system bottlenecks. In addition, the algorithm can be flexibly adjusted according to changes in resource pool topology, ensuring that resources can be most effectively allocated according to actual needs.

[0114] Step S107, based on the storage demand hierarchical strategy, a storage data migration instruction is generated through a storage scheduling engine;

[0115] In terms of storage management, the system uses a storage scheduling engine to generate data migration instructions based on the hierarchical strategy of storage demand. The storage demand hierarchical strategy divides the storage hierarchy of data according to the data access frequency of tasks.

[0116] In the embodiments of the present application, hot data accessed at high frequency can be stored on a high-performance SSD storage, while cold data is migrated to a low-price HDD storage or object storage. In this way, the system can ensure the access speed of hot data and reduce unnecessary overhead of storage media. At the same time, the storage scheduling engine can also merge duplicate data blocks through the cross-service data fingerprint library, thereby reducing redundant storage and releasing storage space.

[0117] In step S108, the task allocation scheme and the storage data migration instruction are sent to the virtualized resource orchestration module to perform the computing node expansion, task migration, and storage data redistribution operations.

[0118] The task allocation scheme and the storage data migration instruction are sent to the virtualized resource orchestration module to perform the computing node expansion, task migration, and storage data redistribution operations. This process is an automatic expansion of computing nodes and storage nodes by the system according to the current load condition to ensure that the resources can meet the task requirements. For example, when the load of the computing node is too high, the system may automatically expand new computing nodes or migrate part of the tasks from the busy computing node to the node with lower load.

[0119] In addition, the storage data migration instruction can ensure that the data is migrated to the most suitable storage node to avoid storage bottlenecks. The resource pool state data after execution will be collected to generate the latest resource pool state snapshot, which includes the node load rate, storage distribution state, and task execution progress, etc.

[0120] It can be understood that dynamically adjusting the computing resources and storage resources according to the changes in task load ensures that the system can flexibly cope with different workloads and avoid overuse of resources or insufficient resources. Through automatic expansion and task migration, the system can maintain efficient operation and reduce human intervention.

[0121] In step S109, the resource pool state data after execution is collected to generate the latest resource pool state snapshot.

[0122] The latest resource pool state snapshot includes the node load rate, storage distribution state, and task execution progress.

[0123] Specifically, after the task migration and resource expansion operations are completed, the system will collect the resource pool state data after execution to generate the latest resource pool state snapshot containing the node load rate, storage distribution state, and task execution progress.

[0124] Further, through analysis of these data, the system can evaluate the effectiveness of scheduling decisions, ensuring that resource allocation and task execution meet expectations. Node load rate can reflect resource utilization, storage distribution state can reveal whether storage resources have been reasonably allocated, and task execution progress can reflect whether tasks are executed smoothly according to the plan.

[0125] Step S110, based on the latest state snapshot of the resource pool and the scheduling failure case library, update the weight parameters of the time series prediction model and the scheduling strategy rule library, get the updated strategy version and apply it to the next scheduling period.

[0126] Among them, the strategy version includes the version identification of the time series prediction model weight file, the scheduling rule library entry and the storage layering strategy configuration file. The strategy version is synchronized to all scheduling nodes of multiple business systems through the distributed configuration center.

[0127] Specifically, the system will update the weight parameters of the time series prediction model based on the latest resource pool state snapshot and the scheduling failure case library, and optimize the scheduling strategy rule library. This process continuously improves the prediction ability of the system through methods such as incremental training, so that the system can improve the scheduling strategy according to historical experience after each scheduling period. For example, when task migration fails, the system can analyze the reasons for the failure and adjust the task migration rules according to the records in the scheduling failure case library to avoid similar situations from happening again.

[0128] In the above implementation, intelligent scheduling of resources of multiple business systems is achieved. By deeply combining real-time monitoring data and prediction algorithms, the system can dynamically adjust resource configuration, improve resource utilization, reduce system bottlenecks, and at the same time guarantee the efficiency of task execution and the stability of the system. In addition, through adaptive optimization strategy, the system can continuously improve the scheduling decision, so that the scheduling effect after a long time running is more accurate and efficient, thereby further improving the performance and reliability of the entire multi-business system.

[0129] Reference Figure 2 As an embodiment of step S104, the step of generating a dynamic resource demand table and a cross-business priority weight matrix according to the computing power demand value and the storage demand layering strategy, combined with the reinforcement learning algorithm to dynamically correct the prediction result includes:

[0130] Step S201, obtain real-time resource pool state data, and receive the computing power demand value and the storage demand layering strategy output by the time series prediction model in the future preset time window;

[0131] The real-time resource pool state data includes node load rate, storage bandwidth occupancy rate, and task queue length, which provide the resource utilization of each computing node and storage node in the system. The node load rate reflects the current load of the computing node, the storage bandwidth occupancy rate indicates the utilization degree of the storage device, and the task queue length reveals the queuing situation of the current system task, which is of great significance for evaluating whether the resources are sufficient and whether the task is delayed.

[0132] In step S202, the computing power demand value, the storage demand hierarchical strategy, and the real-time resource pool state data are combined into a composite state vector and normalized to generate a standardized state matrix.

[0133] The obtained computing power demand value, storage demand hierarchical strategy, and real-time resource pool state data are integrated to form a composite state vector. This composite state vector integrates information in multiple dimensions, such as future computing power demand, storage level demand, and current node load. Through normalization, data of different dimensions (such as load rate and demand value) have the same scale, avoiding bias in the learning process due to differences in data scale. The standardized state matrix is a consistent format representation of these composite state vectors for processing by the reinforcement learning agent.

[0134] In step S203, the action space parameters are defined according to the business type, and the action space parameters are input into the pre-configured reinforcement learning agent. The action space parameters include resource allocation ratio adjustment instructions and storage migration trigger instructions.

[0135] The resource allocation ratio adjustment instruction is a continuous action space, which represents the adjustment of the computing resource allocation ratio for each business type. For example, the system may allocate more resources to high-priority businesses based on current load and demand prediction. The storage migration trigger instruction is a discrete action space that defines the trigger threshold for hot data and cold data migration, helping the system decide when to perform data migration and optimize the use of storage resources. These action space parameters will be input into the pre-configured reinforcement learning agent to guide the reinforcement learning agent to select the most appropriate action (i.e., resource allocation adjustment and storage migration decision).

[0136] In step S204, the resource allocation ratio adjustment instruction and the storage migration trigger instruction are calculated by the reinforcement learning agent based on the standardized state matrix and historical scheduling records. The reinforcement learning agent evaluates the action value according to the preset reward function and updates the policy network parameters.

[0137] The reinforcement learning agent can use a proximal policy optimization algorithm to train the policy network, and the input layer of the policy network fuses the stored hierarchical policy parameters, and the output layer is mapped to the action space. The reinforcement learning algorithm can efficiently and stably update the policy network, so that the agent can make more accurate resource scheduling decisions in actual operation.

[0138] Specifically, the reinforcement learning agent calculates resource allocation ratio adjustment instructions and storage migration trigger instructions based on the standardized state matrix and historical scheduling records. The core of reinforcement learning is to guide the agent's decision-making through "rewards". The agent evaluates the value of each action according to the defined reward function, and continuously updates its decision-making rules through the policy network.

[0139] In some embodiments, the reward function evaluates the following aspects: resource utilization reward (based on the ratio of actual computing power to predicted computing power demand), SLA (service level agreement) compliance reward (based on the timely completion of tasks and the number of delays of high-priority tasks), storage efficiency penalty (based on the amount of cross-rack data migration and migration time).

[0140] Step S205, generating a dynamic resource demand table according to the resource allocation ratio adjustment instructions and combining the preset business feature library;

[0141] The system generates a dynamic resource demand table based on the resource allocation ratio adjustment instructions and the preset business feature library. The business feature library contains feature information of various business types, such as task type, priority, resource consumption mode, etc. The system uses these feature information to guide the dynamic allocation of resources, ensuring that different business types are properly supported in the resource scheduling process.

[0142] For example, compute-intensive tasks may be given more computing power resources, while storage-intensive tasks may be given more storage resources. The generated dynamic resource demand table will provide input data for the next cross-business priority decision.

[0143] Step S206, constructing a cross-business priority weight matrix based on the dynamic resource demand table and the storage migration trigger instructions.

[0144] The priority relationship between business tasks can be determined by constructing a resource competition graph, and then assigning a priority to each task. Specifically, tasks are nodes of the graph, and the resource demand overlap degree between tasks is the weight of the edge. The correlation between tasks can be modeled through a graph neural network (GNN), and the system can extract the correlation features of the tasks and calculate the initial priority score of the tasks based on these features. Then, the system calculates the normalized priority weight matrix based on the resource allocation ratio adjustment instructions and the initial priority score, ensuring that high-priority tasks get more resources.

[0145] It can be understood that through the generation of the priority matrix, the system can ensure reasonable allocation of resources and prioritize important tasks. By modeling the relationships between tasks through graph neural networks, the accuracy and fairness of resource scheduling can be effectively improved, reducing resource competition and task delays.

[0146] In the above embodiments, the dynamic resource scheduling and priority decision method based on reinforcement learning can automatically and intelligently handle resource allocation problems in multi-service systems. By combining real-time resource pool status, time series prediction data, and reinforcement learning algorithms, the system can flexibly adapt to changes in business load and make efficient and accurate resource scheduling decisions.

[0147] Referring to Figure 3 As an embodiment of step S206, after constructing the cross-service priority weight matrix based on the dynamic resource demand table and the storage migration trigger instruction, the system further includes:

[0148] Step S301 inputs the dynamic resource demand table and the cross-service priority weight matrix into the digital twin simulation environment to generate a simulated scheduling result.

[0149] The system will verify the scheduling strategy through the digital twin simulation environment. Digital twin is a virtual model that can reflect the state and behavior of the system in the physical world. In the embodiments of the present application, the digital twin simulation environment can accurately simulate resource allocation, task scheduling, storage management, and other operations in the actual system. By inputting the dynamic resource demand table and the cross-service priority weight matrix, the system can generate a simulated scheduling result.

[0150] It can be understood that the core advantage of the digital twin simulation environment lies in its efficient simulation capability, which can simulate various complex resource scheduling scenarios in a virtual environment to predict various situations that may be encountered in the actual system. Through simulation scheduling, the system can test the effects of different scheduling decisions in actual operations, discover potential problems in a timely manner, and avoid direct operational testing in the actual production environment. The simulation results typically include resource usage, task completion time, system load status, and other key indicators.

[0151] For example, suppose the system tests two different resource allocation strategies in the simulation environment. One is to allocate more resources to high-priority services, and the other is to allocate resources evenly to all services. Through digital twin simulation, the system can observe which strategy can improve resource utilization and task completion speed in actual execution.

[0152] Step S302 triggers the incremental learning module to update the policy network parameters of the reinforcement learning agent according to the error rate of the actual scheduling result and the simulated scheduling result.

[0153] Wherein, the system judges whether the current scheduling strategy is optimal by comparing the error rate between the actual scheduling result and the simulation scheduling result. If the error between the actual scheduling result and the simulation scheduling result is large, it may indicate that the current scheduling strategy needs to be adjusted. At this time, the system triggers the incremental learning module to update the policy network parameters of the reinforcement learning agent.

[0154] Specifically, incremental learning is a process of gradually optimizing a learning model based on existing knowledge without losing previously learned knowledge. Through incremental learning, the reinforcement learning agent can gradually optimize its decision-making strategy according to newly obtained data during actual operation. In this way, the system can continuously optimize itself and improve the accuracy and effectiveness of decision-making.

[0155] Illustratively, if the system tests a certain resource allocation strategy in a simulation environment, the simulation result shows that the high-priority task has been allocated with the expected resources and has been completed smoothly. However, in actual scheduling, the system may cause the high-priority task to be delayed due to some unpredictable factors. At this time, through the comparison of error rates, the system finds the gap between the simulation and actual results, and triggers the incremental learning module to adjust the policy network parameters, so that the system can better adapt to changes in actual operation.

[0156] In the above embodiment, the dynamic resource demand table and the cross-service priority weight matrix are input into the digital twin simulation environment for simulation, and the incremental learning module is triggered for policy optimization according to the error rate between the actual scheduling result and the simulation scheduling result, further strengthening the intelligent scheduling capability of the system in the multi-service environment. The use of digital twin simulation enables the system to test and verify various scheduling strategies in a virtual environment, thereby avoiding unforeseen errors in the production environment. Through the incremental learning module, the system can continuously optimize the decision-making strategy according to new data during actual operation, ensuring that resource scheduling can always meet the efficiency and flexibility requirements in various complex service scenarios. The overall technical solution can significantly improve resource utilization, reduce task delays, improve system stability and overall performance, and adapt to the changing multi-service environment.

[0157] As an embodiment of step S105, initializing preset rules in the scheduling strategy rule library based on the service type includes: when the service type is real-time task, configuring pre-emptive scheduling rules and storage bandwidth reservation threshold; or, when the service type is offline task, configuring elastic resource pool allocation rules and cold data storage migration strategy.

[0158] In the embodiments of the present application, for real-time tasks, due to the high requirement for response time, preemptive scheduling of resources may be involved. For example, if the system is processing multiple tasks, the calculation priority of the real-time task is higher than that of other tasks, and the system will preempt and preferentially allocate computing resources. At the same time, the real-time task has strict requirements on bandwidth, so when the scheduling strategy is initialized, a bandwidth reservation threshold is configured for the real-time task to ensure that the real-time task will not be affected by insufficient bandwidth during execution.

[0159] Unlike real-time tasks, offline tasks have lower real-time requirements, so elastic resource pool allocation rules can be used to more effectively utilize idle resources. Offline tasks can be processed during off-peak hours, and data after task completion does not need to be responded immediately. The cold data storage migration strategy is to ensure that the data of offline tasks is reasonably stored and migrated in the system, reduce the occupation of online storage, and improve the resource utilization of the system.

[0160] In the above embodiments, customized scheduling rules are formulated for each type of task according to the different requirements of real-time tasks and offline tasks. Real-time tasks use preemptive scheduling rules and bandwidth reservation strategies to ensure their priority execution under high load; offline tasks optimize resource utilization through elastic resource pools and cold data migration strategies. The scheduling strategy library of the system is not only initialized according to business requirements at the beginning, but also dynamically adjusted according to feedback information in each subsequent scheduling period, ensuring that the strategy always meets the current resource and business requirements.

[0161] Referring to Figure 4 As an embodiment of step S106, based on the preset rules and resource pool topology information, the task allocation scheme is generated by a hybrid scheduling algorithm according to the dynamic resource demand table and the cross-business priority weight matrix.

[0162] Step S401, obtaining a dynamic resource demand table, a cross-business priority weight matrix, resource pool topology information, and preset rules;

[0163] The dynamic resource demand table contains the demand information of tasks for computing, storage, bandwidth, and other resources; the cross-business priority weight matrix reflects the priority of different businesses, which can help the system make reasonable resource allocation decisions among multiple tasks; the resource pool topology information describes the connection relationship between computing and storage resource nodes, which can help optimize the data transmission path when allocating tasks; the preset rules include preemption strategy, storage affinity rules, and SLA constraints, etc. These rules specify the priority and other constraints when scheduling tasks.

[0164] Step S402, performing time window segmentation and normalization processing on the dynamic resource demand table to generate a standardized resource demand time slice;

[0165] In this way, the resource demand of the task is divided into multiple time windows, usually fixed-length slices, by time slicing the resource demand table. The demand values in each time window are normalized to ensure that data in different time periods can be uniformly compared and processed.

[0166] For example, assume that a task requires a large amount of bandwidth in some time periods and less in other time periods. Through time slicing, the resource demand of the task in different time periods can be clearly defined and normalized, such as normalizing the bandwidth demand value to the interval [0, 1], to facilitate subsequent calculation and comparison.

[0167] In step S403, a weighted directed graph is constructed based on the resource pool topology information, and a topology graph structure file is generated.

[0168] In this way, a weighted directed graph is constructed based on the topology information of the resource pool. The nodes of the graph represent different resource nodes, and the weights of the edges represent factors such as communication delay, bandwidth remaining rate, and storage migration time between nodes. This graph structure file will be used for subsequent task scheduling decisions.

[0169] For example, assume that there are computing nodes A, B, and storage node C in the system. The communication delay between nodes A and B is low, and the bandwidth is large, so the edge weight between them is small. The migration delay between nodes A and C is high, and the bandwidth is limited, so the edge weight is large. Through graph construction, the resource transmission cost between nodes can be reflected.

[0170] It can be understood that the weighted directed graph provides a network structure perspective for subsequent scheduling decisions, allowing the task scheduling to consider the communication cost and migration cost between resource nodes, thereby achieving more precise resource allocation.

[0171] In step S404, the standardized resource demand time slices are classified and weighted according to pre-set rules.

[0172] In this way, the standardized resource demand time slices are classified and weighted according to pre-set rules. Each rule (such as preemption strategy, storage affinity, SLA constraint) will assign different weights according to different task demands and business priorities, thereby affecting the priority and resource allocation of the task.

[0173] For example, for real-time tasks, the preemption strategy may be assigned a high weight, while the storage affinity rule may have little effect on certain tasks. For offline tasks, the weight of the storage affinity rule may be high in order to optimize the data migration path.

[0174] Step S405, in combination with the cross-service priority weight matrix and the topology graph structure file, the matching degree score of the task and the node is calculated to generate a candidate node score table and a rule conflict marking list;

[0175] Wherein, by combining the cross-service priority weight matrix and the topology graph structure file, the matching degree score of each task and resource node is calculated. The score reflects the suitability of the task executing on a certain node, based on factors such as the priority of the task, resource requirements, available resources of the node, and communication cost. At the same time, rule conflict situations are also marked, for example, when a task requires preemption of resources, but the node already has a high-priority task, a rule conflict may occur.

[0176] Illustratively, if a high-priority real-time task needs to be executed on node A, but node A already has a low-priority task running, the calculated matching degree score may be low. At the same time, the system will mark the rule conflict to remind the subsequent scheduling to adjust.

[0177] Step S406, based on the candidate node score table and the rule conflict marking list, a final task allocation scheme is generated by heuristic algorithm.

[0178] Wherein, based on the candidate node score table, a preliminary task allocation scheme is generated by heuristic algorithm. Heuristic algorithm usually selects the best task allocation scheme according to factors such as task priority, node adaptation degree, and resource utilization. Specifically, the heuristic algorithm can be an improved genetic algorithm, and the chromosome code contains task ID, target node ID and resource quota.

[0179] Illustratively, assuming that the system has multiple nodes A, B and C, and multiple tasks T1, T2 and T3, the heuristic algorithm will prioritize task T1 to node A based on the matching degree score of each task and node, while tasks T2 and T3 may be allocated to nodes B and C.

[0180] In the above embodiments, the system can efficiently allocate tasks and resources in the case of dynamic resource demand and service priority changes, optimize system performance, and ultimately achieve accurate task scheduling and resource utilization, ensuring efficient coordination and dynamic adaptation of multi-dimensional resources.

[0181] Reference Figure 5 As an embodiment of step S406, the step of generating a final task allocation scheme by heuristic algorithm based on the candidate node score table and the rule conflict marking list includes:

[0182] Step S501, obtaining the candidate node score table, the rule conflict marking list and the resource pool topology information;

[0183] The candidate node score table contains the matching scores between each task and resource node, which helps to determine whether a task is suitable for execution on a specific node. The rule conflict marker list lists all possible node combinations that may have rule conflicts, for example, the nodes that may have conflicts between preemption policies and load balancing rules. The resource pool topology information is used to describe the layout of each node in the resource pool, including the communication delay, bandwidth, and other information between nodes, which is used for subsequent calculation and decision-making.

[0184] For example, assume that there is a task T1, and the candidate resource nodes include nodes A, B, and C. The candidate node score table has matching scores of 0.9, 0.7, and 0.8 for T1 to A, B, and C, respectively. In the rule conflict marker list, the combination of T1 and node A has no conflict, while T1 and B have a conflict (because node B has been occupied by other high-priority tasks), and the resource pool topology information provides the communication delay and bandwidth between nodes.

[0185] Step S502: According to the candidate node score table, a white list of allocable nodes is screened for each task, excluding illegal node combinations in the rule conflict marker list;

[0186] The system screens a white list of allocable nodes for each task based on the candidate node score table, excluding node combinations that have rule conflicts. The rule conflict marker list indicates which node combinations are not available, usually because of task conflicts, resource constraints, or priority problems.

[0187] For example, for task T1, assuming that its candidate nodes are nodes A, B, and C, if the combination of T1 and node B has a conflict, then B will be excluded from the white list of allocable nodes. Finally, the allocable nodes for T1 are A and C.

[0188] Step S503: An initial population set is constructed based on the white list of allocable nodes; each population individual is represented by a chromosome code indicating a task allocation scheme;

[0189] Specifically, the system constructs an initial population set, and each population individual represents a task allocation scheme. The chromosome code of each population individual contains information such as task ID, target node ID, and resource quota, indicating the mapping of tasks to nodes and resource allocation.

[0190] For example, assume that the system has tasks T1 and T2, and nodes A, B, and C. In the initial population, there may be a population individual indicating that task T1 is allocated to node A and task T2 is allocated to node B, and another population individual indicating that task T1 is allocated to node C and task T2 is allocated to node A, etc.

[0191] Step S504, based on the candidate node score table and the communication delay parameter and node load data in the resource pool topology information, a multi-objective fitness function is constructed;

[0192] The fitness function is a key to measure the quality of the task allocation scheme. In this step, the system constructs a multi-objective fitness function, which combines candidate node scores, communication delay, node load, and other factors to evaluate the effect of each task allocation scheme.

[0193] Specifically, the multiple objectives can include: the matching degree of tasks and nodes (calculated by the candidate node score table), communication cost (data transmission delay between computing resource nodes), and load balancing (whether the load is uniform based on the resource occupation of each node).

[0194] Step S505, dynamically adjust the weight coefficient of each optimization objective in the multi-objective fitness function according to the real-time resource pool state data;

[0195] According to the state of the real-time resource pool (such as node load, bandwidth, communication delay, etc.), the weight of each objective in the fitness function is dynamically adjusted. For example, when the system load is high, load balancing may get a higher weight, and when the network delay is low, the weight of communication cost may be low. For example, if the load of node A is very high, the system may increase the weight of load balancing to encourage more tasks to be allocated to other nodes with lower load.

[0196] Step S506, performing a tournament selection operation on the initial population set to generate a parent population;

[0197] The tournament selection operation is a common genetic algorithm selection method, the system will randomly select several individuals in the initial population for competition, and select individuals with higher fitness as the parent population to ensure that excellent individuals can be passed to the next generation in the genetic process. For example, assuming that there are multiple task allocation schemes in the initial population, the system will randomly select several schemes, calculate their fitness, and then select the individual with the highest fitness as the parent.

[0198] Step S507, based on the communication path table in the resource pool topology information, performing a topology-aware crossover operation on the parent chromosomes in the parent population to generate a child population;

[0199] The topology-aware crossover operation is a crossover operation on the parent chromosomes in the genetic algorithm to generate new child individuals. The crossover operation is based on the communication path in the resource pool topology information to ensure that the task allocation scheme after crossover is more optimized on the network, reducing communication delay and resource conflicts.

[0200] For example, if the parent chromosomes represent task T1 assigned to node A and task T2 assigned to node B, the crossover can generate a scheme where task T1 is assigned to node B and task T2 is assigned to node A, which is optimized based on communication delay.

[0201] At step S508, the mutation rate is dynamically adjusted according to the population diversity evaluation result, and the offspring chromosomes in the offspring population are subjected to probability mutation based on the candidate node score table.

[0202] The mutation operation is an important operation in the genetic algorithm, which is used to maintain the diversity of the population. According to the evaluation of the diversity of the population, the system dynamically adjusts the mutation rate to ensure that new genes are introduced to a certain extent, thereby avoiding the algorithm falling into a local optimal solution. The mutation operation will mutate the offspring individuals based on the candidate node score table. For example, if task T1 is currently assigned to node A, the mutation operation can assign task T1 to node B with higher matching degree.

[0203] At step S509, the chromosomes with fitness scores higher than the preset threshold are subjected to local search optimization.

[0204] Specifically, when the fitness scores of some offspring individuals are high, the system will optimize these individuals through local search, which further improves the fitness by refining the task allocation scheme. For example, for chromosomes with high fitness, local search can further reduce communication delay and optimize load balancing by adjusting the allocation nodes of some tasks.

[0205] At step S510, the individuals with the highest fitness are selected from the parent population and the offspring population to form an elite population.

[0206] The elite strategy retains the individuals with the highest fitness and retains them in the elite population. In this way, the best task allocation scheme will not be lost due to genetic operations, which helps to speed up the convergence of the algorithm.

[0207] At step S511, when the convergence condition is met, the chromosome with the highest fitness in the elite population is decoded as the final task allocation scheme.

[0208] When the genetic algorithm converges (i.e., the fitness of the elite population in consecutive N iterations is improved by less than a threshold, or the maximum number of iterations is reached), the individual with the highest fitness in the elite population is decoded as the final task allocation scheme. After multiple generations of optimization, the individual with the highest fitness in the elite population represents the optimal task allocation scheme.

[0209] In the above embodiments, the combination of heuristic algorithms and genetic algorithms achieves global optimization of the task allocation scheme, improves the efficiency and quality of task allocation, and ultimately generates an optimal task allocation scheme, ensuring efficient use of system resources, reducing communication costs, and enabling load balancing optimization.

[0210] With reference to Figure 6 As an embodiment of step S110, based on the latest state snapshot of the resource pool and the scheduling failure case library, the weight parameters of the time series prediction model and the scheduling strategy rule library are updated to obtain an updated strategy version, which includes the following steps:

[0211] Step S601: According to the latest state snapshot of the resource pool, the node load rate, storage distribution state and task execution progress are obtained.

[0212] For example, assume there are three nodes (A, B, C), with node A load rate of 85%, node B of 60%, and node C of 92%. The storage resource distribution state shows that the storage resource usage rate of node A is 50%, node B is 70%, and node C is almost full. The task execution progress table shows that task T1 is 40% on node A, and task T2 is 60% on node B.

[0213] Step S602: Dynamic normalization processing is performed on the node load rate to generate a standardized load rate matrix.

[0214] Specifically, dynamic normalization processing of node load rate ensures the comparability of different node loads. By normalization method (such as using sliding window method or other standardization techniques), the load rate of each node is converted into a standardized value, so that the load between different nodes can be compared uniformly. The standardized load rate matrix provides a unified reference for subsequent scheduling optimization. For example, assume that the load rates of nodes A, B and C are 85%, 60% and 92% respectively. Through normalization processing, the system may convert the load rates to 0.85, 0.60 and 0.92, which can be directly used to compare the load of different nodes.

[0215] Step S603: Topological modeling is performed on the storage distribution state to generate a storage resource distribution heat map.

[0216] The distribution of storage resources has an important influence on task scheduling and data migration. Through topological modeling, the system can visually represent the usage of storage resources in the resource pool and generate a storage resource distribution heat map. The heat map represents the intensity of storage resource usage through color depth, facilitating quick identification of storage resource bottlenecks.

[0217] Exemplarily, in the storage resource distribution heat map, the storage usage rate of node A is 50%, which is displayed as green; the usage rate of node B is 70%, which is displayed as yellow; and node C is almost full (90%), which is displayed as red. Through the heat map, the storage resource distribution can be directly observed, which facilitates subsequent optimization decisions.

[0218] In step S604, the time series alignment of the task execution progress is performed to generate a task progress time series table.

[0219] The execution progress of the task is dynamically changed, and the task progress time series table can uniformly represent the execution progress of each task at different time points by performing time series alignment on the progress data of multiple tasks. The system can evaluate the status of different tasks on the same time axis and perform cross-task resource scheduling.

[0220] Exemplarily, the execution progress of task T1 at time points t1, t2 and t3 is 30%, 50% and 80% respectively, and the progress of task T2 at time points t1, t2 and t3 is 20%, 40% and 60% respectively. Through time series alignment, the generated task progress time series table will contain the execution progress data of these tasks at each time point.

[0221] In step S605, the time series prediction model is incrementally trained based on the standardized load rate matrix and the task progress time series table, and the model weight parameters are updated.

[0222] The system incrementally trains the existing time series prediction model based on the standardized load rate matrix and the task progress time series table, and updates the weight parameters of the model. Incremental training is a method of continuously optimizing the existing model based on new data, which can more accurately predict the resource demand and system load of future tasks.

[0223] Exemplarily, assuming that at time point t1, the load of node A is 0.85 and the progress of task T1 is 30%. The model will update the weight parameters through incremental training to more accurately predict the execution progress of the task and the load of the node at the next time point.

[0224] In step S606, the storage constraint weight in the loss function of the time series prediction model is dynamically adjusted according to the storage resource distribution heat map.

[0225] Specifically, the state of the storage resource directly affects the storage demand and data migration path of the task. Therefore, during the training process of the time series prediction model, the storage constraint weight in the model loss function is dynamically adjusted based on the storage resource distribution heat map. For example, when the usage rate of a certain storage node is too high, the model will increase the corresponding storage constraint weight to reduce the load of the node.

[0226] Step S607, in combination with the task progress time sequence table and the scheduling failure case library, detect conflict rules in the scheduling strategy rule library that are inconsistent with the task execution progress;

[0227] Among them, through the task progress time sequence table and the scheduling failure case library, the system can detect conflict rules in the existing scheduling strategy rule library that are inconsistent with the task execution progress. These conflict rules may be caused by the fact that some tasks are not executed as expected or that resource allocation is unreasonable.

[0228] Step S608, condition threshold re-calibration and execution priority dynamic sorting are performed on the conflict rules to generate an optimized scheduling strategy rule library;

[0229] Specifically, after detecting scheduling rule conflicts, the system recalibrates the condition threshold of the relevant rules and dynamically adjusts the priority of the rules. This adjustment process ensures that the scheduling rules remain optimized for execution in a constantly changing system state.

[0230] For example, suppose a rule states that task T1 must be executed on a node with a load of less than 80%, but the actual situation is that node A has a load exceeding 80%. The system will adjust the threshold of this rule or change the priority of this rule so that task T1 can be executed on time.

[0231] Step S609, based on the storage resource distribution heat map and the task execution progress, the cold and hot data migration path optimization of the storage demand hierarchical strategy is performed to obtain an updated storage hierarchical configuration file;

[0232] Among them, according to the storage resource distribution heat map and the task execution progress, the system optimizes the hierarchical strategy of storage demand, and reasonably adjusts the migration path of cold and hot data. For example, the system can migrate hot data related to high progress tasks to storage nodes with lower load.

[0233] Step S610, encapsulate the updated time sequence prediction model, the optimized scheduling strategy rule library and the storage hierarchical configuration file into an updated strategy version.

[0234] Among them, the updated strategy version will be synchronized to all scheduling nodes of the system to ensure that the system can uniformly execute the updated strategy.

[0235] In the above embodiments, the continuous optimization of the scheduling strategy is ensured, and the load, storage and task scheduling strategy of the system are adjusted in real time through a data-driven manner, thereby improving the resource utilization of the system, reducing the task execution delay, and ensuring the stable operation of the system.

[0236] The embodiments of the present application also disclose an intelligent computing power and storage scheduling system for a multi-service system.

[0237] The application discloses an intelligent computing power and storage scheduling system of a multi-service system, and specifically comprises the following steps:

[0238] A data acquisition module is used for acquiring real-time monitoring data of each node in the multi-service system.

[0239] A feature extraction module is used for extracting features of the real-time monitoring data, generating a structured feature vector containing a service type label, a task priority and a data access mode, and obtaining a real-time service feature data set.

[0240] A prediction module is used for predicting, based on the real-time service feature data set and historical load logs, a computing power demand value and a storage demand hierarchical strategy in a future preset time window through a time series prediction model.

[0241] A dynamic correction module is used for dynamically correcting the prediction result according to the computing power demand value and the storage demand hierarchical strategy, combining a reinforcement learning algorithm, generating a dynamic resource demand table and a cross-service priority weight matrix.

[0242] A rule initialization module is used for initializing preset rules in a scheduling strategy rule library based on a service type.

[0243] A task allocation module is used for generating a task allocation scheme through a hybrid scheduling algorithm based on preset rules and resource pool topology information, according to the dynamic resource demand table and the cross-service priority weight matrix.

[0244] A storage data migration module is used for generating storage data migration instructions through a storage scheduling engine based on the storage demand hierarchical strategy.

[0245] An execution module is used for sending the task allocation scheme and the storage data migration instructions to a virtualized resource orchestration module, and performing computing node expansion, task migration and storage data redistribution operations.

[0246] A state data acquisition module is used for acquiring resource pool state data after execution, and generating a latest resource pool state snapshot.

[0247] A strategy version updating module is used for updating weight parameters of the time series prediction model and a scheduling strategy rule library based on the latest resource pool state snapshot and a scheduling failure case library, obtaining an updated strategy version and applying the updated strategy version to a next scheduling period.

[0248] The intelligent computing power and storage scheduling system of the multi-service system can implement any one of the above scheduling methods, and the specific working processes of the modules in the scheduling system can refer to the corresponding processes in the above method embodiments.

[0249] In several embodiments provided in the present application, it should be understood that the provided method and system can be implemented in other manners. For example, the embodiments of the system described above are merely schematic; for example, the division of the modules is merely a logical function division; for example, a plurality of modules or a component thereof can be combined or integrated into another system, or some features can be ignored or not executed. Thus, the actual division of the functions can be different from the division in the above description.

[0250] The embodiment of the present application further discloses a computer device.

[0251] The computer device comprises a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor executes the computer program to implement the intelligent computing power and storage scheduling method of the multi-service system.

[0252] The embodiment of the present application further discloses a computer readable storage medium.

[0253] The computer readable storage medium stores a computer program capable of being loaded and executed by a processor to implement any one of the intelligent computing power and storage scheduling methods of the multi-service system.

[0254] The computer readable storage medium can be any tangible medium containing or storing a program, which can be used by or in combination with an instruction execution system, device or apparatus; the program code contained in the computer readable medium can be transmitted by any appropriate medium, including but not limited to wireless, wire, optical cable, RF, etc., or any appropriate combination of the above.

[0255] It should be noted that in the above embodiments, the description of each embodiment has its own emphasis, and the parts not described in detail in a certain embodiment can be referred to the relevant description of other embodiments.

[0256] The above are all preferred embodiments of the present application, and are not intended to limit the protection scope of the present application; any feature disclosed in the specification (including the abstract and the drawings) can be replaced by other equivalent or similar features unless specifically described; that is, each feature is only an example of a series of equivalent or similar features unless specifically described.

Claims

1. An intelligent computing power and storage scheduling method of a multi-service system, characterized in that, The method comprises: collecting real-time monitoring data of each node in a multi-service system; extracting features from the real-time monitoring data to generate a structured feature vector containing service type labels, task priorities, and data access patterns, thereby obtaining a real-time service feature dataset; based on the real-time service feature dataset and historical load logs, predicting the computing power requirement value and storage requirement hierarchical strategy within a future preset time window through a time series prediction model; based on the computing power requirement value and storage requirement hierarchical strategy, dynamically correcting the prediction results by combining a reinforcement learning algorithm to generate a dynamic resource requirement table and a cross-service priority weight matrix; initializing a preset rule in a scheduling strategy rule library based on the service type; based on the preset rule and resource pool topology information, generating a task allocation scheme through a hybrid scheduling algorithm according to the dynamic resource requirement table and the cross-service priority weight matrix; based on the storage requirement hierarchical strategy, generating storage data migration instructions through a storage scheduling engine; sending the task allocation scheme and storage data migration instructions to a virtualized resource orchestration module to perform computing node expansion, task migration, and storage data redistribution operations; collecting resource pool state data after execution to generate a latest resource pool state snapshot; based on the latest resource pool state snapshot and a scheduling failure case library, updating the weight parameters of the time series prediction model and the scheduling strategy rule library to obtain an updated strategy version and apply it to the next scheduling cycle. 2.The intelligent computing power and storage scheduling method of a multi-service system according to claim 1, wherein, The step of dynamically correcting the prediction results by combining a reinforcement learning algorithm based on the computing power requirement value and storage requirement hierarchical strategy to generate a dynamic resource requirement table and a cross-service priority weight matrix comprises: obtaining real-time resource pool state data and receiving the computing power requirement value and storage requirement hierarchical strategy within a future preset time window output by the time series prediction model; merging the computing power requirement value, storage requirement hierarchical strategy, and real-time resource pool state data into a composite state vector and performing normalization processing to generate a standardized state matrix; defining action space parameters based on the service type and inputting the action space parameters into a pre-configured reinforcement learning agent; wherein the action space parameters include resource allocation ratio adjustment instructions and storage migration trigger instructions; based on the standardized state matrix and historical scheduling records, calculating resource allocation ratio adjustment instructions and storage migration trigger instructions through the reinforcement learning agent; wherein the reinforcement learning agent evaluates action values according to a preset reward function and updates policy network parameters; based on the resource allocation ratio adjustment instructions, generating a dynamic resource requirement table in combination with a preset service feature library; based on the dynamic resource requirement table and storage migration trigger instructions, constructing a cross-service priority weight matrix. 3.The intelligent computing power and storage scheduling method of a multi-service system according to claim 2, wherein, After constructing the cross-service priority weight matrix based on the dynamic resource requirement table and storage migration trigger instructions, the method further comprises: inputting the dynamic resource requirement table and cross-service priority weight matrix into a digital twin simulation environment to generate a simulated scheduling result; based on the error rate between the actual scheduling result and the simulated scheduling result, triggering an incremental learning module to update the policy network parameters of the reinforcement learning agent. 4.The intelligent computing power and storage scheduling method of a multi-service system according to claim 1, wherein, Initializing preset rules in a scheduling policy rule library based on a service type comprises: When the service type is a real-time task, configuring pre-emptive scheduling rules and a storage bandwidth reservation threshold; or, when the service type is an offline task, configuring elastic resource pool allocation rules and a cold data storage migration strategy.

5. The intelligent computing power and storage scheduling method of a multi-service system according to claim 1, characterized in that, Generating a task allocation scheme by a hybrid scheduling algorithm based on the preset rules and resource pool topology information, according to the dynamic resource demand table and the cross-service priority weight matrix, comprises: Obtaining a dynamic resource demand table, a cross-service priority weight matrix, resource pool topology information and preset rules; Performing time window segmentation and normalization processing on the dynamic resource demand table to generate a standardized resource demand time slice; Constructing a weighted directed graph based on resource pool topology information to generate a topology graph structure file; Classifying and weighting the standardized resource demand time slice according to the preset rules; Combining the cross-service priority weight matrix and the topology graph structure file, calculating the matching degree score of tasks and nodes to generate a candidate node score table and a rule conflict marker list; Generating a final task allocation scheme by a heuristic algorithm based on the candidate node score table and the rule conflict marker list.

6. The intelligent computing power and storage scheduling method of a multi-service system according to claim 5, characterized in that, Generating a final task allocation scheme by a heuristic algorithm based on the candidate node score table and the rule conflict marker list, comprises: Obtaining a candidate node score table, a rule conflict marker list and resource pool topology information; According to the candidate node score table, filtering the assignable node white list for each task to exclude illegal node combinations in the rule conflict marker list; Constructing an initial population set based on the assignable node white list; wherein each population individual adopts chromosome coding to represent a task allocation scheme; Based on the candidate node score table and the communication delay parameters and node load data in the resource pool topology information, constructing a multi-objective fitness function; According to real-time resource pool state data, dynamically adjusting the weight coefficients of each optimization target in the multi-objective fitness function; Performing a tournament selection operation on the initial population set to generate a parent population; Based on the communication path table in the resource pool topology information, performing a topology-aware crossover operation on the parent chromosomes in the parent population to generate a child population; According to the population diversity evaluation result, dynamically adjusting the mutation rate and performing a probability mutation on the child chromosomes in the child population based on the candidate node score table; Performing a local search optimization on chromosomes with an adaptive score higher than a preset threshold; Selecting individuals with the optimal adaptability from the parent population and the child population to form an elite population; When the convergence condition is met, decoding the chromosome with the highest adaptability in the elite population as the final task allocation scheme.

7. The intelligent computing and storage scheduling method of a multi-service system according to any one of claims 1 to 6, characterized in that, Updating the weight parameters of the time series prediction model and the scheduling policy rule library based on the latest state snapshot of the resource pool and the scheduling failure case library to obtain an updated strategy version, comprises: According to the latest state snapshot of the resource pool, obtaining node load rate, storage distribution state and task execution progress; Performing dynamic normalization processing on the node load rate to generate a standardized load rate matrix; topology modeling is performed on the storage distribution state to generate a storage resource distribution heat map; time sequence alignment is performed on the task execution progress to generate a task progress time sequence table; incremental training is performed on the time sequence prediction model based on the standardized load rate matrix and the task progress time sequence table to update model weight parameters; storage constraint weights in a loss function of the time sequence prediction model are dynamically adjusted according to the storage resource distribution heat map; conflict rules in the scheduling strategy rule library that are contradictory to the task execution progress are detected in combination with the task progress time sequence table and a scheduling failure case library; the conflict rules are subjected to conditional threshold re-calibration and dynamic sorting of execution priorities to generate an optimized scheduling strategy rule library; based on the storage resource distribution heat map and the task execution progress, cold and hot data migration path optimization is performed on the storage demand hierarchical strategy to obtain an updated storage hierarchical configuration file; the updated time sequence prediction model, the optimized scheduling strategy rule library and the storage hierarchical configuration file are packaged as an updated strategy version.

8. An intelligent computing power and storage scheduling system of a multi-service system, characterized in that, The system comprises: a data collection module configured to collect real-time monitoring data of each node in a plurality of service systems; a feature extraction module configured to perform feature extraction on the real-time monitoring data to generate a structured feature vector comprising a service type label, a task priority and a data access mode, thereby obtaining a real-time service feature data set; a prediction module configured to predict, based on the real-time service feature data set and historical load logs, an algorithm demand value and a storage demand hierarchical strategy in a future preset time window through a time sequence prediction model; a dynamic correction module configured to dynamically correct the prediction results based on the algorithm demand value and the storage demand hierarchical strategy in combination with a reinforcement learning algorithm, thereby generating a dynamic resource demand table and a cross-service priority weight matrix; a rule initialization module configured to initialize preset rules in a scheduling strategy rule library based on service types; a task allocation module configured to generate a task allocation scheme through a hybrid scheduling algorithm based on the preset rules and resource pool topology information, the dynamic resource demand table and the cross-service priority weight matrix; a storage data migration module configured to generate storage data migration instructions through a storage scheduling engine based on the storage demand hierarchical strategy; an execution module configured to send the task allocation scheme and the storage data migration instructions to a virtualized resource orchestration module to perform computing node expansion, task migration and storage data redistribution operations; a state data collection module configured to collect resource pool state data after execution to generate a latest resource pool state snapshot; the latest resource pool state snapshot comprises node load rate, storage distribution state and task execution progress; a strategy version updating module configured to update weight parameters of the time sequence prediction model and the scheduling strategy rule library based on the latest resource pool state snapshot and a scheduling failure case library, thereby obtaining an updated strategy version and applying the updated strategy version to a next scheduling period.

9. A computer device, characterized by: A computer program product comprises a memory, a processor and a computer program stored on the memory and executable on the processor, wherein the processor implements the method of any one of claims 1 to 7 when executing the program. The system comprises: a data collection module configured to collect real-time monitoring data of each node in a plurality of service systems; a feature extraction module configured to perform feature extraction on the real-time monitoring data to generate a structured feature vector comprising a service type label, a task priority and a data access mode, thereby obtaining a real-time service feature data set; a prediction module configured to predict, based on the real-time service feature data set and historical load logs, an algorithm demand value and a storage demand hierarchical strategy in a future preset time window through a time sequence prediction model; a dynamic correction module configured to dynamically correct the prediction results based on the algorithm demand value and the storage demand hierarchical strategy in combination with a reinforcement learning algorithm, thereby generating a dynamic resource demand table and a cross-service priority weight matrix; a rule initialization module configured to initialize preset rules in a scheduling strategy rule library based on service types; a task allocation module configured to generate a task allocation scheme through a hybrid scheduling algorithm based on the preset rules and resource pool topology information, the dynamic resource demand table and the cross-service priority weight matrix; a storage data migration module configured to generate storage data migration instructions through a storage scheduling engine based on the storage demand hierarchical strategy; an execution module configured to send the task allocation scheme and the storage data migration instructions to a virtualized resource orchestration module to perform computing node expansion, task migration and storage data redistribution operations; a state data collection module configured to collect resource pool state data after execution to generate a latest resource pool state snapshot; the latest resource pool state snapshot comprises node load rate, storage distribution state and task execution progress; a strategy version updating module configured to update weight parameters of the time sequence prediction model and the scheduling strategy rule library based on the latest resource pool state snapshot and a scheduling failure case library, thereby obtaining an updated strategy version and applying the updated strategy version to a next scheduling period. A computer program product comprises a memory, a processor and a computer program stored on the memory and executable on the processor, wherein the processor implements the method of any one of claims 1 to 7 when executing the program.

10. A computer-readable storage medium, characterized in that: A computer program is stored which can be loaded by a processor and execute the method according to any one of claims 1 to 7.

Images