A cloud computing resource cooperative scheduling management method and platform

Abstract

The application discloses a cloud computing resource cooperative scheduling management method and platform, and relates to the technical field of cloud computing and network communication management. The method collects network communication data between each computing node of a cloud platform, generates an inter-node communication matrix based on the network communication data, and the network communication data comprises bandwidth information and time delay information between node pairs; based on the communication matrix, the bandwidth information and the time delay information are normalized, and the network affinity value between the node pairs is calculated through a weighted fusion method, the network affinity value being used to represent the network communication quality between the node pairs; the computing load index of each computing node and the network affinity value are input into a pre-constructed multi-constraint optimization model to generate a task scheduling scheme; the task scheduling scheme is executed, and the network performance change is continuously monitored after the task is deployed, and rescheduling is triggered when the network performance deviation exceeds a preset threshold. The application realizes dynamic feedback between the network state and the scheduling behavior.

Description

Technical Field

[0001] This application relates to the field of cloud computing and network communication management technology, specifically to a cloud computing resource collaborative scheduling management method and platform. Background Technology

[0002] As cloud computing platforms continue to expand, resource scheduling has become a crucial element in ensuring system performance and resource utilization. Existing cloud resource scheduling methods primarily rely on static indicators such as computing load, storage occupancy, or energy consumption levels, achieving dynamic resource balancing through virtual machine migration and task allocation. However, these scheduling strategies generally overlook the impact of inter-node network communication characteristics on task execution. In distributed application scenarios, task instances often involve frequent data interactions and remote calls. When scheduling strategies make decisions solely based on computing resources or CPU load, high-frequency communication tasks may be assigned to nodes with high network latency, leading to increased data transmission latency, uneven bandwidth utilization, and frequent link congestion, ultimately reducing overall system throughput and user response speed.

[0003] While some existing technologies attempt to optimize scheduling by incorporating network traffic information, most remain at the level of link monitoring or bandwidth allocation, lacking the ability to quantify network communication characteristics and directly participate in scheduling decisions. Furthermore, existing solutions are mostly unidirectional control, triggering task migration from the resource scheduling end, failing to establish dynamic feedback between network status and scheduling behavior, resulting in insufficient coordination between resource scheduling and network communication. Summary of the Invention

[0004] The purpose of this application is to provide a cloud computing resource collaborative scheduling and management method and platform to solve the problems mentioned in the background art.

[0005] In a first aspect, one embodiment of this application provides a cloud computing resource collaborative scheduling and management method. The method includes: collecting network communication data between computing nodes of a cloud platform and generating a communication matrix between nodes based on the network communication data, wherein the network communication data includes bandwidth information and latency information between node pairs; based on the communication matrix, normalizing the bandwidth information and latency information, and calculating the network affinity value between node pairs through a weighted fusion method, wherein the network affinity value is used to characterize the network communication quality between node pairs; inputting the computing load index and network affinity value of each computing node into a pre-constructed multi-constraint optimization model to generate a task scheduling scheme; executing the task scheduling scheme, and continuously monitoring network performance changes after task deployment, triggering rescheduling when the network performance deviation exceeds a preset threshold.

[0006] In conjunction with the first aspect, in some implementations of the first aspect, network communication data also includes packet loss rate information between node pairs. Based on the communication matrix, bandwidth information and latency information are normalized, and then the network affinity value between node pairs is calculated through a weighted fusion method. This includes: extracting the extreme values ​​of various network performance parameters in the communication matrix as the normalization benchmark; using the normalization benchmark, normalizing the bandwidth information and latency information respectively; assigning corresponding weight coefficients to the normalized bandwidth information, latency information, and packet loss rate information respectively, and calculating the network affinity value through a weighted fusion method.

[0007] In conjunction with the first aspect, in some implementations of the first aspect, corresponding weight coefficients are assigned to the normalized bandwidth information, latency information, and packet loss rate information, including: determining the task type of the task to be scheduled, including data-intensive, latency-sensitive, and reliability-sensitive tasks; assigning different weight coefficients to the normalized bandwidth information, latency information, and packet loss rate information according to the task type of the task to be scheduled; wherein, the highest bandwidth weight coefficient is assigned to data-intensive tasks, the highest latency weight coefficient is assigned to latency-sensitive tasks, and the highest packet loss rate weight coefficient is assigned to reliability-sensitive tasks.

[0008] In conjunction with the first aspect, in some implementations of the first aspect, generating a communication matrix between nodes based on network communication data includes: applying a sliding time window to smooth the collected network communication data, and using a weighted moving average method to calculate the smoothed network performance parameter values; organizing the smoothed network performance parameter values ​​into a communication matrix, wherein the communication matrix is ​​stored in a sparse matrix format, recording only the node pairs with communication relationships and their corresponding network performance parameter values.

[0009] In conjunction with the first aspect, in some implementations of the first aspect, the sliding time window smoothing process uses a weighted method to assign different weights to data collected at different time points. The closer the data point is to the current time, the higher the corresponding weight coefficient is assigned.

[0010] In conjunction with the first aspect, in some implementations of the first aspect, the computational load index and network affinity value of each computing node are input into a pre-constructed multi-constraint optimization model to generate a task scheduling scheme, including: constructing an objective function based on network communication cost and computational load cost, wherein the network communication cost is calculated based on the network affinity value between the task to be scheduled and the associated nodes, and the computational load cost is calculated based on the computational load index of the candidate nodes; taking at least one of node compatibility constraints, network traffic capacity constraints, quality of service assurance constraints, and load balancing soft constraints as constraints of the multi-constraint optimization model; and solving the objective function to obtain a task scheduling scheme that allocates tasks to computing nodes.

[0011] In conjunction with the first aspect, in some implementations of the first aspect, network performance changes are continuously monitored after task deployment, and rescheduling is triggered when the network performance deviation exceeds a preset threshold, including: if the deviation between the actual network performance parameter value and the expected network performance parameter value of any key node pair exceeds a first preset threshold; and / or, if the proportion of the number of congested links to the total number of monitored links exceeds a second preset threshold, then rescheduling is triggered.

[0012] Secondly, one embodiment of this application provides a cloud computing resource collaborative scheduling and management platform, comprising: a network awareness module, used to collect network communication data between computing nodes of the cloud platform, the network communication data including bandwidth information and latency information between node pairs, and generating a communication matrix between nodes based on the network communication data; an affinity calculation module, used to normalize the bandwidth information and latency information based on the communication matrix, and calculate the network affinity value between node pairs through a weighted fusion method, the network affinity value representing the network communication quality between node pairs; a scheduling engine, used to input the computing load index and network affinity value of each computing node into a pre-built multi-constraint optimization model to generate a task scheduling scheme; and an execution module, used to execute the task scheduling scheme and continuously monitor network performance changes after task deployment, triggering rescheduling when the network performance deviation exceeds a preset threshold.

[0013] In conjunction with the second aspect, in some implementations of the second aspect, the execution module is also used to: after triggering rescheduling, feed back the updated network performance data to the network awareness module to update the communication matrix, and trigger the scheduling engine to regenerate the task scheduling scheme based on the updated communication matrix.

[0014] In conjunction with the second aspect, in some implementations of the second aspect, the scheduling management platform further includes: a policy management module, which receives scheduling policy parameters configured by the administrator. The scheduling policy parameters include the weight coefficients of network performance parameters, the rescheduling trigger threshold, and the optimization target preference settings, and provides the corresponding policy parameter configurations to the affinity calculation module and the scheduling engine.

[0015] Compared with the prior art, the beneficial effects of this application are: 1. This application introduces a network traffic affinity model to combine task scheduling decisions with network link status, achieving joint optimization allocation of computing and communication resources. It can dynamically adjust task deployment locations based on inter-node bandwidth, latency, and traffic distribution, reducing cross-node data transmission overhead, minimizing network congestion, and improving system throughput and task execution efficiency, thereby significantly enhancing the service quality of the cloud platform in a multi-tenant environment.

[0016] 2. This application, based on real-time monitoring and feedback, dynamically perceives and adjusts strategies for network and computing status, forming a self-learning mechanism for scheduling optimization. It not only enables rapid rescheduling when traffic shifts or node load anomalies occur, ensuring continuous optimal resource utilization, but also possesses modularity and scalability, allowing it to work collaboratively with various cloud orchestration systems and software-defined network controllers, reducing operational complexity and improving the system's intelligence and long-term scalability. Attached Figure Description

[0017] Figure 1 A flowchart illustrating a cloud computing resource collaborative scheduling and management method provided in an embodiment of this application; Figure 2 This is a flowchart illustrating the process of calculating the network affinity value between node pairs based on a communication matrix after normalizing bandwidth and latency information, using a weighted fusion method, according to an embodiment of this application. Figure 3 This is a flowchart illustrating the process of assigning corresponding weight coefficients to normalized bandwidth information, latency information, and packet loss rate information according to an embodiment of this application. Figure 4 A schematic diagram illustrating the process of generating an inter-node communication matrix based on network communication data, provided in an embodiment of this application; Figure 5 This is a flowchart illustrating how a task scheduling scheme is generated by inputting the computational load index and network affinity value of each computing node into a pre-built multi-constraint optimization model, as provided in one embodiment of this application. Figure 6 This is a schematic diagram of a process provided by an embodiment of the present application for continuously monitoring network performance changes after task deployment and triggering rescheduling when the network performance deviation exceeds a preset threshold; Figure 7 This is a schematic diagram of the structure of a cloud computing resource collaborative scheduling and management platform provided in an embodiment of this application; Figure 8 This is a schematic diagram of the structure of a cloud computing resource collaborative scheduling and management platform provided in another embodiment of this application. Detailed Implementation

[0018] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0019] The following is combined with Figures 1 to 6 This application provides a detailed description of the cloud computing resource collaborative scheduling and management method.

[0020] Figure 1 This is a flowchart illustrating a cloud computing resource collaborative scheduling and management method provided in an embodiment of this application. Figure 1 As shown in the embodiments of this application, the cloud computing resource collaborative scheduling and management method includes the following steps: Step 100: Collect network communication data between computing nodes on the cloud platform, and generate a communication matrix between nodes based on the network communication data. The network communication data includes bandwidth information and latency information between node pairs.

[0021] Inter-node communication matrix: refers to a data structure generated based on network communication data, used to store performance parameters such as bandwidth and latency between computing node pairs in a cloud platform, with storage optimized in sparse matrix format.

[0022] Bandwidth information: refers to the bandwidth value of the network link between node pairs, representing the data transmission capacity, and is used to calculate network affinity after normalization processing.

[0023] Latency information: refers to the latency value of the network link between node pairs, representing the data transmission delay, and is used in network affinity calculation after inverse normalization.

[0024] Step 101: Based on the communication matrix, after normalizing the bandwidth and latency information, the network affinity value between node pairs is calculated by weighted fusion. The network affinity value is used to characterize the network communication quality between node pairs.

[0025] Step 102: Input the computing load index and network affinity value of each computing node into the pre-built multi-constraint optimization model to generate a task scheduling scheme.

[0026] For example, computational load metrics include CPU utilization, memory utilization, and storage I / O utilization.

[0027] It should be understood that the multi-constraint optimization model refers to the mathematical model used to generate task scheduling schemes in this cloud computing resource collaborative scheduling and management method. This model constructs a function with the objective of a weighted sum of network communication costs and computing load costs, and imposes various conditions, including node compatibility constraints, network traffic capacity constraints, quality of service assurance constraints, and load balancing soft constraints. Ultimately, it solves for a task allocation scheme with the lowest total cost while satisfying all hard constraints.

[0028] Step 103: Execute the task scheduling scheme and continuously monitor network performance changes after task deployment. When the network performance deviation exceeds the preset threshold, trigger rescheduling.

[0029] The beneficial effects of this embodiment are as follows: By combining network communication characteristics with computing load indicators for resource scheduling, joint optimization of computing and network resources is achieved, which effectively reduces cross-node data transmission overhead, reduces network congestion, and significantly improves system throughput and task execution efficiency. At the same time, based on real-time monitoring and feedback, it can dynamically perceive and quickly respond to changes in network status, thereby improving the overall service quality and intelligent operation and maintenance level of the cloud platform.

[0030] Figure 2 This is a schematic diagram illustrating a process for calculating the network affinity value between node pairs based on a communication matrix, after normalizing bandwidth and latency information. Figure 2 As shown in the embodiments of this application, the cloud computing resource collaborative scheduling and management method, based on a communication matrix, normalizes bandwidth and latency information, and then calculates the network affinity value between node pairs through a weighted fusion method, including the following steps: Step 200: Extract the extreme values ​​of all network performance parameters in the communication matrix as the normalization benchmark.

[0031] Specifically, iterate through all node pairs with communication relationships in the communication matrix and extract the maximum bandwidth across the entire network at the current moment. and maximum delay Since bandwidth is measured in Mbps and latency in milliseconds, they belong to different units and have significantly different numerical ranges, making direct comparison and fusion calculation impossible. By extracting the extreme values ​​of the entire network as the normalization denominator, the original values ​​can be mapped to a dimensionless range of 0 to 1, eliminating the influence of the unit difference.

[0032] After each data collection cycle, the system automatically scans the updated communication matrix and recalculates the maximum bandwidth and maximum latency of the entire network. When the cloud platform expands by adding high-performance nodes, Accordingly, when network congestion causes an increase in overall latency, The corresponding increase is necessary. In practical implementation, to prevent extreme outliers from affecting the normalization benchmark, a quantile method is used, such as using the 95th percentile value to replace the absolute maximum value, thereby improving the stability and representativeness of the normalization results.

[0033] Step 201: Normalize the bandwidth information and latency information using a normalization benchmark.

[0034] Specifically, for node pairs Bandwidth information Apply the normalization formula: ; Obtain the normalized bandwidth value This value represents the node pair. The percentage of bandwidth that accounts for the maximum bandwidth of the entire network, ranging from 0 to 1. The closer the value is to 1, the higher the data transmission capability of the node. The closer a value is to 0, the more limited the bandwidth resources of that node pair are.

[0035] For node pairs Delay information Since latency is a negative metric—meaning a smaller value indicates better performance—inverse normalization is required. First, the normalization formula is applied: ; Obtain the percentage of latency relative to the maximum latency across the entire network, and then use the formula: ; Perform a reverse transformation to convert the negative indicator into a positive one. The converted latency contribution... The numerical range is also between 0 and 1, with larger values ​​indicating lower latency and faster communication response. For example, if the latency of a certain node pair is 10 milliseconds, and the maximum latency of the entire network is 100 milliseconds, then... = 0.1, = 0.9, this high contribution value reflects the low latency advantage of this node pair.

[0036] After normalization, both bandwidth and latency information are converted into dimensionless positive indices within the range of 0 to 1, eliminating the dimensional and directional differences of the original data. The normalization results are stored and updated in real time. When changes in the network environment cause the normalization benchmark to be updated, the system automatically triggers batch recalculation to ensure that the normalized values ​​of all node pairs remain consistent based on a unified benchmark.

[0037] Step 202: Assign corresponding weight coefficients to the normalized bandwidth information, latency information, and packet loss rate information, and calculate the network affinity value through weighted fusion.

[0038] Specifically, network communication data also includes packet loss rate information between node pairs. Since its value is already within the range of 0 to 1, and typically remains at an extremely low level below 0.01% in a normally operating data center network, it does not require normalization and can directly participate in the weighted fusion calculation. The network affinity calculation formula can then be applied to the node pairs. Perform the calculation: ; Among them, the weighting coefficient Reflects the importance of bandwidth in network communication quality evaluation; weighting coefficient Reflects the importance of latency; weighting coefficient Reflecting the importance of reliability, ensuring the final network affinity value. It also falls within the range of 0 to 1. It should be understood that the packet loss rate information between point pairs is the proportion of data packets lost on the network link between two specific computing nodes in the cloud platform. This parameter is used to quantify the reliability of the communication link between nodes and is one of the key performance indicators for calculating the network affinity value.

[0039] The specific values ​​of the weight coefficients are dynamically configured based on the task type of the task to be scheduled. The system maintains a mapping table between task types and weight coefficients, which defines recommended weight configuration schemes for different task types. For data-intensive tasks, such as big data processing, video encoding / transcoding, and database backup and synchronization, the performance bottleneck mainly lies in the large-capacity data transmission between nodes; therefore, [the following is omitted as the text is incomplete and likely refers to a different topic]. Set to a high weight value of 0.7 to 0.9. Set to a low weight value of 0.1 to 0.2. A secondary weight value set to 0.05 to 0.15 allows the scheduling engine to prioritize high-bandwidth nodes for deploying relevant tasks. For latency-sensitive tasks, such as online stock trading, real-time video conferencing, online gaming, and VoIP calls, even millisecond-level latency differences can impact user experience or business outcomes; therefore, [the following is missing from the original text]. Set to a high weight value of 0.7 to 0.9. Set it to 0.1 to 0.2. Setting it to 0.05 to 0.15 prompts the scheduling engine to select low-latency node pairs. For reliability-sensitive tasks, such as financial settlements, medical data transmission, and blockchain consensus scenarios where data loss is intolerable, this will... Set to a high weight value of 0.6 to 0.8, while also taking into account and The configuration. For balanced applications, the default configuration is used. = 0.5, = 0.4, = 0.1, achieving a balance between bandwidth and latency.

[0040] When a new task is submitted to the scheduling system, the system first extracts the type identifier field from the task's metadata. The task type can be determined in several ways: explicit labeling (users specify the type using tags or configuration files); automatic identification (the system analyzes historical task characteristics, monitors network traffic and computing resource usage, input / output patterns, etc., and uses classification algorithms to automatically categorize the task); and application mapping (inference based on a predefined mapping table of the task's application category). After determining the task type, the system queries a weight mapping table to obtain the corresponding... , , The numerical value is used if the task type is not defined in the mapping table, and the default load balancer configuration is used.

[0041] The obtained weight coefficients are substituted into the affinity calculation formula to calculate the network affinity value for each node pair. The calculation process is performed in batches using matrix operations to improve computational efficiency. The calculated network affinity values ​​are then presented. The closer the value is to 1, the stronger the node pair. The better the network communication quality between nodes, the more suitable it is for deploying related tasks that require frequent communication; the closer the value is to 0, the worse the network communication quality, and communication-intensive tasks should be avoided when assigning them to that node pair. The network affinity values ​​of all node pairs are organized into an N×N dimensional affinity matrix A_matrix. This matrix is ​​symmetric, and only the upper or lower triangular portion is stored during actual storage to save approximately 50% of storage space. The affinity matrix is ​​updated once per acquisition cycle. The update process uses incremental calculation optimization, recalculating only the node pairs that have changed in the communication matrix; unchanged node pairs directly reuse the calculation results from the previous cycle. This optimization can reduce redundant calculations by approximately 60% to 80%. After the update, the affinity matrix is ​​made available to the scheduling decision engine for reading and use via shared memory or a remote procedure call interface.

[0042] The beneficial effects of this embodiment are as follows: The normalization and weighted fusion method of network performance parameters realizes the accurate quantification of communication quality between node pairs, and creates a unified network affinity index. This index can dynamically adjust the weight allocation according to different task types (data-intensive, latency-sensitive, or reliability-sensitive), effectively eliminating the difference in the dimensions of different parameters, and converting factors such as bandwidth and latency into comparable positive indicators in the range of 0 to 1. This provides a decision basis for the scheduling system, ensuring that tasks are assigned to the nodes with the best network communication conditions, thereby optimizing the overall system performance.

[0043] Figure 3 This is a flowchart illustrating the process of assigning corresponding weight coefficients to normalized bandwidth information, latency information, and packet loss rate information according to an embodiment of this application. Figure 3 As shown in the embodiments of this application, the cloud computing resource collaborative scheduling and management method assigns corresponding weight coefficients to the normalized bandwidth information, latency information, and packet loss rate information, including the following steps: Step 300: Determine the task type of the task to be scheduled. The task types include data-intensive, latency-sensitive, and reliability-sensitive tasks.

[0044] Specifically, task type refers to the classification and labeling of tasks based on their computational and communication characteristics, used to guide the scheduling system in selecting appropriate optimization strategies. In a cloud computing environment, different applications have significantly different network resource requirements. Data-intensive tasks mainly include big data processing, video encoding / transcoding, and database backup / synchronization. Their performance bottleneck lies primarily in the large-capacity data transmission between nodes, requiring high bandwidth but having relatively high tolerance for latency. Latency-sensitive tasks mainly include online stock trading, real-time video conferencing, and online games, requiring millisecond-level response times and being extremely sensitive to latency but with relatively lower absolute bandwidth requirements. Reliability-sensitive tasks mainly include financial settlement, medical data transmission, and blockchain consensus applications, which cannot tolerate data loss or transmission errors and have strict requirements for network reliability. In addition, there are computationally intensive tasks such as scientific computing and deep learning training, which primarily consume computational resources with relatively less network communication and can be categorized as balanced tasks.

[0045] There are three ways to determine task types. The first is explicit labeling, where users explicitly specify the task type through metadata when submitting a task. Specifically, users can add identifiers to the tag field of the task description file or declare fields in a JSON-formatted configuration file. When parsing the task submission request, the system prioritizes reading these explicitly declared type identifiers. The second is automatic identification, where the system automatically categorizes tasks by analyzing their historical execution characteristics using a classifier. Specifically, this involves monitoring the ratio of network traffic to computing resource usage during historical task execution, calculating a network communication density index, defined as the total network traffic generated by the task divided by the total CPU time of the task. If this ratio is greater than a threshold T1 (e.g., 100 MB / CPU second), it is classified as data-intensive; if the average response latency requirement is less than a threshold T2 (e.g., 50 milliseconds), it is classified as latency-sensitive; if the task configuration sets strict data consistency requirements or enables multi-replica synchronization, it is classified as reliability-sensitive. The classifier can be implemented using decision trees or rule engines, determining the final category through a combination of multiple feature dimensions. The third is application mapping, which infers the task type based on a predefined mapping table of the application category to which the task belongs. The system maintains an application type mapping table, which records the correspondence between common application frameworks and task types. For example, Hadoop MapReduce jobs are mapped to data-intensive tasks by default, Spark Streaming jobs are mapped to latency-sensitive tasks by default, and Cassandra data replication tasks are mapped to reliability-sensitive tasks by default. When a task is submitted, it carries an application identifier such as app_name='hadoop', and the system can infer its task type by querying the mapping table.

[0046] In actual execution, the system applies the three determination methods mentioned above in order of priority. First, it checks whether there is any explicit user labeling; if so, it uses that method directly. If there is no explicit labeling, it attempts to automatically identify the task based on historical data. If the task is running for the first time and there is no historical data, it uses the application mapping method. If no corresponding record is found in the application mapping table, it uses the system's default balanced configuration. After the task type is determined, the system records the type identifier in the task metadata and triggers the weight coefficient query process.

[0047] Step 301: Based on the task type of the task to be scheduled, assign different weight coefficients to the normalized bandwidth information, latency information, and packet loss rate information.

[0048] Specifically, a mapping table is maintained between task types and weight coefficients. This mapping table stores the weight configuration schemes corresponding to different task types in key-value pairs. The data structure of the mapping table is designed in a multi-level nested format. The first level key is a task type identifier string such as "data_intensive", "latency_sensitive", "reliability_sensitive", "balanced", etc. The second level contains weight coefficient triples. Bandwidth weight, Delay weight, Packet loss rate weighting. The mapping table is stored in the form of a configuration file, which allows administrators to dynamically modify and manage versions through the policy management module, and the changes take effect without restarting the system.

[0049] Once the type of the task to be scheduled is determined, the scheduling system extracts the task type identifier field from the task metadata and uses this identifier as the query key to access the weight mapping table. The query process first performs an exact match, checking if a record in the mapping table exists that is exactly the same as the task type identifier. If the exact match is successful, the corresponding weight coefficient triplet is directly read. If the exact match fails, a fuzzy match is attempted; for example, if the task type identifier is "data_intensive_batch", the system extracts the prefix "data_intensive" for a second query. If the fuzzy match still fails, the default balanced weight configuration is returned. =0.5、 =0.4、 =0.1.

[0050] After obtaining the weight coefficients, the system performs parameter verification to ensure that the weight coefficients meet the constraints. First, the normalization constraints are verified. If the conditions are not met, normalize proportionally and calculate the sum. Then reassign the value. , , Secondly, the non-negativity constraint is verified to ensure that all weight coefficients are non-negative. If a negative value is found, it is set to 0 and normalized again. Finally, the validity constraint is verified to ensure that at least one weight coefficient is greater than a certain minimum threshold, such as 0.05, to avoid the calculation results losing their discriminative power because all weights are close to 0.

[0051] After verification, the weight coefficients are passed to the affinity calculation module. The transmission method uses parameter injection; for each task to be scheduled, its unique weight coefficients are temporarily bound to the task's scheduling context. These weight coefficients are used when calculating the network affinity values ​​of the node pairs related to that task, without affecting the calculation of other tasks. To support finer-grained weights, the mapping table also supports extended field configuration. In addition to the basic triplet weights, administrators can configure weight adjustment factors for specific task types. These factors fine-tune the base weights based on task priority, Service Level Agreement (SLA) requirements, or the current system load. For example, for latency-sensitive tasks marked as high priority, latency weights can be adjusted. The base value of 0.8 is increased by an additional 0.1 to 0.9 to further strengthen the selection preference for low-latency nodes. The adjustment factor is calculated using either linear superposition or nonlinear function mapping, and the specific strategy can be defined by the administrator through a strategy template.

[0052] Step 302: Assign the highest bandwidth weight coefficient to data-intensive tasks, the highest latency weight coefficient to latency-sensitive tasks, and the highest packet loss rate weight coefficient to reliability-sensitive tasks.

[0053] Specifically, for data-intensive tasks, typical application scenarios include large-scale data analysis, batch file transfer, video stream distribution, and data warehouse synchronization. These tasks require the transfer of gigabytes or even terabytes of data between nodes during execution. The main component of task execution time is data transfer time rather than computation time, making network bandwidth a performance bottleneck. Therefore, the system configures a bandwidth weighting coefficient for data-intensive tasks. Set the weight to a high weight range of 0.7 to 0.9. The specific value should be further refined based on the ratio of data transfer volume to computational volume. When this ratio is greater than 10, it means the data transfer volume is more than 10 times the computational volume. Take 0.9; when the ratio is between 5 and 10, Take 0.8; when the ratio is between 3 and 5, Set it to 0.7. Correspondingly, the delay weighting coefficient... Set to a lower value of 0.1 to 0.2, because data-intensive tasks are relatively less sensitive to transmission latency and can tolerate transmission delays of tens or even hundreds of milliseconds. As long as the total bandwidth is sufficient, the task can be completed within a reasonable time. Packet loss rate weighting coefficient. A secondary weight value set to 0.05 to 0.15 serves as the baseline requirement for ensuring basic transmission reliability. For example, a typical configuration is... =0.8、 =0.15、 =0.05, this configuration makes the scheduling engine prioritize high-bandwidth node pairs when selecting nodes for data-intensive tasks. Even if the latency of these node pairs is slightly higher than average, they will still be selected as long as the bandwidth advantage is obvious and the link stability meets the basic requirements.

[0054] For latency-sensitive tasks, typical application scenarios include real-time trading systems, online multiplayer games, remote desktops, and real-time audio and video communication. These tasks have strict requirements for response time, typically requiring end-to-end latency to be controlled within 10 to 50 milliseconds. Even a small increase in latency can lead to a significant deterioration in user experience or failure of business logic. For example, in online trading systems, latency exceeding a threshold may cause transaction failures or price slippage; in online games, high latency can cause lag and screen desynchronization. Therefore, the system configures a latency weighting coefficient for latency-sensitive tasks. Set to a high-weight range of 0.7 to 0.9. The specific value is further refined based on the task's latency requirement threshold; when the task requires a latency of less than 10 milliseconds... Take 0.9; when the required latency is between 10 and 30 milliseconds, Take 0.8; when the required latency is between 30 and 50 milliseconds, Set it to 0.7. Correspondingly, the bandwidth weighting coefficient... Set to a lower value between 0.1 and 0.2, because latency-sensitive tasks typically involve small data transfer volumes, with each interaction's packet size often in the KB range. The absolute bandwidth requirement is low; sufficient bandwidth is enough to support basic data transmission. Packet loss rate weighting coefficient. Also set to auxiliary weight values ​​between 0.05 and 0.15. For example, a typical configuration is... =0.1、 =0.85、 =0.05. This configuration enables the scheduling engine to prioritize low-latency node pairs when selecting nodes for latency-sensitive tasks. Even if these node pairs do not have the highest bandwidth, they will be selected as long as the latency is minimal and the basic bandwidth and reliability requirements are met.

[0055] For reliability-sensitive tasks, typical application scenarios include financial transaction settlement, medical image transmission, critical business data backup, and blockchain node synchronization. These tasks have extremely high requirements for data integrity and transmission reliability, and cannot tolerate any data loss, corruption, or out-of-order delivery. Even the loss of a single data packet can lead to the rollback of the entire transaction or data inconsistency, causing serious business consequences. Therefore, the system configures a packet loss rate weighting coefficient for reliability-sensitive tasks. Set to a high-weight range of 0.6 to 0.8. The specific value is further refined based on the task's reliability level requirements. When the task is marked as critical and requires a packet loss rate below 0.001%, Take 0.8; when the task is marked as important and the packet loss rate is required to be less than 0.01%, Take 0.7; when the task is marked as normal level but still has reliability requirements, Set it to 0.6. Correspondingly, the bandwidth weighting coefficient... and delay weighting coefficient Allocation is performed while satisfying normalization constraints, adjusting the ratio between the two based on the specific characteristics of the task. If a reliability-sensitive task involves a large amount of data transmission, then... Set it to a higher value, such as 0.15 to 0.25; if the task also has certain latency requirements, then... Set it to a higher value, such as 0.15 to 0.25; if the task is purely concerned with reliability and is not sensitive to bandwidth and latency, then... and All are set to low values, such as 0.1 each. For example, a typical configuration is... =0.15、 =0.15、 =0.7. This configuration enables the scheduling engine to prioritize stable links with low packet loss rates when selecting nodes for reliability-sensitive tasks, choosing node pairs with the best historical packet loss statistics, while also taking into account certain bandwidth and latency performance.

[0056] The beneficial effects of this embodiment are as follows: By identifying task types and dynamically allocating network parameter weights, the system implements a differentiated resource scheduling strategy, prioritizing high bandwidth for data-intensive tasks, ensuring low latency for latency-sensitive tasks, and selecting stable links for reliability-sensitive tasks. This provides customized quality of service assurance for different business characteristics, significantly improving the overall performance and resource utilization efficiency of the cloud platform.

[0057] Figure 4 This is a schematic diagram illustrating a process for generating an inter-node communication matrix based on network communication data, as provided in an embodiment of this application. Figure 4As shown in the embodiments of this application, the cloud computing resource collaborative scheduling and management method generates an inter-node communication matrix based on network communication data, including the following steps: Step 400: Apply sliding time window smoothing to the collected network communication data and calculate the smoothed network performance parameter values ​​using a weighted moving average method.

[0058] Specifically, set the sliding time window length. The data acquisition period is 5 to 10 cycles. A weighted moving average method is used for smoothing, and the weighting coefficients are allocated using an exponential decay pattern. Specifically, let the current time be t, and the sliding window contain the data points from the most recent W acquisitions, numbered sequentially from the most recent to the furthest as acquisition number 0, 1, 2, up to W-1. For the data point acquired in the kth acquisition, its weighting coefficient is defined as: ; in, As the attenuation factor, This represents the interval between data points and the current time point. Attenuation factor. The value ranges from 0.7 to 0.9, with a typical value of 0.8. This parameter controls the rate of weight decay. To ensure that the sum of all weight coefficients equals 1 and meets the normalization requirement, the weight coefficients need to be normalized. The sum of all weights within the window is calculated, and then each original weight is divided by the sum to obtain the normalized weight. The normalized weight coefficients ensure that the weighted average result will not be subject to systematic bias due to changes in window length.

[0059] For node pairs The raw bandwidth values ​​collected in each instance within the window are denoted as follows: ,in The smoothed bandwidth value is calculated using the weighted moving average formula: ; This formula performs a weighted summation of all historical data collected within the window. The weights are distributed according to an exponential decay law, ensuring that data points closer to the current time contribute more to the final result. The same method is applied to smoothing time delay information, calculating: ; And the smoothing of packet loss rate information, calculation: ; The network performance parameters obtained after smoothing can reflect the steady-state communication characteristics between node pairs, effectively avoiding the interference of transient disturbances on scheduling decisions.

[0060] Step 401: Organize the smoothed network performance parameter values ​​into a communication matrix. The communication matrix is ​​stored in a sparse matrix format, recording only the node pairs with communication relationships and their corresponding network performance parameter values.

[0061] Specifically, the constructed communication matrix M is logically represented as an N×N dimensional square matrix, where N is the total number of computing nodes in the cloud platform, and the row and column indices of the matrix correspond to the source and destination nodes, respectively. Matrix elements Storage node pairs Network performance parameter information, including smoothed bandwidth values. Delay value and packet loss rate The three dimensions are represented using a triplet structure as follows: This triple comprehensively characterizes the network communication characteristics between node pairs, where bandwidth reflects data transmission capacity, latency reflects response speed, and packet loss rate reflects link stability.

[0062] The communication matrix update operation is performed once per acquisition cycle, with the update frequency synchronized with the data acquisition cycle T, typically ranging from 5 to 30 seconds. The update process employs incremental updates; the system compares newly acquired network performance data with existing data in the matrix, updating only node pairs that have changed. For node pairs with newly established communication relationships, a new record is inserted into the sparse matrix; for node pairs where communication relationships disappear, the corresponding record is deleted from the matrix; for node pairs where communication relationships persist but performance parameters change, their triplet values ​​are updated. This incremental update method avoids the high overhead of full matrix reconstruction. In scenarios with relatively stable communication relationships, each update involves only 5% to 15% of the matrix elements, significantly improving update efficiency.

[0063] The beneficial effects of this embodiment are: by combining sliding time window smoothing with sparse matrix storage technology, short-term fluctuations in network performance parameters are effectively filtered, ensuring that scheduling decisions are based on steady-state network characteristics rather than instantaneous outliers. At the same time, storage overhead is greatly reduced, computing efficiency is improved, and frequent task migrations are avoided, thereby significantly improving the overall performance and resource utilization of the cloud platform.

[0064] In another embodiment, the sliding time window smoothing process uses a weighted method to assign different weights to data collected at different time points. The closer the data point is to the current time, the higher the corresponding weight coefficient is assigned.

[0065] Specifically, within the sliding time window, network communication data collected at different time points are assigned differentiated weight coefficients based on the time distance between their collection time and the current time. The weight allocation follows the time decay principle, meaning that the further away a historical data point is from the current time, the smaller its influence on the smoothing result. This weight allocation is based on the time-sensitive characteristics of network conditions; recently collected data can more accurately reflect the true state of the current network environment, while older historical data, although having statistical reference value, gradually loses its timeliness over time.

[0066] By applying normalized weighting coefficients, the network performance parameter values ​​collected within the window are weighted and summed to obtain smoothed parameter values. The same weighted smoothing method is used for bandwidth, latency, and packet loss rate information. This smoothing method effectively filters out instantaneous fluctuations and sudden anomalies in network traffic while maintaining responsiveness to real changes in network conditions. It avoids the lag problem caused by the simple averaging method's equal weighting of all historical data, ensuring that the smoothed network performance parameter values ​​possess both statistical stability and timely representativeness, providing a reliable data foundation for network affinity calculation and scheduling decisions.

[0067] Figure 5 This is a flowchart illustrating how a task scheduling scheme is generated by inputting the computational load index and network affinity value of each computing node into a pre-built multi-constraint optimization model, as provided in one embodiment of this application. Figure 5 As shown in the embodiments of this application, the cloud computing resource collaborative scheduling and management method inputs the computing load index and network affinity value of each computing node into a pre-constructed multi-constraint optimization model to generate a task scheduling scheme, including the following steps: Step 500: Construct an objective function based on the network communication cost item and the computational load cost item. The network communication cost item is calculated based on the network affinity value between the task to be scheduled and the associated nodes, and the computational load cost item is calculated based on the computational load index of the candidate nodes.

[0068] Specifically, when constructing a multi-objective optimization model for task scheduling, the basic variables and parameters of the problem are first defined. Let the set of tasks to be scheduled be... ,in This represents the total number of tasks to be scheduled. Each task... It has specific resource requirements and communication characteristics, including the required number of CPU cores, memory size, storage space, and communication dependencies with other tasks or services. The candidate node set is as follows: ,in This represents the total number of computing nodes available for task deployment in the cloud platform. Define the decision variables. It is a binary variable used to represent the task. Should it be assigned to a node? ,when When equal to 1, it indicates a task. Assigned to a node ,when A value of 0 indicates a task. Not assigned to a node .

[0069] The core of constructing the objective function lies in quantifying the overall cost of the task allocation scheme, which includes both network communication and computational load dimensions. The mathematical expression of the objective function is defined as follows: ; This expression iterates through all possible combinations of task node assignments, summing the cost of each assignment decision. Iterate through all tasks awaiting scheduling. Iterate through all candidate nodes. The first term in the objective function is the network communication cost term, which converts network affinity into a cost metric, based on the affinity value. The value ranges from 0 to 1, with a higher value indicating better network communication quality. The converted value represents the normalized cost of network communication; when affinity is close to 1, the cost is close to 0, and when affinity is close to 0, the cost is close to 1. This item... For the task Frequently communicating related nodes can be identified by analyzing the application topology of the task to determine data source nodes, service dependency nodes, or collaborative computing nodes. For example, for a distributed data processing task, its related node might be the node storing the input data; for a front-end service task in a microservice architecture, its related node might be the node where the back-end API service resides. Weighting coefficients. The relative importance of network communication costs in the total cost is controlled; the value of this coefficient reflects the system's emphasis on network optimization. The second term in the objective function is the load cost term, which directly uses candidate nodes. Comprehensive load index As a cost metric, the comprehensive load index The calculation method is as follows: ; in, For nodes CPU utilization percentage This represents the percentage of memory utilization. Percentage of storage I / O utilization. Weighting factor. , , The weights are set to 0.5, 0.3, and 0.2 respectively. This weight allocation reflects the different impacts of different resource types on task performance. CPU, as the core resource for execution, has the highest weight; insufficient memory, leading to frequent page swapping, has the next highest weight; and disk I / O, with its relatively small impact in compute-intensive scenarios, has the lowest weight. (Comprehensive Load Index) The value ranges from 0 to 100. A higher value indicates a heavier node load and fewer available resources. Assigning tasks to high-load nodes increases the risk of resource contention and performance degradation. Weighting coefficient Control the relative importance of computing load costs in the total cost.

[0070] Weighting coefficient and The specific numerical settings reflect the relative emphasis on network communication optimization and computational load optimization. In a typical configuration, Set to 0.6, The weight is set to 0.4, based on experimental observations of the proportion of network transmission time to total task execution time in distributed application scenarios. In data-intensive distributed processing scenarios, network transmission time typically accounts for 50% to 70% of the total task execution time, while computation time accounts for 30% to 50%. Therefore, network optimization offers greater potential for performance improvement. Weight coefficient and Satisfy normalization constraints + = 1, ensuring that the contributions of the two types of costs are comparable and that the numerical scale of the total cost is stable.

[0071] The objective function aims to minimize the total cost (Cost), which means finding a set of decision variables. The value of is chosen to minimize the weighted sum of network communication costs and computational load costs generated by all task allocations. This objective function achieves joint optimization of computational and network resources, overcoming the local optimum problem caused by optimizing only a single dimension of resources in existing technologies.

[0072] Step 501: Use at least one of the following constraints as the multi-constraint optimization model: node compatibility constraint, network traffic capacity constraint, quality of service guarantee constraint, and load balancing soft constraint.

[0073] Specifically, the multi-constraint optimization model sets various constraints to ensure that the generated task scheduling scheme meets the basic requirements of system operation and business needs while optimizing costs. Constraints are divided into two categories: hard constraints and soft constraints. Hard constraints are rigid restrictions that must be strictly satisfied, and scheduling schemes that violate hard constraints are considered infeasible solutions. Soft constraints are guiding preferences for the optimization objective, influencing scheduling decisions by adding penalty terms to the objective function, but do not exclude schemes that slightly violate soft constraints.

[0074] First, define a basic task assignment uniqueness constraint, which ensures that each task is assigned exactly once. The constraint expression is as follows: For any task The sum of the decision variables of all candidate nodes equals 1, i.e., the task... There is one and only one node among all candidate nodes whose assignment flag is 1. This constraint avoids errors such as duplicate or missed task assignments, ensuring the integrity and uniqueness of the scheduling scheme.

[0075] Set a node resource capacity constraint to ensure that the node's physical resources are not over-allocated. The constraint expression is: ; For any candidate node The total resources required by all tasks assigned to this node cannot exceed the available resource capacity of that node. For the task The resource requirement vector contains three dimensions of resource requirements: CPU core count, memory size in bytes, and storage space in bytes. For nodes The available resource vector also includes three dimensions: the number of available CPU cores, the available memory size, and the available storage space. This constraint is in the form of a vector inequality, meaning that each dimension of the resource demand vector must be less than or equal to the corresponding dimension of the available resource vector. Specifically, the total CPU demand cannot exceed the number of available CPU cores, the total memory demand cannot exceed the available memory size, and the total storage demand cannot exceed the available storage space. Scheduling schemes that violate resource capacity constraints will lead to node resource exhaustion, causing task execution failures or system crashes.

[0076] Set a node compatibility constraint that ensures tasks are only assigned to nodes that meet their runtime environment requirements. The constraint expression is: ; in, For a binary compatibility matrix, when When equal to 1, it indicates a task. Available at node On run, when A value of 0 indicates a task. With nodes Incompatible. This constraint achieves selective filtering through matrix multiplication, when... When it is 0, even Setting it to 1 will also cause the constraint to be unsatisfied, and the optimization solver will automatically exclude such infeasible allocation schemes.

[0077] Setting a network traffic capacity constraint treats network links as independent, constrained resources within the optimization framework, preventing task allocation from overloading network links. The constraint expression is: ; This constraint applies to each physical network link. A separate setting, where the summation symbol iterates through all satisfying conditions. Task node allocation pairs equal to 1 Indicates task Its communication partner nodes The expected communication frequency between them, which can be obtained by analyzing the historical running logs of the task or the application topology declaration, is measured in the number of communications per second. Indicates task The average bandwidth requirement, which is the average amount of data transmitted in a single communication, is measured in megabits per second. Indicates link The physical transmission capacity limit is determined by the network hardware specifications; for example, a gigabit Ethernet link has a capacity of 1000 Mbps, and a 10 gigabit Ethernet link has a capacity of 10000 Mbps. The left side of the constraint calculates all possible links that can be used. The sum of task communication requirements is shown on the right, along with the physical capacity limit of the link, ensuring that the total traffic demand does not exceed the link's carrying capacity. Violating this constraint will lead to continuous link congestion, a sharp increase in packet queuing delay, and severely impact task execution performance.

[0078] Set a Quality of Service (QoS) constraint. This constraint applies to tasks marked as high priority or critical to ensure they receive a high-quality network communication environment. The constraint expression is: ; For tasks marked as high priority If it is assigned to a node And its associated communication node is Then the node pair The network affinity value between them must be greater than or equal to the service level agreement threshold. Threshold The typical value range is 0.7 to 0.9, with the specific value determined according to the service level agreement requirements of the task. This constraint ensures that critical business tasks are deployed to node pairs with the best network communication quality by limiting the range of deployment locations for high-priority tasks, even if this may result in other low-priority tasks being assigned to suboptimal locations, prioritizing the performance of critical tasks.

[0079] A soft constraint for load balancing is set. This soft constraint guides the scheduling result towards a more even load distribution across nodes by adding a load variance penalty term to the objective function. The expression for the penalty term is as follows: ,in This represents the variance calculation function. to This represents the overall load index of each node after task allocation. This is the load balancing penalty coefficient, typically ranging from 0.1 to 0.3. The variance reflects the unevenness of the load distribution; it is smaller when the load on each node is close to the average, and larger when some nodes are overloaded while others are underloaded. By adding this penalty term to the objective function, the optimization solver tends to choose the solution with a more even load distribution when searching for the optimal solution, avoiding resource waste and performance hotspots caused by a few nodes being continuously overloaded while most nodes are idle for extended periods. Unlike hard constraints, soft constraints allow for slight violations when necessary. When all candidate solutions cannot simultaneously satisfy cost minimization and perfect load balancing, the system will weigh the two. The selection and combination of constraints can be flexibly configured according to the needs of the actual application scenario.

[0080] Step 502: Solve the objective function to obtain the task scheduling scheme that allocates tasks to computing nodes.

[0081] Specifically, before allocation begins, the system first prioritizes the set of tasks T to be scheduled, based on the task's network communication density index. Network communication density is defined as the expected network traffic generated by the task divided by the task's computational resource requirements; the formula is: ; in, Indicates task The expected total network traffic, in gigabytes, can be obtained by analyzing historical execution statistics or application claims for the task; CPUi represents the task... The system determines the number of CPU cores required. A higher density value indicates that the performance bottleneck of a task is more likely to be network transmission than computation processing. Such tasks are more sensitive to the network location of deployment nodes. The system sorts tasks in descending order according to network communication density, prioritizing the scheduling of the most sensitive tasks. The technical advantage of prioritizing network-sensitive tasks is that these tasks have the strictest constraints on node location selection. In the early scheduling stages, there are more available nodes, making it easier to find the optimal location that meets the network affinity requirements. On the other hand, network-insensitive tasks have less stringent location requirements, and even if the number of available nodes decreases in later scheduling stages, an acceptable allocation scheme can still be found.

[0082] The algorithm processes tasks one by one according to their sorted priority order. For the currently pending task... The system iterates through all candidate node sets N and calculates the task... Assigned to each candidate node Increment of Cost in the post-objective function The calculation of incremental costs considers two aspects: firstly, the... The increase in network communication cost after changing from 0 to 1, i.e. ,in For the task The associated communication nodes; secondly, the nodes Receive task Increase in the overall load index The calculation method is to calculate the task Resource requirements added to nodes The load index is recalculated based on the current load, and then subtracted from the load index before allocation to calculate the load cost increment. The system selection makes incremental costs... Candidate nodes that reach the minimum value As a task The target deployment node will Set to 1 and update the node. The available resource status and load index are considered. The greedy strategy ensures that each decision moves in the direction of minimizing the objective function. Although it cannot guarantee a global optimum, it can quickly converge to a high-quality approximate optimum.

[0083] In the task Select candidate nodes When assigning a node, the system first checks whether the assignment satisfies all hard constraints. This check includes verifying node compatibility and querying the compatibility matrix. Is it equal to 1? Verify resource capacity and compute nodes. Receive task Does the subsequent resource usage exceed the available resource vector? Any dimension; verify network traffic capacity, computational tasks. After the communication requirements are added, does the total traffic of the relevant links exceed the link capacity limit? Verify the quality of service guarantee; if the task... For high-priority nodes, the network affinity of the node pairs is checked to see if it meets the threshold requirement. Only candidate nodes that pass all hard constraint checks are included in the incremental cost calculation and selection. If all candidate nodes fail to meet the hard constraints, it indicates that the current resource state cannot accommodate the task. At this point, the resource reclamation process is triggered. The resource reclamation process assesses the priority of deployed tasks, identifies low-priority tasks with high resource consumption, marks these tasks as pending migration or suspension, releases their occupied resources, and then re-executes the scheduling decision for the current task. Resource reclamation follows the priority protection principle, ensuring that high-priority tasks are not affected, achieving dynamic resource reallocation and differentiated protection of task priorities.

[0084] The time complexity of the heuristic algorithm is ,in The number of tasks to be scheduled. To calculate the number of nodes, the derivation of this complexity is based on the algorithm's two-loop structure, with the outer loop iterating through... Each task is processed by an inner loop. The algorithm calculates the incremental cost for each candidate node and selects the optimal node. After execution, a task scheduling scheme is generated, output in the form of a standardized scheduling decision table. The scheduling decision table, named `Schedule_Table`, is organized in a table structure, with each row recording the scheduling decision information for a task, including the following fields: **Task Identifier (`task_id`):** Stores the unique identifier of the task, used to locate and operate the task instance in the cloud platform management system. **Target Node Identifier (`target_node`):** Stores the identifier of the computing node assigned to the task, indicating the deployment destination. **Expected Network Affinity (`expected_affinity`):** Records the network affinity value between the task and its associated nodes. This value serves as a benchmark for subsequent performance monitoring, used to evaluate whether the actual running effect meets scheduling expectations. **Expected Load (`expected_load`):** Records the expected comprehensive load index of the target node after task deployment, used to monitor node load change trends. **Execution Priority (`priority`):** Defines the execution order and priority of tasks in resource contention. High-priority tasks are prioritized when resources are insufficient, while low-priority tasks may be paused or migrated. The timestamp field records the time when the scheduling decision was generated, which is used to trace the history of decisions and analyze scheduling delays.

[0085] The beneficial effects of this embodiment are as follows: By constructing a multi-constraint optimization model that considers network affinity and computing load, and using an efficient heuristic algorithm to solve it, the joint optimization scheduling of computing resources and network resources is realized, ensuring that multiple constraints such as node compatibility, network capacity, and service quality are met. At the same time, the solution process that originally required several hours is shortened to a second-level response time, which significantly improves the decision quality and efficiency of the cloud platform in large-scale task scheduling.

[0086] Figure 6 This is a schematic diagram illustrating a process provided by an embodiment of this application for continuously monitoring network performance changes after task deployment and triggering rescheduling when the network performance deviation exceeds a preset threshold. Figure 6 As shown in the embodiments of this application, the cloud computing resource collaborative scheduling and management method continuously monitors network performance changes after task deployment, and triggers rescheduling when the network performance deviation exceeds a preset threshold, including the following steps: Step 600: If the deviation between the actual network performance parameter value and the expected network performance parameter value of any key node pair exceeds the first preset threshold.

[0087] Specifically, after the task scheduling plan is executed and the task deployment is completed, the system immediately initiates a performance monitoring process to continuously track and evaluate the network performance during task execution. The purpose of performance monitoring is to verify whether the actual effect of the scheduling decision meets expectations, and to promptly identify the adverse effects of changes in the network environment on task execution, providing data for dynamic adjustments.

[0088] First, the scope of critical node pairs is determined. Critical node pairs refer to those that carry important task communication relationships; the network performance between these pairs directly affects task execution efficiency and user experience. Critical node pairs are selected based on task priority, communication density, and service level agreement (SLA) standards. A performance monitoring collection period is set to T' seconds, with a typical range of 10 to 60 seconds. Actual network performance parameter values ​​between critical node pairs are collected. Expected network performance parameter values ​​are obtained as a comparison benchmark; these values ​​originate from performance prediction data recorded during the scheduling decision-making phase. When generating the scheduling decision table in step 502, the system has already recorded the expected performance indicators, such as network affinity, bandwidth information, and latency information, on which the scheduling decision is based, in the corresponding fields of the decision table. For node pairs... Expected bandwidth value The bandwidth component of the node pair is extracted from the communication matrix M at the time of the scheduling decision. This value has been smoothed using a sliding window and represents the bandwidth capability that the system believes the node pair should possess at the time of the scheduling decision. The expected latency value Dij_expected is also extracted from the latency component of the communication matrix M. Expected affinity value... Recorded in the `expected_affinity` field of the scheduling decision table, although affinity is a comprehensive indicator of bandwidth and latency rather than an independent parameter, it still serves as a basis for scheduling optimization and needs to be verified during the monitoring phase. The role of the expected value is to provide a performance baseline; the degree of deviation between the actual monitored value and the expected value reflects the accuracy of the scheduling decision and the stability of the network environment.

[0089] Calculate the performance deviation between the actual and expected values. For bandwidth parameters, the deviation is calculated using the relative error form, as shown in the formula: ; The numerator represents the absolute difference between the actual bandwidth and the expected bandwidth, the denominator represents the expected bandwidth, and the result is a dimensionless relative deviation ratio. The advantage of using relative error is that it eliminates the influence of absolute numerical values, making node pairs with different bandwidth levels comparable. For example, for a low-speed link with an expected bandwidth of 100Mbps, a bandwidth deviation of 20% is achieved when the actual measured bandwidth is 80Mbps; for a high-speed link with an expected bandwidth of 1000Mbps, a bandwidth deviation of 20% is also achieved when the actual measured bandwidth is 800Mbps. Although the absolute differences are different, the relative deviations are the same, indicating that the network performance degradation is consistent. The same relative error formula is used for calculating the deviation of the latency parameter: ; Latency deviation is also expressed as a percentage, which facilitates the setting of a unified threshold standard. It should be noted that when the actual value is lower than the expected value, such as when the actual bandwidth is higher than the expected bandwidth or the actual latency is lower than the expected latency, the deviation value is still a positive number. However, this positive deviation is usually regarded as a benign deviation, indicating that the actual performance is better than expected, and will not trigger an alarm. The system only pays attention to and handles negative deviations, that is, when the actual performance is worse than expected.

[0090] A first preset threshold is set as the tolerable upper limit of the deviation. This first preset threshold actually comprises two independent threshold parameters, one for bandwidth deviation and the other for latency deviation. The bandwidth deviation threshold is denoted as... A typical value is 30%. This threshold means the system allows for a fluctuation of up to 30% between actual and expected bandwidth. Only when the deviation exceeds 30% is the scheduling scheme considered significantly flawed and requires replanning. The technical consideration for setting the threshold at 30% is based on the inherent fluctuation characteristics of network traffic. In cloud computing environments, influenced by factors such as multi-tenant sharing, traffic bursts, and routing changes, network bandwidth measurements typically exhibit a natural fluctuation range of 10% to 30%. This fluctuation is normal and should not trigger frequent task migrations. The latency deviation threshold is denoted as... A typical value is 50%. The reason the latency threshold is set more leniently than the bandwidth threshold is that latency measurement is more affected by various random factors such as network jitter, queuing delays, and measurement errors. Its fluctuation range is usually higher than that of bandwidth fluctuations, and many applications have a relatively high tolerance for increased latency. As long as the latency remains within an acceptable range, fluctuations within 50% will not have a significant impact on user experience. Administrators can adjust these two threshold parameters through the policy management module, setting stricter or more lenient standards according to the performance sensitivity and business requirements of different applications.

[0091] The system performs deviation exceeding limits judgment and alarm triggering. At the end of each monitoring cycle, it iterates through the performance deviation indicators of all key node pairs and performs threshold comparison judgment. The judgment logic is as follows: if the bandwidth deviation of any key node pair exceeds the limit... Greater than the threshold or time delay deviation Greater than the threshold If either bandwidth deviation or latency deviation exceeds the limit, the node is marked as having performance issues, triggering a deviation exceedance event. The judgment uses a logical OR relationship; either bandwidth deviation exceeding the limit or latency deviation exceeding the limit triggers the event, as both reduced bandwidth and increased latency can become performance bottlenecks for task execution and require equal attention. Deviation exceedance events are recorded in the monitoring log, containing detailed information such as the abnormal node pair identifier, deviation type, deviation value, expected value, actual value, and monitoring time, for subsequent problem diagnosis and root cause analysis. Simultaneously, the system generates an alarm notification, notifying administrators and operations personnel through alarm channels such as management console push notifications, email notifications, and instant messaging messages, enabling them to promptly understand the system anomaly and take manual intervention measures.

[0092] Step 601: If the proportion of the number of congested links to the total number of monitored links exceeds the second preset threshold, then rescheduling is triggered.

[0093] Specifically, in addition to monitoring performance deviations for individual key node pairs, the system also assesses overall congestion from a global network perspective. It uses the congested link ratio metric to determine if network capacity is nearing saturation, providing supplementary information for triggering rescheduling. This global congestion assessment can uncover systemic network pressure issues that might have been missed in step 600 based on key node pair monitoring, achieving complementarity between local and global monitoring.

[0094] The system defines the criteria for determining link congestion. Link congestion refers to a phenomenon where the traffic load on a network link exceeds its reasonable carrying capacity, leading to a significant increase in packet queuing delay, a higher packet loss rate, and a decrease in transmission throughput. The system uses link utilization as a quantitative indicator for congestion determination. When the utilization rate exceeds the second preset threshold of 80%, the link is marked as congested. The monitoring system maintains a global link status table to count and distribute congested links. This table records the real-time status information of all monitored links in the cloud platform. The scope of monitored links includes all physical and virtual links carrying inter-task communication traffic, especially backbone links connecting compute nodes, compute nodes and storage nodes, and links across racks or data centers. In each monitoring cycle, the system traverses the link status table, reads the current utilization rate of each link, applies the 80% congestion threshold for each link, and marks links with utilization rates exceeding the threshold as congested links and accumulates the count. The congested link count is denoted as Ncongested, and the total number of monitored links, Ntotal, is also recorded, which is the total number of entries in the link status table. The congested link percentage is calculated as the number of congested links divided by the total number of monitored links, using the following formula: The result is a percentage value between 0 and 1, usually expressed as a percentage. This percentage reflects the extent of network congestion at a macro level. A low percentage value indicates that congestion is a localized phenomenon, possibly caused by a sudden surge in traffic from individual tasks or a link failure. A high percentage value indicates that the overall network capacity is close to saturation, representing a systemic problem rather than an isolated event. The congestion percentage is calculated once per monitoring period, and its value is updated in real time with dynamic changes in network load, providing the latest global network health assessment for scheduling decisions.

[0095] A second preset threshold is set as the tolerable upper limit for the congestion ratio, and the second preset threshold is denoted as... A typical value is 15%. This threshold means that the system allows no more than 15% of the monitored links to be in a congested state. Only when the proportion of congested links exceeds 15% is the overall network pressure deemed too high, requiring rescheduling and traffic redistribution. The threshold can also be set to other values ​​depending on the scenario.

[0096] The system executes congestion ratio exceedance judgment and rescheduling trigger, and the monitoring system calculates the actual congestion ratio. With the second preset threshold If a comparison is made, Greater than If the global network congestion exceeds the tolerable range, a rescheduling event is triggered. The rescheduling trigger condition uses the logical OR relationship between steps 600 and 601, meaning that meeting either condition is sufficient to trigger the event. This could be triggered by a critical node exceeding performance limits, a global congestion percentage exceeding limits, or both conditions being met simultaneously. This design ensures that the scheduling system can comprehensively perceive network anomalies from both local and global perspectives, without overlooking any issues that might affect system performance.

[0097] The beneficial effects of this embodiment are as follows: By monitoring network performance in real time and setting up dual triggers, it is possible to identify performance deviations of key node pairs and global network congestion in a timely manner, and proactively trigger resource rescheduling before the network environment deteriorates and seriously affects the business, thereby ensuring service quality stability, improving the system's adaptability to network changes, and realizing dynamic optimization and self-repair of cloud platform resource allocation.

[0098] Figure 7 This is a schematic diagram of the structure of a cloud computing resource collaborative scheduling and management platform provided in one embodiment of this application. Figure 7As shown in the embodiment of this application, a cloud computing resource collaborative scheduling and management platform is provided. The platform includes: a network perception module 700, used to collect network communication data between computing nodes of the cloud platform, the network communication data including bandwidth information and latency information between node pairs, and generating a communication matrix between nodes based on the network communication data; an affinity calculation module 701, used to calculate the network affinity value between node pairs by normalizing the bandwidth information and latency information based on the communication matrix and then using a weighted fusion method, the network affinity value representing the network communication quality between node pairs; a scheduling engine 702, used to input the computing load index and network affinity value of each computing node into a pre-built multi-constraint optimization model to generate a task scheduling scheme; and an execution module 703, used to execute the task scheduling scheme and continuously monitor network performance changes after task deployment, triggering rescheduling when the network performance deviation exceeds a preset threshold.

[0099] Furthermore, the execution module 703 is also used to: after triggering rescheduling, feed back the updated network performance data to the network awareness module to update the communication matrix, and trigger the scheduling engine to regenerate the task scheduling scheme based on the updated communication matrix.

[0100] It should be understood that the specific operational process after triggering rescheduling includes the following steps: The first step is to feed back the latest monitored network performance data to the network awareness module 700. This feedback data includes the actual bandwidth, actual latency, and link utilization values ​​for all node pairs, as well as a detailed list of congested links. Upon receiving the feedback data, the network awareness module 700 immediately updates its maintained communication matrix, replacing the network performance parameters in the matrix elements with the latest measured values ​​to ensure that the communication matrix reflects the current real network environment rather than outdated historical data. The second step is to notify the scheduling decision engine 702 to recalculate the scheduling scheme. The scheduling engine 702 reads the updated communication matrix from the network awareness module 700 and re-executes the optimization process from steps 500 to 502 based on the new network affinity values ​​to generate a new task scheduling scheme. The new scheme automatically considers the current network congestion situation, transferring tasks from node pairs with degraded performance to node pairs with better communication quality, or adjusting task distribution to balance network traffic and avoid overloading specific links. The third step is to execute a task migration operation to implement the new scheduling scheme. The task migration employs a seamless failover strategy of starting first and then stopping. This involves first starting a new instance of the task on the target node and warming it up to a ready state, then switching traffic to the new instance, and finally stopping the old instance on the source node and releasing its resources. This strategy ensures service continuity during the task migration process, avoiding business disruptions caused by task interruptions. During the migration, the system continuously monitors the migration progress and the health status of the new instance. If migration fails or the new instance becomes abnormal, the system immediately rolls back to the original deployment to ensure system stability.

[0101] Complete information on rescheduled events, their triggering reasons, execution processes, and effect evaluations is recorded in the scheduling history database. The recorded content includes the trigger time, the trigger condition type (whether it's a critical node pair deviation exceeding limits or a global congestion exceeding limits), the specific deviation value or congestion percentage, a list of involved abnormal node pairs or congested links, the content of the generated new scheduling scheme, and a task migration list including the migrated task identifier and node positions before and after migration, migration execution time, and performance improvement effect after migration.

[0102] The beneficial effects of this embodiment are as follows: The cloud computing resource collaborative scheduling and management platform achieves comprehensive optimization through an integrated architecture of network awareness, affinity computing, multi-constraint scheduling, and execution modules. It continuously monitors network performance and automatically triggers rescheduling when the deviation exceeds the threshold, achieving seamless task migration with zero service interruption and ensuring optimal resource utilization, network traffic balance, and service quality.

[0103] Figure 8 This is a schematic diagram of the structure of a cloud computing resource collaborative scheduling and management platform provided in another embodiment of this application. Figure 8 As shown in another embodiment of this application, a cloud computing resource collaborative scheduling management platform is provided. The platform further includes a policy management module 800, which is used to receive scheduling policy parameters configured by the administrator. The scheduling policy parameters include the weight coefficient of network performance parameters, the rescheduling trigger threshold, and the optimization target preference settings, and provide the corresponding policy parameter configuration to the affinity calculation module 701 and the scheduling engine 702.

[0104] The beneficial effects of this embodiment are: the policy management module enhances the platform's adaptability and governance capabilities, enabling administrators to customize scheduling behavior through configurable parameters (including network performance weight coefficients, rescheduling thresholds, and optimization preferences), thereby enabling fine-grained control over resource allocation priorities without system rebuilding, and ensuring that the platform can dynamically optimize for different operational needs.

[0105] It will be apparent to those skilled in the art that this application is not limited to the details of the exemplary embodiments described above, and that this application can be implemented in other specific forms without departing from the spirit or essential characteristics of this application. Therefore, the embodiments should be considered illustrative and non-limiting in all respects, and the scope of this application is defined by the appended claims rather than the foregoing description. Thus, all variations falling within the meaning and scope of equivalents of the claims are intended to be included within this application. No reference numerals in the claims should be construed as limiting the scope of the claims.

Claims

1. A cloud computing resource collaborative scheduling and management method, characterized in that, include: Collect network communication data between computing nodes on the cloud platform, and generate an inter-node communication matrix based on the network communication data. The network communication data includes bandwidth information and latency information between node pairs. Based on the communication matrix, the bandwidth information and the delay information are normalized, and the network affinity value between node pairs is calculated by weighted fusion. The network affinity value is used to characterize the network communication quality between node pairs. The computational load index of each computing node and the network affinity value are input into a pre-constructed multi-constraint optimization model to generate a task scheduling scheme. The task scheduling scheme is executed, and network performance changes are continuously monitored after task deployment. When the network performance deviation exceeds a preset threshold, rescheduling is triggered.

2. The cloud computing resource collaborative scheduling and management method according to claim 1, characterized in that, The network communication data also includes packet loss rate information between node pairs. The process of normalizing the bandwidth and latency information based on the communication matrix and then calculating the network affinity value between node pairs using a weighted fusion method includes: The extreme values ​​of various network performance parameters in the communication matrix are extracted as the normalization benchmark. Using the normalization benchmark, the bandwidth information and the latency information are normalized respectively; The normalized bandwidth information, latency information, and packet loss rate information are each assigned a corresponding weight coefficient, and the network affinity value is calculated by weighted fusion.

3. The cloud computing resource collaborative scheduling and management method according to claim 2, characterized in that, The process of assigning corresponding weight coefficients to the normalized bandwidth information, latency information, and packet loss rate information includes: Determine the task type of the task to be scheduled, including data-intensive, latency-sensitive, and reliability-sensitive tasks; Based on the task type of the task to be scheduled, different weighting coefficients are assigned to the normalized bandwidth information, latency information and packet loss rate information. Among them, the highest bandwidth weight coefficient is assigned to data-intensive tasks, the highest latency weight coefficient is assigned to latency-sensitive tasks, and the highest packet loss rate weight coefficient is assigned to reliability-sensitive tasks.

4. The cloud computing resource collaborative scheduling and management method according to claim 1, characterized in that, The generation of the inter-node communication matrix based on the network communication data includes: The collected network communication data is smoothed using a sliding time window, and the smoothed network performance parameter values ​​are calculated using a weighted moving average method. The smoothed network performance parameter values ​​are organized into the communication matrix, wherein the communication matrix is ​​stored in a sparse matrix format, recording only the node pairs with communication relationships and their corresponding network performance parameter values.

5. The cloud computing resource collaborative scheduling and management method according to claim 4, characterized in that, The sliding time window smoothing process uses a weighted method to assign different weights to data collected at different time points. The closer the data point is to the current time, the higher the corresponding weight coefficient.

6. A cloud computing resource collaborative scheduling and management method according to any one of claims 1 to 5, characterized in that, The step of inputting the computational load index of each computing node and the network affinity value into a pre-constructed multi-constraint optimization model to generate a task scheduling scheme includes: An objective function is constructed based on network communication cost and computational load cost. The network communication cost is calculated based on the network affinity between the task to be scheduled and the associated nodes, and the computational load cost is calculated based on the computational load index of the candidate nodes. The multi-constraint optimization model uses at least one of the following constraints: node compatibility constraints, network traffic capacity constraints, quality of service assurance constraints, and load balancing soft constraints. Solving the objective function yields a task scheduling scheme that allocates tasks to computing nodes.

7. A cloud computing resource collaborative scheduling and management method according to any one of claims 1 to 5, characterized in that, The step of continuously monitoring network performance changes after task deployment and triggering rescheduling when the network performance deviation exceeds a preset threshold includes: If the deviation between the actual network performance parameter value and the expected network performance parameter value of any key node pair exceeds the first preset threshold; And / or, If the proportion of congested links to the total number of monitored links exceeds the second preset threshold, rescheduling will be triggered.

8. A cloud computing resource collaborative scheduling and management platform, characterized in that, include: The network awareness module is used to collect network communication data between computing nodes of the cloud platform. The network communication data includes bandwidth information and latency information between node pairs, and generates an inter-node communication matrix based on the network communication data. The affinity calculation module is used to calculate the network affinity value between node pairs by normalizing the bandwidth information and the delay information based on the communication matrix and then using a weighted fusion method. The network affinity value characterizes the network communication quality between node pairs. The scheduling engine is used to input the computing load index of each computing node and the network affinity value into a pre-built multi-constraint optimization model to generate a task scheduling scheme. The execution module is used to execute the task scheduling scheme and continuously monitor network performance changes after task deployment. When the network performance deviation exceeds a preset threshold, rescheduling is triggered.

9. A cloud computing resource collaborative scheduling and management platform according to claim 8, characterized in that, The execution module is also used for: After triggering rescheduling, the updated network performance data is fed back to the network awareness module to update the communication matrix, and the scheduling engine is triggered to regenerate the task scheduling scheme based on the updated communication matrix.

10. The cloud computing resource collaborative scheduling and management platform according to claim 8, characterized in that, Also includes: The policy management module is used to receive scheduling policy parameters configured by the administrator. The scheduling policy parameters include the weight coefficients of network performance parameters, the rescheduling trigger threshold, and the optimization target preference settings, and provide the corresponding policy parameter configurations to the affinity calculation module and the scheduling engine.

Images