Cloud computing resource scheduling method and system based on big data

Abstract

The invention relates to the technical field of resource allocation, in particular to a cloud computing resource scheduling method and system based on big data, and the method comprises the following steps: collecting resource use data from various cloud computing environments, and obtaining resource consumption dynamic data by recording the access frequency and use duration of each resource; according to the method, the resource node graph is drawn, the data transmission record between the nodes is analyzed, and the resource dependency matrix is established, so that the insight of resource mobility in the cloud environment is enhanced. The key resource path can be effectively identified by means of interaction strength weight analysis between resource nodes, and the bottleneck and the efficient area in resource distribution can be accurately judged. The resource utilization efficiency and the system response speed are improved by reconfiguring the load abnormal nodes and optimizing the resource allocation. Besides, resource allocation tasks are scheduled according to windows, and resource allocation is dynamically adjusted according to real-time information, so that resource management is more flexible and can adapt to rapid change of requirements.

Description

Technical Field

[0001] The present invention relates to the technical field of resource allocation, and in particular to a cloud computing resource scheduling method and system based on big data. Background Art

[0002] The field of resource allocation technology involves the optimization management and allocation strategies of key resources such as computing resources, network bandwidth, and storage capacity. This ensures the efficient operation of the system and adapts to the needs of different applications and services. It includes resource scheduling from data center management to cloud computing platforms, multi-tenant environments, and other large-scale distributed systems. Effective resource allocation strategies can improve resource utilization, reduce energy consumption, optimize costs, and improve user experience.

[0003] Among them, cloud computing resource scheduling methods are usually used to focus on how to efficiently allocate and manage computing resources in a cloud environment. The method optimizes the allocation of resources through dynamic scheduling algorithms to support the performance requirements and cost efficiency of different applications. Uses include ensuring the stability of application performance, maximizing the utilization of hardware and software resources, and automatically adjusting resource allocation when demand changes. This scheduling technology is particularly suitable for cloud services that need to process large amounts of data and high concurrent requests, such as big data analysis, online transaction processing, and Internet applications.

[0004] Although existing technologies have made some progress in resource allocation and management, they often show their limitations in the face of rapid changes and high concurrent requests in cloud environments. When processing large amounts of data and high concurrent requests, it is difficult to fully adapt to the dynamic changes in resource demand, resulting in untimely or over-concentrated resource allocation, which affects system performance and user experience. In addition, existing technologies lack in-depth analysis of resource usage patterns during resource scheduling, resulting in non-optimal resource allocation, which sometimes leaves resources idle and causes unnecessary waste of energy and costs. For example, in a data center, inaccurate resource scheduling may lead to excessive cooling and power consumption, further causing an increase in operating costs and an increase in environmental burdens. Summary of the invention

[0005] The purpose of the present invention is to solve the shortcomings of the prior art and to propose a cloud computing resource scheduling method and system based on big data.

[0006] In order to achieve the above object, the present invention adopts the following technical solution: a cloud computing resource scheduling method based on big data, comprising the following steps:

[0007] S1: Collect resource usage data from multiple cloud computing environments, and obtain dynamic resource consumption data by recording the access frequency and usage duration of each resource;

[0008] S2: using the resource consumption dynamic data, drawing a resource node graph, establishing the relationship weights between nodes by analyzing the data transmission records and interaction strength between nodes, and obtaining a resource dependency matrix;

[0009] S3: Based on the resource dependency matrix, analyze the interaction intensity weights between resource nodes, identify key resource paths, and obtain key resource analysis results;

[0010] S4: Based on the key resource analysis results, reallocate resources of nodes with abnormal distribution or load, adjust the weights of nodes connected to other nodes, and obtain optimized resource configuration;

[0011] S5: According to the optimized resource configuration, the urgency and resource processing capacity of each resource request are determined, and the resource allocation tasks are scheduled according to the windows. Each window is allocated resources according to the priority of resource requirements to obtain a time window resource allocation result;

[0012] S6: Based on the time window resource allocation result, continuously monitor resource consumption and request data, dynamically adjust resource allocation within the time window according to real-time information, and obtain a dynamically adjusted resource plan.

[0013] The present invention has the following improvements: the dynamic data of resource consumption includes the average load of computing resources, the remaining capacity ratio of storage resources, and the peak flow of network resources; the resource dependency matrix includes the dependency between resources, the connectivity index of resource nodes, and the frequency of resource exchange; the key resource analysis results include traffic-intensive resource nodes, network areas with concentrated connections, and nodes with response times exceeding standard thresholds; the optimized resource configuration includes an adjusted resource allocation ratio, inter-node communication efficiency, and data transmission path; the time window resource allocation results include the resource utilization rate of the time window, the satisfaction degree of resource requests, and the timeliness of resource allocation; and the dynamically adjusted resource plan includes real-time updated resource configuration and an adjusted priority queue.

[0014] The present invention is improved in that the specific steps of collecting resource usage data from a cloud computing environment and obtaining resource consumption dynamic data by recording the access frequency and usage duration of each resource are as follows:

[0015] S101: Collect resource usage data from multiple cloud computing environments, including usage records of computing power, storage space, and network bandwidth, to obtain a resource data set;

[0016] S102: Based on the resource data set, analyzing the upload and download traffic data of the network bandwidth, recording the access frequency and usage duration of each resource, and obtaining a resource usage frequency record;

[0017] S103: Based on the resource usage frequency records, a resource consumption trend model is constructed by clustering analysis of the usage peaks and troughs of each resource type to obtain dynamic resource consumption data.

[0018] The present invention is improved in that the resource consumption dynamic data is used to draw a resource node diagram, and the relationship weights between nodes are established by analyzing the data transmission records and interaction strength between nodes. The specific steps of obtaining the resource dependency matrix are as follows:

[0019] S201: Based on the resource consumption dynamic data, mark the location and basic status of each resource, and form a resource node interaction record by recording data transmission events between nodes;

[0020] S202: Based on the resource node interaction records, the frequency and volume of data exchange between each pair of nodes are counted through the Laplace matrix, and the relationship strength index between the nodes is quantified to generate the node relationship weight;

[0021] S203: Based on the node relationship weights, according to the response time and resource request amount between the nodes, the weight values ​​between the nodes are adjusted to obtain a resource dependency matrix.

[0022] The present invention is improved in that the Laplace matrix is ​​according to the formula:

[0023]

[0024] The frequency and volume of data exchange between each pair of nodes are counted to obtain the node relationship weight;

[0025] Among them, L is the Laplace matrix, which represents the relationship weight between nodes, and β is the coefficient used to adjust The influence of the term, D is the frequency matrix of data exchange between nodes, P is the volume matrix of data exchange between nodes, ∥A∥ is the norm of matrix A, which represents the total exchange volume of nodes, α is the coefficient used to adjust the influence of the term (A⊙C), A is the direct exchange information matrix, C is the connectivity matrix, and each element C ij represents the connection strength or quality between node i and node j, ⊙ is the Hadamard product, which represents the item-by-item product of the corresponding elements of matrices A and C.

[0026] The present invention is improved in that, based on the resource dependency matrix, the interaction intensity weights between resource nodes are analyzed, key resource paths are identified, and the specific steps of obtaining key resource analysis results are as follows:

[0027] S301: extracting the interaction data of the node pairs based on the resource dependency matrix, recording the frequency and volume of each data exchange, and generating an interaction intensity record table;

[0028] S302: Based on the interaction intensity record table, select nodes exceeding a set threshold as key nodes by screening through frequency and intensity data, mark them, and generate a key node list;

[0029] S303: Based on the key node list, a network diagram is constructed, key nodes are connected and data flow paths are mapped to obtain key resource analysis results.

[0030] The present invention is improved in that, based on the key resource analysis results, the resources of the nodes with abnormal distribution or load are reallocated, and the weights of the nodes connected to the remaining nodes are adjusted to obtain the optimized resource configuration in the following specific steps:

[0031] S401: Based on the key resource analysis results, determine the resource usage of the abnormal load node, recalculate the resource allocation ratio according to the node performance parameters, and generate a load adjustment plan;

[0032] S402: adopting the load adjustment plan, selecting a path to reallocate resources, optimizing data transmission efficiency by adjusting the path, recording resource flow adjustment, and generating a resource reallocation record;

[0033] S403: Using the resource reconfiguration record, update the weights between nodes, redefine the data transmission path between nodes, and obtain optimized resource configuration.

[0034] The present invention is improved in that, according to the optimized resource configuration, the urgency and resource processing capacity of each resource request are judged, resource allocation tasks are scheduled according to windows, and each window is allocated resources according to the priority of resource requirements. The specific steps of obtaining the time window resource allocation result are as follows:

[0035] S501: judging the urgency and resource processing capacity of each resource request according to the optimized resource configuration, collecting and arranging the demand data of each resource, classifying them, and generating a resource priority list;

[0036] S502: Formulate a resource allocation plan for each time window through the resource priority list, adjust the proportion and order of on-demand allocation of each type of resource, and generate an allocation strategy table;

[0037] S503: Execute resource allocation in each time window according to the allocation strategy table, record the execution status and effect of resource allocation in real time, and obtain the time window resource allocation result.

[0038] The present invention is improved in that, based on the time window resource allocation result, resource consumption and request data are continuously monitored, and resource allocation within the time window is dynamically adjusted according to real-time information. The specific steps of obtaining the dynamically adjusted resource plan are as follows:

[0039] S601: Based on the time window resource allocation result, set monitoring points to track the consumption speed and request frequency of each type of resource, and generate resource consumption monitoring records;

[0040] S602: Analyze the usage trend and demand change of each type of resource by using the resource consumption monitoring record, identify the resource allocation area that needs to be optimized, and generate resource adjustment demand information;

[0041] S603: updating and adjusting the resource allocation strategy within the time window according to the resource adjustment requirement information to obtain a dynamic adjustment resource plan.

[0042] A cloud computing resource scheduling system based on big data, the system comprising:

[0043] The data collection module collects the usage records of computing power, storage space and network bandwidth based on various cloud computing environments to generate resource usage data sets;

[0044] The resource dependency analysis module draws a resource node diagram based on the resource usage data set, records data transmission events between nodes, quantifies the interaction intensity between nodes, and generates a resource dependency matrix;

[0045] The key resource identification module analyzes the interaction data based on the resource dependency matrix, identifies nodes whose interaction intensity exceeds a set threshold, marks them as key nodes, and constructs a key resource path network diagram to generate key resource analysis results;

[0046] The resource scheduling module prioritizes resource requirements based on the key resource analysis results, formulates resource allocation plans as needed, and executes the plans within each time window to generate time window resource allocation results;

[0047] The dynamic adjustment module sets real-time monitoring points to track resource consumption and request data based on the time window resource allocation result, dynamically adjusts the resource allocation strategy according to the resource usage trend, and generates a dynamically adjusted resource plan.

[0048] Compared with the prior art, the advantages and positive effects of the present invention are:

[0049] In the present invention, by drawing a resource node diagram and analyzing the data transmission records between nodes, a method of establishing a resource dependency matrix is ​​used to enhance the insight into the mobility of resources in the cloud environment. Relying on the interaction intensity weight analysis between resource nodes, the key resource path can be effectively identified, which helps to accurately determine the bottlenecks and efficient areas in resource distribution. By reconfiguring the load abnormality nodes and optimizing the resource allocation method, the resource utilization efficiency and system response speed are improved. In addition, the resource allocation tasks are scheduled according to the window, and the resource allocation is dynamically adjusted according to the real-time information, so that resource management is more flexible, can adapt to the rapid changes in demand, and reduce resource waste. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] Figure 1 A flowchart of a cloud computing resource scheduling method based on big data is proposed for the present invention;

[0051] Figure 2 This is a detailed flow chart of step S1 in the present invention;

[0052] Figure 3 This is a detailed flow chart of step S2 in the present invention;

[0053] Figure 4 This is a detailed flow chart of step S3 in the present invention;

[0054] Figure 5 This is a detailed flow chart of step S4 in the present invention;

[0055] Figure 6 This is a detailed flow chart of step S5 in the present invention;

[0056] Figure 7 This is a detailed flow chart of step S6 in the present invention;

[0057] Figure 8 A module diagram of a cloud computing resource scheduling system based on big data is proposed for the present invention. DETAILED DESCRIPTION

[0058] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0059] In the description of the present invention, it should be understood that the terms "length", "width", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside", etc., indicating positions or positional relationships, are based on the positions or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as limiting the present invention. In addition, in the description of the present invention, the meaning of "multiple" is two or more, unless otherwise clearly and specifically defined.

[0060] Example

[0061] See also Figure 1 The present invention provides a technical solution: a cloud computing resource scheduling method based on big data, comprising the following steps:

[0062] S1: Collect resource usage data from multiple cloud computing environments, and obtain dynamic resource consumption data by recording the access frequency and usage duration of each resource;

[0063] S2: Use dynamic data of resource consumption to draw a resource node diagram, and establish the relationship weights between nodes by analyzing the data transmission records and interaction intensity between nodes to obtain the resource dependency matrix;

[0064] S3: Based on the resource dependency matrix, analyze the interaction intensity weights between resource nodes, identify key resource paths, and obtain key resource analysis results;

[0065] S4: Based on the key resource analysis results, redistribute the resources of nodes with abnormal distribution or load, adjust the weights of their connections to the remaining nodes, and obtain the optimized resource configuration;

[0066] S5: Based on the optimized resource configuration, the urgency and resource processing capacity of each resource request are determined, and the resource allocation tasks are scheduled by window. Each window is allocated resources according to the priority of resource requirements to obtain the time window resource allocation result;

[0067] S6: Based on the time window resource allocation result, continuously monitor resource consumption and request data, dynamically adjust resource allocation within the time window according to real-time information, and obtain a dynamically adjusted resource plan.

[0068] Dynamic data on resource consumption include the average load of computing resources, the remaining capacity ratio of storage resources, and the peak traffic of network resources. The resource dependency matrix includes the dependency between resources, the connectivity index of resource nodes, and the frequency of resource exchange. Key resource analysis results include traffic-intensive resource nodes, network areas with concentrated connections, and nodes whose response time exceeds the standard threshold. The optimized resource configuration includes the adjusted resource allocation ratio, communication efficiency between nodes, and data transmission path. The time window resource allocation results include the resource utilization rate of the time window, the satisfaction degree of resource requests, and the timeliness of resource allocation. Dynamic adjustment of resource plans includes real-time updated resource configuration and adjusted priority queues.

[0069] See also Figure 2 ,To collect resource usage data from the cloud computing environment, the specific steps to obtain dynamic data of resource consumption by recording the access frequency and usage duration of each resource are as follows:

[0070] S101: Collect resource usage data from multiple cloud computing environments, including usage records of computing power, storage space, and network bandwidth, and obtain a resource data set in the following process;

[0071] In various cloud computing environments, resource usage data is systematically collected, covering detailed usage records of computing power, storage space, and network bandwidth. This includes collecting CPU usage, memory usage, storage I / O operations, and network bandwidth consumption data from different physical and virtual servers. For example, for computing power, the usage and idle time of each CPU core are recorded; for storage space, the total capacity, used capacity, and usage frequency of each storage unit are recorded; for network bandwidth, the upload and download rates of each network interface are recorded. These data are classified and aggregated into a resource data set, in which each data item is timestamped and resource identified, providing accurate raw data for subsequent analysis.

[0072] S102: Based on the resource data set, the upload and download traffic data of the network bandwidth are analyzed, and the access frequency and usage duration of each resource are recorded. The process of obtaining the resource usage frequency record is as follows;

[0073] Based on the resource data set obtained, detailed data processing and analysis are performed. First, the upload and download traffic data of the network bandwidth are counted by time period, and then the access frequency and usage duration of each resource are calculated. For the calculation of access frequency, the formula is used: Among them, f i represents the average access frequency of the i-th type of resources, u it is the number of accesses to the i-th resource in the t-th time unit, and T is the total number of time units. In addition, the weighted average method is used to calculate the usage time, and the formula is as follows: Here, di is the average usage time of the i-th resource, l it is the usage time of the i-th resource in the t-th time unit, where t represents the time unit. These two calculation methods reasonably reflect the usage and frequency of each resource type.

[0074] S103: Based on the resource usage frequency records, the peak and valley usage of each resource type are analyzed by clustering, and a model of resource consumption trend is constructed. The process of obtaining dynamic resource consumption data is as follows;

[0075] Based on the resource usage frequency and usage duration records, cluster analysis is used to identify the peak and valley periods of each resource type. The K-means clustering algorithm is used to classify the data points according to the resource usage frequency f i and duration of use i The data points are grouped into two categories: peak and trough. After the classification is completed, a quadratic curve fitting is applied to the data points of each category to reveal the trend pattern of resource usage. The formula of the fitting model is: y = ax 2 +bx+cHere, x represents the time unit, y represents the resource usage, and a, b, c are coefficients obtained by data fitting, representing the shape and trend of the curve. This method not only reveals the dynamic changes in resource consumption, but also provides a scientific basis for the effective management and optimal allocation of resources. Through this series of analysis, dynamic data on various resource consumption are obtained, providing accurate data support for resource scheduling and optimization.

[0076] See also Figure 3 ,Using the dynamic data of resource consumption, we draw a resource node graph, and establish the relationship weights between nodes by analyzing the data transmission records and the interaction intensity between nodes. The specific steps to obtain the resource dependency matrix are as follows:

[0077] S201: Based on the dynamic data of resource consumption, the location and basic status of each resource are marked, and the process of forming resource node interaction records by recording data transmission events between nodes is as follows;

[0078] In the process based on dynamic data of resource consumption, the location information and resource status of each node are first tracked by network monitoring. Through sensors and logs, CPU usage, memory usage, storage I / O operations, and network bandwidth utilization are collected in real time. Data is sampled every five minutes and stored in a central database. Features are extracted from these raw data through data mining technology and mapped to each node to ensure that the current status of each node is accurately recorded. Next, data transmission events are recorded through network transmission monitoring, including information such as the size of the data packet, the source node and the destination node, and the timestamp.

[0079] S202: Based on the resource node interaction records, the data exchange frequency and volume between each pair of nodes are counted through the Laplace matrix, and the relationship strength index between the nodes is quantified. The process of generating the node relationship weight is as follows;

[0080] Extract key information from the collected resource node interaction data, including the frequency and volume of data exchange between each pair of nodes. With this information, a matrix reflecting the direct connection between nodes is constructed. Then, another matrix is ​​further constructed, each element of which represents the sum of exchanges between a node and all other nodes. By combining the information of these two matrices, the strength of the connection between nodes is revealed. The direct exchange information between nodes and the total exchange volume of each node are taken into account to quantify and understand the strength of the relationship between any two nodes. The result of this analysis is a new matrix that provides a quantitative view of the relationship between nodes and provides the necessary basic data for the next step of weight adjustment.

[0081] The Laplace matrix follows the formula:

[0082]

[0083] The frequency and volume of data exchange between each pair of nodes are counted to obtain the node relationship weight;

[0084] Among them, L is the Laplace matrix, which represents the relationship weight between nodes. β is the coefficient used to adjust The influence of the term depends on the specific configuration of the system or the weight determined by the experiment. D is the frequency matrix of data exchange between nodes, and each element D ij represents the number of exchanges between nodes i and j. P is the volume matrix of data exchange between nodes, where P ij represents the amount of data exchanged between node i and node j. ∥A∥ is the norm of matrix A, which represents the total amount of data exchanged between nodes. α is a coefficient used to adjust the influence of the (A⊙C) term, which is based on system requirements. A is the direct exchange information matrix. C is the connectivity matrix, and each element C ij It represents the strength or quality of the connection between node i and node j. ⊙ is the Hadamard product, which represents the item-by-item product of the corresponding elements of matrices A and C.

[0085] The calculation process is as follows:

[0086] The data exchange frequency matrix D and data volume matrix P between nodes:

[0087] D and P are derived from the resource node interaction records. Each data exchange event between nodes is recorded, including the frequency of exchange f ij and the amount of data exchanged v ij , where i and j represent different nodes.

[0088] D ij =f ij Represents the number of exchanges between node i and node j.

[0089] P ij =v ij Represents the amount of data exchanged between node i and node j.

[0090] The norm of the total exchange matrix A is ∥A∥:

[0091] A can be D, or another matrix representing the frequencies of exchanges.

[0092] Assume A = D. The norm ∥A∥ is a scalar that represents the total amount of all node exchanges and can be calculated as the square root of the sum of the squares of all elements of the A matrix.

[0093] Connectivity Matrix C:

[0094] C is obtained through network monitoring, where C ij Reflects the quality or strength of the connection between nodes i and j. Including factors such as latency and bandwidth utilization.

[0095] Hadamard product A⊙C:

[0096] It represents the product of the corresponding elements of matrices A and C. The resulting matrix reflects the comprehensive consideration of the amount of exchange between nodes and the quality of the connection.

[0097] First calculate

[0098] D·P represents the product of corresponding elements of D and P, that is, d ij ×p ij , which represents the product of exchange frequency and exchange volume between nodes.

[0099] Dividing this product matrix by the square root of the norm of A, we get

[0100] calculate

[0101] Multiply the result of the previous step by the coefficient β to adjust its contribution to the final result L.

[0102] Calculate α·(A⊙C):

[0103] The matrix calculated by the Hadamard product A⊙C is multiplied by α, which reflects the influence of the adjustment of the direct exchange of information and connection quality between nodes.

[0104] Finally, from Subtracting α·(A⊙C) yields the Laplace matrix L. The Laplace matrix L describes the interrelationships and dependencies between nodes based on the amount of data exchange and the quality of connections, helping the system determine critical resource paths, optimize resource allocation, and improve data transmission efficiency and overall system performance. In this way, the system can dynamically adjust resource allocation to respond to real-time demand changes, ultimately ensuring efficient resource utilization and rapid response to business needs.

[0105] S203: Based on the node relationship weights, according to the response time and resource request amount between the nodes, the weight values ​​between the nodes are adjusted to obtain the resource dependency matrix;

[0106] The obtained node relationship weights are combined with the actual operation data and adjusted dynamically. The specific operation includes analyzing the response time and resource request volume between nodes. By integrating these actual operation data into the previously calculated weights, the weight values ​​between each pair of nodes can be adjusted to make them closer to the needs of actual network operations. For example, if a node has a long response time to another node but frequent resource requests, the weight between the two nodes will be increased accordingly, indicating that although the response time is long, there is a strong dependence or important resource exchange demand between the two nodes.

[0107] See also Figure 4 ,Based on the resource dependency matrix, the interaction intensity weights between resource nodes are analyzed,,key resource paths are identified, and the specific steps to obtain the key resource analysis results are as follows:

[0108] S301: Based on the resource dependency matrix, the interaction data of the node pairs is extracted, the frequency and volume of each data exchange are recorded, and the process of generating an interaction intensity record table is as follows;

[0109] Based on the resource dependency matrix, a detailed analysis is conducted on the interactions between nodes. First, the interaction data of node pairs are extracted, including the data exchange frequency f between each pair of nodes. ij and data volume v ij For each pair of nodes i and j, record all data exchange instances within a specific time window and calculate the total frequency and total volume of these instances. The interaction intensity I is calculated using the following formula ij : Here, f ijk is the exchange frequency of nodes i and j in the kth exchange, v ijk is the corresponding data volume, and K is the number of exchanges considered. This formula helps determine the average intensity of exchanges between nodes. These data are organized into an interaction intensity record table, which lists the interaction frequency and volume of each pair of nodes in detail, providing basic data for subsequent analysis.

[0110] S302: Based on the interaction intensity record table, through frequency and intensity data screening, nodes exceeding a set threshold are selected as key nodes, and marked, and the process of generating a key node list is as follows;

[0111] By analyzing the interaction strength record table, the key nodes are screened using the set threshold T. The selection of key nodes is based on the criterion that the interaction strength exceeds T. The specific method is as follows: for each node pair i, j, if its interaction strength I ij If the threshold T is exceeded, the interaction between the two nodes is considered critical. The threshold T is set based on historical data and network performance indicators. The specific formula is: T = μ + λσ, where μ is the average value of the interaction strength of all node pairs, σ is the standard deviation, and λ is the adjustment coefficient used to adjust the sensitivity according to network conditions. Applying this threshold, the selected critical node pairs will be marked and included in the critical node list.

[0112] S303: Based on the key node list, a network diagram is constructed, key nodes are connected and data flow paths are mapped, and the process of obtaining key resource analysis results is as follows;

[0113] Based on the list of key nodes, a network diagram is constructed to visualize the key nodes and their connections. This network diagram is constructed based on the data flow path between nodes. Each key node is treated as a vertex in the diagram. The connections between key nodes are represented by edges. The width and color intensity of the edges reflect the strength of the interaction. The construction of the network diagram uses graph algorithms, such as the force-directed graph algorithm, to optimize the visualization of node positions and connections. In this way, the generated network diagram not only intuitively shows the key path of data flow, but also reveals possible performance bottlenecks or optimization points, completing the overall process of key resource analysis and effectively guiding resource management and optimization decisions.

[0114] See also Figure 5 ,Based on the key resource analysis results, the resources of nodes with abnormal distribution or load are reallocated, and the weights of nodes connected to the rest of the nodes are adjusted. The specific steps to obtain the optimized resource configuration are as follows:

[0115] S401: Based on the key resource analysis results, determine the resource usage of the abnormal load node, recalculate the resource allocation ratio according to the node performance parameters, and generate a load adjustment plan;

[0116] Based on the key resource analysis results, first identify those nodes that show abnormal load. The abnormality of these nodes is usually manifested as resource utilization significantly higher than the average level or abnormal response time delay. For these nodes, collect their CPU usage, memory usage, network bandwidth utilization and other performance parameters, and recalculate the resource allocation ratio using the following formula: Among them, P iis the new resource allocation ratio of node i, N i is the performance score of node i, which is a comprehensive assessment of factors such as CPU usage and memory usage, and n is the total number of nodes. The performance score is calculated using a weighted average method, where the weight of each performance indicator is determined based on historical data and business importance. This method can be used to perform targeted resource reallocation on abnormally loaded nodes and generate a detailed load adjustment plan.

[0117] S402: adopting a load adjustment plan, selecting a path to reallocate resources, optimizing data transmission efficiency by adjusting the path, and recording resource flow adjustment conditions, and generating a resource reallocation record;

[0118] When implementing a load adjustment plan, the optimal path is selected for resource reallocation. This process includes analyzing the network topology to determine the optimal path for data transmission. The shortest path algorithm in graph theory is used to calculate the shortest path for data transmission between nodes, and the path is adjusted according to the bandwidth and delay characteristics to optimize data transmission efficiency. After the adjustment, the resource flow direction of each node is recorded and a resource reconfiguration record is generated. The record details the reallocation of resources from overloaded nodes to other nodes, including resource type, quantity, and flow direction.

[0119] S403: Using the resource reconfiguration record, updating the weights between nodes, redefining the data transmission path between nodes, and obtaining the optimized resource configuration process is as follows;

[0120] Use resource reconfiguration records to update the weights between nodes. The weight adjustment between nodes is based on their data transmission efficiency and resource load. The specific weight adjustment formula is: Among them, W ij is the weight from node i to node j, E ij is the efficiency between nodes (considering bandwidth and delay), L ij is the load index of the connection between nodes, and the addition of one is to prevent division by zero errors. In this way, the data flow between nodes can be precisely controlled to ensure a more reasonable resource allocation. Ultimately, these adjustments help form an optimized resource configuration and improve the response speed to business needs.

[0121] See also Figure 6 ,According to the optimized resource configuration, the urgency and resource processing capacity of each resource request are judged, and the resource allocation tasks are scheduled by window. Each window allocates resources according to the priority of resource requirements. The specific steps to obtain the time window resource allocation result are as follows:

[0122] S501: According to the optimized resource configuration, the urgency and resource processing capacity of each resource request are determined, the demand data of each resource is collected and sorted, and the classification is performed to generate a resource priority list;

[0123] Based on the optimized resource configuration, a resource demand analysis is performed. During this process, the request urgency and processing capacity of each resource are evaluated to determine its priority. The specific operations include: analyzing each resource of CPU, memory, storage and network bandwidth separately to evaluate its current load and demand forecast in the short term. To this end, the usage data of each resource is collected, including peak usage, average usage and request frequency. Then, based on the business importance and usage data of the resource, the following formula is applied to prioritize the resource: P r =α·U avg +β·U peak +γ·F, where P r represents the priority score of resource r, U avg is the average utilization rate, U peak is the peak utilization rate, F is the request frequency, and α, β, and γ are the weight coefficients of each factor. These coefficients are adjusted according to the resource type and business impact. Finally, a resource priority list is generated, which clearly lists the priority of each resource for effective resource allocation.

[0124] S502: Formulate a resource allocation plan for each time window through the resource priority list, adjust the proportion and order of each type of resource to be allocated on demand, and generate an allocation strategy table;

[0125] Use the resource priority list to develop a resource allocation plan for each time window. This process involves determining the allocation ratio and order of each type of resource in different time windows. First, determine the allocation ratio of each resource in each time window based on the urgency and processing capacity of the resource. Then, use the following formula to calculate the allocation index of each type of resource: Among them, D rt is the allocation index of resource r in time window t, Q r is the total number of requests for the resource, T t is the length of the time window, and S is the proportional coefficient adjusted according to the system capacity. Through this method, a detailed allocation strategy table is generated, which includes the specific allocation strategy of each resource in each time window, ensuring the fairness and efficiency of resource allocation.

[0126] S503: According to the allocation strategy table, resource allocation is performed in each time window, and the execution status and effect of resource allocation are recorded in real time. The process of obtaining the resource allocation result of the time window is as follows;

[0127] According to the allocation strategy table, resource allocation is performed in each time window. During this process, the system will record the execution and effect of resource allocation in real time, including the load situation before and after resource allocation and the change in processing capacity. Finally, the time window resource allocation result is generated, which shows the efficiency and satisfaction of each resource, providing a basis for further optimization, ensuring that resource allocation can meet business needs while maximizing resource utilization efficiency.

[0128] See also Figure 7 ,Based on the time window resource allocation results, we continuously monitor resource consumption and request data, and dynamically adjust the resource allocation within the time window according to real-time information. The specific steps for dynamically adjusting the resource plan are as follows:

[0129] S601: Based on the time window resource allocation result, set monitoring points to track the consumption speed and request frequency of each type of resource, and generate the resource consumption monitoring record process as follows;

[0130] To ensure the efficiency and effectiveness of resource allocation, we first set up multiple monitoring points in the system, which are responsible for tracking the consumption speed and request frequency of each type of resource. Specific operations include: setting up independent monitoring modules for CPU, memory, storage and network resources, each module records the real-time consumption data and request events of the corresponding resources. For example, the CPU monitoring module records the usage changes and task request frequency of each core, and the storage monitoring module tracks the speed and frequency of read and write operations. The collected data will be updated in real time to the resource consumption monitoring record, forming a continuous data stream, providing a basis for subsequent data analysis.

[0131] S602: Analyze the usage trend and demand change of each type of resource by using the resource consumption monitoring record, identify the resource allocation area that needs to be optimized, and generate the resource adjustment demand information;

[0132] Using the updated resource consumption monitoring records, in-depth analysis is performed to identify trends in resource usage and changes in demand. By comparing data from different time periods, peak and trough periods of resource usage, as well as significant changes in request frequency, are identified. In this step, statistical analysis methods such as moving average and differential analysis are applied. The specific analysis formula can be expressed as: Among them, ΔR t is the change in resource usage in the tth time window, R i is the resource usage in the i-th time window, and n is the number of time windows considered. Through this method, the changing trend of resource demand can be clearly seen, and resource adjustment demand information can be generated based on it to identify resource areas that need to be optimized or reconfigured.

[0133] S603: According to the resource adjustment demand information, the allocation strategy of resources in the time window is updated and adjusted to obtain the process of dynamically adjusting the resource plan;

[0134] Based on the generated resource adjustment demand information, update and adjust the resource allocation strategy within the time window. During this process, the allocation ratio and priority of each resource will be re-evaluated, and resource allocation will be dynamically adjusted based on the actual resource usage and business needs. The adjustment strategy uses the following formula to optimize resource allocation: new =S old ×(1+α×ΔR t ) Among them, S new is the updated resource allocation strategy, S old is the previous strategy, α is the adjustment coefficient, which depends on the sensitivity of the resource and the urgency of the adjustment, ΔR t It is the change in resource demand. In this way, it is possible to flexibly respond to the actual situation of resource use, optimize resource allocation, improve overall efficiency and response speed, and thus form a dynamically adjusted resource plan to ensure efficient use of resources.

[0135] See also Figure 8 , a cloud computing resource scheduling system based on big data, the system comprising:

[0136] The data collection module collects the usage records of computing power, storage space and network bandwidth based on various cloud computing environments to generate resource usage data sets;

[0137] The resource dependency analysis module draws a resource node diagram based on the resource usage data set, records data transmission events between nodes, quantifies the interaction intensity between nodes, and generates a resource dependency matrix;

[0138] The key resource identification module is based on the resource dependency matrix. By analyzing the interaction data, it identifies the nodes whose interaction intensity exceeds the set threshold, marks them as key nodes, and constructs a key resource path network diagram to generate key resource analysis results.

[0139] The resource scheduling module prioritizes resource requirements based on key resource analysis results, formulates resource allocation plans as needed, and executes the plans within each time window to generate time window resource allocation results;

[0140] The dynamic adjustment module sets real-time monitoring points to track resource consumption and request data based on the time window resource allocation results, dynamically adjusts the resource allocation strategy according to resource usage trends, and generates a dynamically adjusted resource plan.

[0141] The above are only preferred embodiments of the present invention and are not intended to limit the present invention in other forms. Any technician familiar with the profession may use the technical contents disclosed above to change or modify them into equivalent embodiments with equivalent changes and apply them to other fields. However, any simple modification, equivalent change and modification made to the above embodiments based on the technical essence of the present invention without departing from the technical solution of the present invention still falls within the protection scope of the technical solution of the present invention.

Claims

1. A cloud computing resource scheduling method based on big data, characterized in that: The following steps are involved: Collect resource usage data from various cloud computing environments, and obtain dynamic resource consumption data by recording the access frequency and usage duration of each resource; Using the resource consumption dynamic data, a resource node diagram is drawn, and by analyzing the data transmission records and interaction strength between nodes, the relationship weights between nodes are established to obtain a resource dependency matrix; Based on the resource dependency matrix, analyzing the interaction intensity weights between resource nodes, identifying key resource paths, and obtaining key resource analysis results; Based on the key resource analysis results, reallocate resources of nodes with abnormal distribution or load, adjust the weights of nodes connected to other nodes, and obtain optimized resource configuration; According to the optimized resource configuration, the urgency and resource processing capacity of each resource request are judged, and the resource allocation tasks are scheduled according to the windows. Each window is allocated resources according to the priority of resource requirements to obtain the time window resource allocation result; Based on the time window resource allocation result, resource consumption and request data are continuously monitored, and resource allocation within the time window is dynamically adjusted according to real-time information to obtain a dynamically adjusted resource plan.

2. The cloud computing resource scheduling method based on big data according to claim 1, characterized in that: The dynamic data of resource consumption includes the average load of computing resources, the remaining capacity ratio of storage resources, and the peak traffic of network resources. The resource dependency matrix includes the dependency between resources, the connectivity index of resource nodes, and the frequency of resource exchange. The key resource analysis results include traffic-intensive resource nodes, network areas with concentrated connections, and nodes whose response time exceeds the standard threshold. The optimized resource configuration includes the adjusted resource allocation ratio, communication efficiency between nodes, and data transmission path. The time window resource allocation results include the resource utilization rate of the time window, the satisfaction degree of resource requests, and the timeliness of resource allocation. The dynamically adjusted resource plan includes real-time updated resource configuration and adjusted priority queues.

3. The cloud computing resource scheduling method based on big data according to claim 1, characterized in that: The resource consumption dynamic data is used to draw a resource node diagram, and the relationship weights between nodes are established by analyzing the data transmission records and interaction strength between nodes. The specific steps for obtaining the resource dependency matrix are as follows: Based on the resource consumption dynamic data, mark the location and basic status of each resource, and form a resource node interaction record by recording data transmission events between nodes; Based on the resource node interaction records, the frequency and volume of data exchange between each pair of nodes are counted through the Laplace matrix, and the relationship strength index between the nodes is quantified to generate the node relationship weight; Based on the node relationship weights, according to the response time and resource request amount between the nodes, the weight values ​​between the nodes are adjusted to obtain a resource dependency matrix.

4. The cloud computing resource scheduling method based on big data according to claim 3 is characterized in that: The Laplace matrix follows the formula: The frequency and volume of data exchange between each pair of nodes are counted to obtain the node relationship weight; Among them, L is the Laplace matrix, which represents the relationship weight between nodes, and β is the coefficient used to adjust The influence of the term, D is the frequency matrix of data exchange between nodes, P is the volume matrix of data exchange between nodes, ∥A∥ is the norm of matrix A, which represents the total exchange volume of nodes, α is the coefficient used to adjust the influence of the term (A⊙C), A is the direct exchange information matrix, C is the connectivity matrix, and each element C ij represents the connection strength or quality between node i and node j, ⊙ is the Hadamard product, which represents the item-by-item product of the corresponding elements of matrices A and C.

5. The cloud computing resource scheduling method based on big data according to claim 1, characterized in that: Based on the resource dependency matrix, the interaction intensity weights between resource nodes are analyzed, key resource paths are identified, and the specific steps for obtaining key resource analysis results are as follows: Based on the resource dependency matrix, extract the interaction data of the node pairs, record the frequency and volume of each data exchange, and generate an interaction intensity record table; Based on the interaction intensity record table, nodes exceeding a set threshold are selected as key nodes through frequency and intensity data screening, and are marked to generate a key node list; Based on the key node list, a network diagram is constructed, key nodes are connected and data flow paths are mapped to obtain key resource analysis results.

6. The cloud computing resource scheduling method based on big data according to claim 1, characterized in that: Based on the key resource analysis results, the specific steps of reallocating the resources of nodes with abnormal distribution or load and adjusting the weights of nodes connected to the remaining nodes to obtain the optimized resource configuration are as follows: Based on the key resource analysis results, determine the resource usage of the abnormal load node, recalculate the resource allocation ratio according to the node performance parameters, and generate a load adjustment plan; Adopting the load adjustment plan, selecting a path to reallocate resources, optimizing data transmission efficiency by adjusting the path, and recording resource flow adjustment to generate a resource reconfiguration record; The resource reconfiguration record is used to update the weights between nodes, redefine the data transmission path between nodes, and obtain optimized resource configuration.

7. The cloud computing resource scheduling method based on big data according to claim 1, characterized in that: According to the optimized resource configuration, the urgency and resource processing capacity of each resource request are judged, and the resource allocation tasks are scheduled according to the window. Each window allocates resources according to the priority of resource requirements. The specific steps for obtaining the time window resource allocation result are as follows: According to the optimized resource configuration, the urgency and resource processing capacity of each resource request are determined, the demand data of each resource is collected and sorted, and classified to generate a resource priority list; By using the resource priority list, a resource allocation plan is formulated for each time window, the proportion and order of each type of resource allocated on demand are adjusted, and an allocation strategy table is generated; According to the allocation strategy table, resource allocation is performed in each time window, and the execution status and effect of resource allocation are recorded in real time to obtain the time window resource allocation result.

8. The cloud computing resource scheduling method based on big data according to claim 1, characterized in that: Based on the time window resource allocation result, the resource consumption and request data are continuously monitored, and the resource allocation within the time window is dynamically adjusted according to the real-time information. The specific steps of obtaining the dynamic adjustment resource plan are as follows: Based on the time window resource allocation result, monitoring points are set to track the consumption speed and request frequency of each type of resource, and a resource consumption monitoring record is generated; Utilizing the resource consumption monitoring records, analyzing the usage trends and demand changes of each type of resources, identifying resource allocation areas that need to be optimized, and generating resource adjustment demand information; According to the resource adjustment demand information, the allocation strategy of resources within the time window is updated and adjusted to obtain a dynamically adjusted resource plan.

9. A cloud computing resource scheduling system based on big data, characterized in that: According to the big data-based cloud computing resource scheduling method according to any one of claims 1 to 8, the system comprises: The data collection module collects the usage records of computing power, storage space and network bandwidth based on various cloud computing environments to generate resource usage data sets; The resource dependency analysis module draws a resource node diagram based on the resource usage data set, records data transmission events between nodes, quantifies the interaction intensity between nodes, and generates a resource dependency matrix; The key resource identification module analyzes the interaction data based on the resource dependency matrix, identifies nodes whose interaction intensity exceeds a set threshold, marks them as key nodes, and constructs a key resource path network diagram to generate key resource analysis results; The resource scheduling module prioritizes resource requirements based on the key resource analysis results, formulates resource allocation plans as needed, and executes the plans within each time window to generate time window resource allocation results; The dynamic adjustment module sets real-time monitoring points to track resource consumption and request data based on the time window resource allocation result, dynamically adjusts the resource allocation strategy according to the resource usage trend, and generates a dynamically adjusted resource plan.

Images