A method for network resource elastic scheduling

Abstract

The application relates to the technical field of network resource scheduling engineering, and discloses a network resource elastic scheduling method, resource monitoring and demand sensing: distributed monitoring probes are arranged at key nodes of a network, multi-dimensional network resource data is collected, bandwidth utilization, packet delay, packet loss rate, server CPU load, memory occupation, storage I / O rate, the sampling frequency is 1 time per second to 10 times per second, a business flow prediction model is constructed by using a machine learning algorithm, and resource demand is perceived in combination with real-time business request information; the network resource elastic scheduling method has the following beneficial effects: efficient resource utilization: through accurate resource monitoring and dynamic allocation, resource idling and waste are avoided, network resource utilization is greatly improved under the premise of guaranteeing business demand, unnecessary bandwidth leasing cost of a data center can be reduced, and an enterprise can fully exert the efficiency of existing network infrastructure; and business flow is smoothly guaranteed.

Description

Technical Field

[0001] This invention relates to the field of network resource scheduling engineering technology, specifically an innovative method for elastic scheduling of network resources. Background Technology

[0002] In today's digital age, the network has become an indispensable infrastructure for the operation of various fields. From large-scale data centers supporting massive Internet services to enterprise internal networks ensuring daily office collaboration, and smart Internet of Things connecting various terminal devices, network traffic is experiencing explosive growth and drastic fluctuations.

[0003] Traditional network resource scheduling strategies are mostly static or based on simple rules and pre-configuration. For example, in data center networks, network administrators estimate traffic demand during peak periods based on experience and pre-allocate fixed bandwidth resources to each server. During off-peak periods, these reserved resources are largely idle, resulting in waste. However, once a sudden traffic surge occurs, such as a surge in traffic caused by a limited-time promotion of a popular application, the lack of a dynamic adjustment mechanism can lead to service disruptions or even outages, severely impacting the user experience.

[0004] Enterprise office networks also face challenges. During working hours, different departments have varying workloads. When the R&D department compiles and tests code, it needs a lot of network bandwidth to transmit data, while the administrative department only needs basic network connectivity for daily office work. The fixed resource allocation model cannot accommodate both, resulting in hindered R&D efficiency or wasted resources.

[0005] The smart IoT scenario is more complex, with numerous sensors and smart devices constantly connecting to and disconnecting from the network. Data transmission needs change rapidly, and traditional scheduling methods are unable to respond in real time. This may cause delays in critical data transmission, affecting the accuracy of equipment control and decision-making, and failing to meet the stringent requirements for network timeliness and reliability, such as real-time monitoring of industrial automation and instant control of intelligent traffic lights.

[0006] In addition, existing scheduling methods often focus only on a single resource dimension, such as bandwidth, while ignoring the coordination of other related resources such as computing and storage. This makes it difficult to achieve the optimal overall network performance, and there is an urgent need for an innovative and flexible scheduling method that can adapt to complex and ever-changing network environments. Summary of the Invention

[0007] (a) Technical problems to be solved

[0008] To address the shortcomings of existing technologies, this invention provides a method for elastic scheduling of network resources, which solves the technical problems mentioned in the background section.

[0009] (II) Technical Solution

[0010] To achieve the above objectives, the present invention provides the following technical solution: a multi-domain collaborative and intelligent management architecture subsystem interaction method, comprising the following steps: Unified interaction standard formulation: defining a common data format covering structured, semi-structured, and unstructured data; establishing a standard communication protocol stack, encapsulating a custom application layer protocol on top of a reliable network protocol, specifying message header, message body structure, and interaction flow; designing a unified interface specification, including interface function definitions and interface calling methods, adapting to different programming languages ​​and development platforms; middleware platform construction: building a data conversion middleware to automatically identify and convert input data formats in real time; developing a communication proxy middleware responsible for underlying communication protocol adaptation, sending and receiving data packets, and reverse parsing; establishing an intelligent management middleware to collect subsystem operating status data in real time, judging system health status through a built-in data analysis model, dynamically adjusting subsystem interaction strategies; data interaction process optimization: subsystems encapsulate, send, receive, and parse data according to a unified interface specification, the intelligent management middleware monitors the entire process, adjusts interaction parameters according to the real-time operating status of the subsystems, periodically archives and stores interaction data, and utilizes distributed storage technology to ensure data reliability.

[0011] Preferably, structured data in the general data format adopts a relational database table structure, semi-structured data is mainly in XML or JSON format, and unstructured data supports common image and audio formats.

[0012] Preferably, the custom application layer protocol of the standard communication protocol stack adopts a request-response or publish-subscribe pattern. The message header contains key metadata such as source subsystem identifier, target subsystem identifier, message type, and timestamp, while the message body carries specific business data.

[0013] Preferably, the data conversion middleware uses template matching and machine learning algorithms to improve conversion accuracy and efficiency, while the intelligent management middleware uses a deep learning-based anomaly detection model to determine the system's health status.

[0014] Preferably, the periodic archiving and storage of interactive data is performed hourly, daily, or at other time intervals set according to system characteristics, using distributed storage technologies such as Ceph and GlusterFS.

[0015] Preferably, the intelligent management middleware prioritizes critical business data interactions and optimizes resource allocation when the system is busy. The critical business is predefined by the system administrator according to actual needs.

[0016] (III) Beneficial Effects

[0017] Compared with the prior art, the present invention provides a method for elastic scheduling of network resources, which has the following beneficial effects:

[0018] Efficient resource utilization: Through precise resource monitoring and dynamic allocation, resource idleness and waste are avoided. While ensuring business needs, the utilization rate of network resources is greatly improved. For example, data centers can reduce unnecessary bandwidth leasing costs, and enterprises can give full play to the efficiency of existing network infrastructure.

[0019] Business continuity assurance: Resources are flexibly allocated based on business priorities and real-time needs to ensure that high-priority businesses (such as real-time financial transactions and remote medical surgery assistance) are not affected by low-priority businesses (such as routine software updates), significantly reducing business latency and packet loss, ensuring business continuity and reliability, and improving user satisfaction.

[0020] Adapting to dynamic changes: The demand perception and adaptive optimization mechanism driven by machine learning algorithms enables the scheduling method to track changes in network traffic and business needs in real time. Whether it is daily fluctuations or sudden peaks, it can respond quickly, flexibly adjust resources, and adapt to complex and ever-changing network environments.

[0021] Resource synergy and efficiency enhancement: Breaking the limitations of traditional single-dimensional resource scheduling, it considers the synergy of related resources such as computing and storage to achieve optimal overall network performance. For example, in big data processing scenarios, bandwidth, computing and storage resources are allocated synchronously to accelerate data processing and improve system operating efficiency.

[0022] Continuous optimization and upgrades: The built-in effect evaluation and feedback optimization mechanism continuously adjusts the strategy parameters based on actual operation data, so that the scheduling method can continuously evolve with network development and business evolution, always maintain high efficiency, reduce manual intervention, and reduce operation and maintenance costs. Detailed Implementation

[0023] The technical solutions of the present invention will be clearly and completely described below with reference to the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0024] Example 1:

[0025] A method for elastic scheduling of network resources includes the following key steps: Resource monitoring and demand perception: Deploy distributed monitoring probes to comprehensively cover key network nodes, including core routers, switches, server network cards, and IoT gateways, etc., to collect multi-dimensional network resource data in real time, covering bandwidth utilization, packet latency, packet loss rate, server CPU load, memory usage, storage I / O rate, as well as IoT device connection status, data transmission frequency, etc. The sampling frequency is dynamically set according to network scale and real-time requirements, ranging from 1 to 10 times per second. Machine learning algorithms are used to analyze the collected data in real time to build a business traffic prediction model, such as predicting the traffic trend of different business types in the future short period (5-10 minutes) based on Long Short-Term Memory (LSTM) networks. Combined with real-time business request information, the dynamic demand of each node and each business for network resources is accurately perceived. Elastic strategy formulation: Based on the results of resource monitoring and demand perception, a multi-level elastic scheduling strategy is designed.First, at the macro level, based on the overall network load, different load ranges are divided, such as light load (resource utilization below 30%), medium load (30%-70%), and heavy load (above 70%). For each range, a dynamic network topology adjustment plan is developed. During light load, redundant links are appropriately reduced to lower energy consumption; during heavy load, backup paths are activated to distribute traffic. At the micro level, for specific service flows, differentiated resource allocation rules are formulated based on their priority (pre-defined by factors such as service importance and real-time requirements, e.g., real-time video conferencing is high priority, file download is low priority) and resource urgency. Dynamic bandwidth allocation technology is adopted, such as token bucket algorithms to allocate tokens according to priority, ensuring the bandwidth needs of high-priority services while preventing low-priority services from excessively preempting resources. Considering resource synergy, a coordinated resource allocation strategy is developed. When a surge in bandwidth demand for a service is detected, its potential demand for computing and storage resources is simultaneously assessed, and idle computing resources are pre-allocated (e.g., enabling idle server virtual machine instances) and storage access paths are optimized to ensure smooth overall service operation. Resource allocation execution involves building an intelligent management and control platform that integrates network configuration management protocols. The system (e.g., NETCONF, SNMP, etc.) receives elastic policy instructions and translates them into specific configuration operations for network devices, servers, and storage systems, achieving automated resource allocation. When network topology adjustments are needed, the intelligent management platform issues link enable / disable instructions to relevant routers and switches, dynamically updates flow table rules through software-defined networking (SDN) technology, and guides traffic to rerouting. During resource allocation adjustments, it sends CPU and memory resource allocation instructions to the server management system and I / O optimization instructions to the storage array, ensuring accurate and efficient resource allocation. For effect evaluation and optimization, a real-time evaluation indicator system is established, including the improvement in resource utilization, the reduction in average service latency, and the success rate of high-priority service assurance. The system continuously tracks the effect of resource allocation, collecting and analyzing indicator data every 5 minutes. Based on the evaluation results, the feedback control system automatically optimizes elastic policy parameters, such as traffic prediction model weights and bandwidth allocation ratio coefficients. If insufficient resource allocation for a certain type of service leads to increased latency, the priority weights or allocation algorithm parameters are adjusted in a timely manner to achieve adaptive evolution of the scheduling method and continuously improve network performance.

[0026] In summary, this invention achieves efficient resource utilization by accurately monitoring and dynamically allocating resources, avoiding idle and wasted resources, and significantly improving network resource utilization while ensuring business needs are met. For example, data centers can reduce unnecessary bandwidth leasing costs, and enterprises can fully leverage the efficiency of their existing network infrastructure.

[0027] Business Smoothness Guarantee: Resources are flexibly allocated based on business priority and real-time demand, ensuring that high-priority services (such as real-time financial transactions and remote medical surgery assistance) are not interfered with by low-priority services (such as routine software updates). This significantly reduces business latency and packet loss, ensuring business smoothness and reliability, improving user satisfaction, and adapting to dynamic changes. A demand-aware and adaptive optimization mechanism driven by machine learning algorithms enables the scheduling method to track changes in network traffic and business demands in real time. Whether it's daily fluctuations or sudden peaks, it can respond quickly and flexibly adjust resources to adapt to complex and ever-changing network environments. Resource Synergy and Efficiency Enhancement: Breaking the limitations of traditional single-dimensional resource scheduling, it considers the synergy of related resources such as computing and storage to achieve optimal overall network performance. For example, in big data processing scenarios, bandwidth, computing, and storage resources are allocated synchronously to accelerate data processing and improve system operating efficiency. Continuous Optimization and Upgrades: A built-in effect evaluation and feedback optimization mechanism continuously adjusts strategy parameters based on actual operating data, allowing the scheduling method to continuously evolve with network development and business evolution, maintaining high efficiency, reducing manual intervention, and lowering operation and maintenance costs.

[0028] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A method for elastic scheduling of network resources, characterized in that, Includes the following steps: Resource monitoring and demand perception: Deploy distributed monitoring probes at key network nodes to collect multi-dimensional network resource data, including bandwidth utilization, packet latency, packet loss rate, server CPU load, memory usage, storage I / O rate, as well as IoT device connection status and data transmission frequency. The sampling frequency is from 1 to 10 times per second. Use machine learning algorithms to build a business traffic prediction model and combine it with real-time business request information to perceive resource demand. Elastic strategy formulation: Based on monitoring and perception results, formulate multi-level elastic scheduling strategies. At the macro level, divide the network into intervals according to the overall network load and formulate topology adjustment schemes. At the micro level, formulate differentiated resource allocation rules for business flows according to priority and urgency of demand, and formulate related resource linkage and allocation strategies. Resource allocation execution: Build an intelligent management and control platform, integrate network configuration management protocols, receive elastic policy instructions and transform them into specific configuration operations for network devices, servers, and storage systems, and use SDN technology to achieve automated resource allocation; Effect evaluation and optimization: Establish a real-time evaluation indicator system, collect and analyze indicator data every 5 minutes, and use the feedback control system to automatically optimize the elastic strategy parameters.

2. The method for elastic scheduling of network resources according to claim 1, characterized in that, The machine learning algorithm is a traffic prediction model based on Long Short-Term Memory (LSTM) networks, used to predict traffic trends for different service types within the next 5-10 minutes.

3. The method for elastic scheduling of network resources according to claim 1, characterized in that, In the aforementioned multi-level elastic scheduling strategy, the network load at the macro level is divided into light load (resource utilization rate below 30%), medium load (30%-70%), and heavy load (above 70%), with each range corresponding to a different topology adjustment scheme.

4. The method for elastic scheduling of network resources according to claim 1, characterized in that, The differentiated resource allocation rules at the micro level employ a token bucket algorithm to allocate tokens according to priority, ensuring the bandwidth requirements of high-priority services.

5. The method for elastic scheduling of network resources according to claim 1, characterized in that, The associated resource linkage allocation strategy simultaneously assesses and allocates computing and storage resources in advance when a surge in bandwidth demand for a certain service is detected.

6. The method for elastic scheduling of network resources according to claim 1, characterized in that, The real-time evaluation indicator system includes the improvement in resource utilization, the reduction in average business latency, and the success rate of high-priority business assurance, which are used to continuously track the effectiveness of resource allocation.