Programmable network resource and heterogeneous computing power adaptive scheduling method and device based on reinforcement learning, electronic equipment and storage medium

Abstract

The invention relates to a programmable network resource and heterogeneous computing power adaptive scheduling method based on reinforcement learning. The method comprises the steps of obtaining a network and computing power state data set, processing a network topology structure through GNN to generate an observation space of reinforcement learning, adopting a reinforcement learning method to generate an optimal scheduling strategy, and adjusting a data flow path and computing power distribution. The invention relates to a programmable network resource and heterogeneous computing power adaptive scheduling device based on reinforcement learning. The programmable network resource and heterogeneous computing power adaptive scheduling device comprises an acquisition module, a first generation module, a second generation module and an adjustment module. The invention further relates to electronic equipment and a storage medium. The invention designs a programmable network resource and heterogeneous computing power adaptive scheduling method and device based on reinforcement learning, electronic equipment and a storage medium. The problems of lack of joint optimization, difficulty in coping with dynamic topology and burst load, difficulty in balancing target conflicts and inconsistency of data aperture and time sequence caused by splitting of a network side and a calculation side in a traditional scheme can be solved.

Description

Technical Field

[0001] This invention relates to the field of joint scheduling technology for communication and computing resources, specifically to an adaptive scheduling method, apparatus, electronic device, and storage medium for programmable network resources and heterogeneous computing power based on reinforcement learning. Background Technology

[0002] For scenarios such as data centers, cloud-edge integration, and 5G / MEC, the end-to-end performance of services is constrained by two types of resources: one is network transmission resources (bandwidth, latency, packet loss, etc.), and the other is computing resources (the availability and queuing status of CPU / GPU / FPGA).

[0003] Traditional solutions often employ domain-specific scheduling: the network side uses traffic engineering / static strategies, while the computing side uses a separate cluster scheduler. This fragmented approach leads to the following problems: Lack of joint optimization: the network side's bypassing of congestion and the computing side's selection of idle nodes are not synchronized, easily resulting in mismatches such as "network congestion but idle computing power" or "network idleness but computing queues"; Difficulty in handling dynamic topologies and sudden loads: node online / offline status, link quality fluctuations, and peak business periods can cause static rules to become invalid, and centralized fixed strategies are lagging in updates and difficult to adapt; Difficulty in balancing conflicting objectives: there is a trade-off between resource utilization, task latency, and energy consumption, and fixed weighted or threshold rules are insensitive to time-varying business; Inconsistent data caliber and time series: multi-source monitoring data is asynchronous and has different dimensions; direct input into the model can lead to perceptual distortion and affect the effectiveness of the strategy.

[0004] Therefore, existing technologies have shortcomings and need to be improved and developed. Summary of the Invention

[0005] The present invention provides a method, apparatus, electronic device and storage medium for adaptive scheduling of programmable network resources and heterogeneous computing power based on reinforcement learning, which is used to solve the following problems caused by the separation of network side and computing side in the domain scheduling of traditional schemes in the prior art: lack of joint optimization, difficulty in dealing with dynamic topology and sudden load, difficulty in balancing target conflicts, and inconsistency between data scope and timing.

[0006] In a first aspect, this invention provides a method for adaptive scheduling of programmable network resources and heterogeneous computing power based on reinforcement learning, comprising:

[0007] Collect status data of network devices and heterogeneous computing nodes to obtain a network and computing power status dataset after preprocessing;

[0008] Based on the network and computing power status dataset, the network topology is processed by GNN to generate an observation space for reinforcement learning;

[0009] Based on the observation space, a reinforcement learning method is used to generate the optimal scheduling strategy;

[0010] Based on the optimal scheduling strategy, adjust the data flow path and computing power allocation.

[0011] Furthermore, the collection of network device and heterogeneous computing node status data, after preprocessing, to obtain a network and computing power status dataset, includes:

[0012] The system collects network data monitored in real time by the SDN controller and P4 switch, and heterogeneous computing node status data provided by the hardware abstraction layer interface. The network data includes at least node load, link bandwidth, and latency, and the heterogeneous computing node status data includes at least the current CPU and GPU utilization. The link bandwidth is updated once per second in high-traffic scenarios, and the node load is updated once per minute in low-load scenarios.

[0013] After performing outlier filtering, data interpolation, and normalization on the network data and the computing power status data in sequence, the network and computing power status dataset that can be used by the graph neural network is obtained.

[0014] Furthermore, the step of processing the network topology using a GNN to generate an observation space for reinforcement learning based on the network and computing power state dataset includes:

[0015] Dynamic topology modeling is performed based on the network and computing power state dataset to generate a network graph G. t =(V t E t ), where V t E represents a set of nodes, including at least the currently active physical network nodes and virtual computing nodes. t This represents the set of links connecting the physical network nodes and the virtual computing nodes. Each edge carries the link state that changes over time, and the link state includes at least available bandwidth, latency, and packet loss rate.

[0016] A graph neural network is used to extract topological features from the network graph to obtain a set of node embedding vectors {h} for reinforcement learning. v,t In the node embedding vector set, each graph node v at time t corresponds to a low-dimensional embedding vector, which is used to capture the topological features and dynamic attributes of node v at time t.

[0017] When a topology change event occurs, a time-aware graph convolution algorithm is used to update the node embedding vector set in real time, where the node state update formula is:

[0018] ;

[0019] in Represents a node At any moment The set of neighboring nodes; For the first The learnable weight matrix of the layer; For activation functions; It is a time decay function; Neighboring nodes Recent active timestamp;

[0020] The observation space for reinforcement learning is generated based on the updated set of node embedding vectors.

[0021] Furthermore, the step of generating an optimal scheduling policy based on the observation space using a reinforcement learning method includes:

[0022] Based on the observation space, extract from the observation space , , , The current value, where Represents network bandwidth utilization; Represents the utilization rate of computing resources; This represents the time from the start to the completion of the task; Represents node energy consumption;

[0023] Create a multi-objective reward function, where the function is:

[0024] ;

[0025] in, , , , For hyperparameters, where and The value is 0.5. The value is 0.1. The value is 50. , , , These are weighting coefficients that are dynamically adjusted over time or according to the state.

[0026] The multi-objective reward function is imported into the deep Q-network algorithm to generate the objective Q-value representing the expected total reward of the current action a in state s. The calculation formula is:

[0027] ;

[0028] in, This represents the reward value calculated by the multi-objective reward function at the current moment; The discount factor, representing the balance between immediate and future rewards, has a value of 0.95. The Q value represents the action a' to be chosen in the next state s'; This represents the target network parameters that are periodically synchronized from the main network to ensure stability during the training process;

[0029] Import the target Q-value into the reinforcement learning policy In the process, the optimal scheduling strategy is obtained. The reinforcement learning strategy formula is:

[0030] ;

[0031] in, This is the set of all scheduled actions.

[0032] Furthermore, adjusting the data flow path and computing power allocation according to the optimal scheduling strategy includes:

[0033] The SDN controller will use the optimal scheduling strategy This is converted into forwarding rules that can be used by the corresponding P4 switches so that the P4 switches can update their network control actions using these forwarding rules. and based on the network control actions Adjust the data flow path;

[0034] The abstraction layer interface is based on the optimal scheduling strategy. Generate computing power control actions And control actions based on the computing power. Distribute computing tasks to heterogeneous computing nodes.

[0035] A second aspect of this invention provides a programmable network resource and heterogeneous computing power adaptive scheduling device based on reinforcement learning, comprising:

[0036] The acquisition module is used to collect status data of network devices and heterogeneous computing nodes, so as to obtain a network and computing power status dataset after preprocessing.

[0037] The first generation module is used to generate an observation space for reinforcement learning by processing the network topology structure through a GNN based on the network and computing power status dataset.

[0038] The second generation module is used to generate an optimal scheduling strategy based on the observation space using a reinforcement learning method.

[0039] The adjustment module is used to adjust the data flow path and computing power allocation according to the optimal scheduling strategy.

[0040] In a third aspect, the present invention provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program, when executed by the processor, implements the aforementioned reinforcement learning-based adaptive scheduling method for programmable network resources and heterogeneous computing power.

[0041] In a fourth aspect, the present invention provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the aforementioned reinforcement learning-based adaptive scheduling method for programmable network resources and heterogeneous computing power.

[0042] Beneficial effects:

[0043] As can be seen from the above technical solutions, the present invention provides a method, apparatus, electronic device, and storage medium for adaptive scheduling of programmable network resources and heterogeneous computing power based on reinforcement learning, which has the following beneficial effects:

[0044] 1. Achieve end-to-end joint optimization closed loop: Under a unified observation space, the joint actions of network path and computing power allocation are generated simultaneously and executed via P4 and HAL respectively, forming a closed loop of data acquisition—modeling—decision-execution—re-acquisition. This mechanism helps reduce cross-domain mismatches such as "network is available but computing is queuing" or "computing is idle but links are congested," improving overall throughput and reducing end-to-end completion time and tail latency.

[0045] 2. Enhanced adaptability to time-varying environments: Through dynamic topology modeling and time-aware graph convolutional networks, node embeddings are incrementally updated when events such as node additions / deletions, link quality fluctuations, and sudden service disruptions occur. Based on the new node embeddings, a deep Q-network is used to recalculate the Q-value and generate a new scheduling strategy. This combination enables the strategy to remain effective as the environment changes, demonstrating excellent resilience to latency jitter and sudden congestion.

[0046] 3. Achieved multi-objective adaptive trade-offs: Network utilization, computing power utilization, task completion time and energy consumption are incorporated into the non-linear reward, and the optimization focus at different stages is adjusted through dynamic weighting to avoid the side effects caused by over-optimization of a single indicator. This helps to improve resource utilization efficiency and control energy consumption levels while meeting latency and SLA requirements.

[0047] 4. Improved stability of adaptive scheduling: The GNN is provided with a time-aligned network and computing power state dataset by adopting the sequence of "time alignment - unit unification - anomaly handling - missing data filling - smoothing - normalization - feature construction"; the readout layer embeds and aggregates all network nodes into a fixed-dimensional observation vector, which preserves the topological context and controls the input size, improves the stability of training and online inference, and supports the expansion deployment on larger-scale topologies.

[0048] It should be understood that all combinations of the foregoing concepts and the additional concepts described in more detail below can be considered part of the inventive subject matter of this disclosure, provided that such concepts do not contradict each other.

[0049] The foregoing and other aspects, embodiments, and features of the teachings of the present invention will be more fully understood from the following description in conjunction with the accompanying drawings. Other additional aspects of the invention, such as features and / or beneficial effects of exemplary embodiments, will become apparent from the following description or may be learned through practice of specific embodiments according to the teachings of the present invention. Attached Figure Description

[0050] The accompanying drawings are not drawn to scale. In the drawings, each identical or nearly identical component shown in the various figures may be denoted by the same reference numeral. For clarity, not every component is labeled in each figure. Embodiments of various aspects of the invention will now be described by way of example and with reference to the accompanying drawings, wherein:

[0051] Figure 1 This is a flowchart illustrating an adaptive scheduling method for programmable network resources and heterogeneous computing power based on reinforcement learning, as described in this application.

[0052] Figure 2 This is a flowchart of step S102 of an adaptive scheduling method for programmable network resources and heterogeneous computing power based on reinforcement learning in an embodiment of this application.

[0053] Figure 3 This is a flowchart of step S104 of an adaptive scheduling method for programmable network resources and heterogeneous computing power based on reinforcement learning in an embodiment of this application.

[0054] Figure 4 This is a flowchart of step S106 of an adaptive scheduling method for programmable network resources and heterogeneous computing power based on reinforcement learning in an embodiment of this application.

[0055] Figure 5 This is a flowchart of step S108 of an adaptive scheduling method for programmable network resources and heterogeneous computing power based on reinforcement learning in an embodiment of this application. Detailed Implementation

[0056] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. All other embodiments obtained by those skilled in the art based on the described embodiments of the present invention without creative effort are within the scope of protection of the present invention. Unless otherwise defined, the technical or scientific terms used herein should have the ordinary meaning understood by those skilled in the art to which this invention pertains.

[0057] The terms "first," "second," and similar words used in the specification and claims of this patent application do not indicate any order, quantity, or importance, but are merely used to distinguish different components. Similarly, unless the context clearly indicates otherwise, the singular forms of "an," "a," or "the," etc., do not indicate a quantity limitation, but rather indicate the presence of at least one. Terms such as "comprising" or "including" indicate that the element or object preceding "comprising" encompasses the features, integrals, steps, operations, elements, and / or components listed following "comprising" or "including," and do not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components, and / or sets thereof. Terms such as "upper," "lower," "left," and "right" are used only to indicate relative positional relationships; these relative positional relationships may change accordingly when the absolute position of the described object changes.

[0058] In existing technologies, traditional solutions mostly employ domain-specific scheduling: the network side uses traffic engineering / static strategies, while the computing side uses a separate cluster scheduler. This fragmented approach leads to the following problems: Lack of joint optimization: the network side's bypassing of congestion and the computing side's selection of idle nodes are not synchronized, easily resulting in mismatches such as "network congestion but idle computing power" or "network idleness but computing queues"; Difficulty in handling dynamic topologies and sudden loads: node online / offline status, link quality fluctuations, and peak business periods can cause static rules to fail, and centralized fixed strategies are lagging in updates and difficult to adapt; Difficulty in balancing conflicting objectives: there is a trade-off between resource utilization, task latency, and energy consumption, and fixed weighted or threshold rules are insensitive to time-varying business; Inconsistent data caliber and time series: multi-source monitoring data is asynchronous and has different dimensions; direct input into the model can lead to perceptual distortion and affect the effectiveness of the strategy.

[0059] In view of this, refer to Figure 1 This invention provides a reinforcement learning-based adaptive scheduling method for programmable network resources and heterogeneous computing power, comprising:

[0060] Step S102: Collect network device and heterogeneous computing node status data to obtain a network and computing power status dataset after preprocessing.

[0061] Step S104: Based on the network and computing power state dataset, process the network topology using GNN to generate an observation space for reinforcement learning.

[0062] Step S106: Based on the observation space, a reinforcement learning method is used to generate the optimal scheduling strategy.

[0063] Step S108: Adjust the data flow path and computing power allocation according to the optimal scheduling strategy.

[0064] This invention proposes an adaptive scheduling method for programmable network resources and heterogeneous computing power based on reinforcement learning. This method deeply integrates deep reinforcement learning with programmable network technology, extracting state features through dynamic graph topology modeling and graph neural networks, and jointly optimizing network and computing resource scheduling in an agent-like manner. Under the centralized management of the SDN controller, a reinforcement learning agent is introduced to perceive the global network-computing power state and issues fine-grained control through the P4 programmable switch and Hardware Abstraction Layer (HAL) interface, achieving automatic adjustment of data transmission paths and task computation allocation. This intelligent scheduling can make decisions in milliseconds based on real-time status, significantly improving network bandwidth and computing resource utilization efficiency, shortening task completion time, and reducing energy consumption. It is suitable for scenarios requiring network computing collaboration, such as 5G base station edge computing and cloud data centers, ensuring service quality while achieving intensive and efficient resource utilization when dealing with sudden traffic surges or high-concurrency tasks.

[0065] In some embodiments, reference is made to Figure 2 The system collects status data from network devices and heterogeneous computing nodes to obtain a network and computing power status dataset after preprocessing, including:

[0066] Step S1021: Collect network data monitored in real time by the SDN controller and P4 switch, and heterogeneous computing node status data provided by the hardware abstraction layer interface. The network data includes at least node load, link bandwidth, and latency. The heterogeneous computing node status data includes at least the current CPU and GPU utilization. The link bandwidth is updated once per second in high-traffic scenarios, and the node load is updated once per minute in low-load scenarios.

[0067] Step S1022: After performing outlier filtering, data interpolation, and normalization on the network data and heterogeneous computing node status data in sequence, a network and computing power status dataset that can be used by the graph neural network is obtained.

[0068] The data collection includes multiple dimensions such as network node load, link performance, latency, and heterogeneous computing resource status. Node load covers CPU utilization and GPU utilization; link status includes indicators such as bandwidth utilization, end-to-end latency, and packet loss rate; heterogeneous computing resource status refers to the available resources of each computing node (physical or virtual), such as remaining CPU resources, memory resources, and GPU resources.

[0069] Global visibility is achieved through multi-source collaborative data collection. The SDN controller periodically pulls or subscribes to link statistics from the switches to obtain network topology and link bandwidth utilization. Programmable P4 switches can monitor changes in the topology, such as new nodes connecting to a switch port or link failure recovery, and notify the controller of these events, thus gaining awareness of dynamic changes in the network topology. Simultaneously, through the HAL interface, the computing resource usage status is collected from various heterogeneous computing nodes, such as the current CPU and GPU utilization and task queue length of each server. The aggregated multi-source data forms a dataset containing both network and computing power status.

[0070] Because network and load conditions change at different rates, the system employs an adaptive sampling frequency dynamic adjustment strategy. In high-traffic or rapidly changing scenarios, the sampling frequency is increased to obtain the latest status; for example, link bandwidth utilization can be updated at a second-level frequency, i.e., once per second, to capture sudden traffic changes. Conversely, in low-load or stable scenarios, the sampling frequency can be reduced to decrease overhead; for example, node CPU and GPU load can be updated once per minute.

[0071] The collected multi-source data undergoes preprocessing before entering the next model step to ensure quality and consistency. Preprocessing includes outlier filtering to remove obviously abnormal or erroneous data points, data interpolation to handle missing values, and normalization. Simultaneously, a sliding window smoothing process is applied to historical data to mitigate the impact of momentary fluctuations on decision-making. Through these mechanisms, the system constructs a high-quality real-time dataset covering the entire network-computing power dimension, providing reliable input for subsequent dynamic topology modeling and policy optimization.

[0072] In some embodiments, reference is made to Figure 3 Based on the network and computing power state dataset, the network topology is processed using GNN to generate an observation space for reinforcement learning, including:

[0073] Step S1041: Perform dynamic topology modeling based on the network and computing power state dataset to generate the network graph G. t =(V t E t ), where V t E represents a set of nodes, including at least the currently active physical network nodes and virtual computing nodes.t It represents the set of links connecting physical network nodes and virtual computing nodes. Each edge carries the link state that changes over time. The link state includes at least available bandwidth, latency, and packet loss rate.

[0074] Step S1042: Use a graph neural network to extract topological features from the network graph to obtain a set of node embedding vectors {h} for reinforcement learning. v,t In the node embedding vector set, each graph node v at time t corresponds to a low-dimensional embedding vector, which is used to capture the topological features and dynamic attributes of node v at time t.

[0075] Step S1043: When a topology change event occurs, a time-aware graph convolution algorithm is used to update the node embedding vector set in real time, where the node state update formula is:

[0076] ;

[0077] in Represents a node At any moment The set of neighboring nodes; For the first The learnable weight matrix of the layer; For activation functions; It is a time decay function; Neighboring nodes Recent active timestamp;

[0078] Step S1044: Generate a state observation space that provides topology awareness for the reinforcement learning policy based on the updated set of node embedding vectors.

[0079] When modeling dynamic network topology, a graph model is used to represent the structure and state of the network and computing resources, and a network graph G that evolves over time is defined. t =(V t E t The addition and removal of nodes are dynamic; for example, terminals or microservices in edge computing scenarios may dynamically go online or offline. This model supports various topologies, including star, ring, and tree topologies, and can represent their time-varying characteristics, dynamically reflecting changes in network structure and resource status to meet the needs of highly dynamic environments. The monitoring system detects topology change events, such as the addition of new nodes or changes in link status, and updates the network graph in a timely manner.

[0080] A graph convolutional neural network (GNN) is used to extract features from the aforementioned network graph, yielding a state vector representation for reinforcement learning. The GNN's computation process iteratively aggregates neighbor information through multiple layers of time-aware graph convolutions. An exponential decay function is used, with hyperparameters controlling the rate of historical information decay: the further back in time a neighbor node's most recent activity occurred, the less influence its features have on the current node. Through this spatiotemporal neighbor feature aggregation, the GNN achieves joint encoding of the topology and node states, making recent topological changes have a greater impact on the embeddings. Whenever a topological event occurs, such as node additions / deletions or sudden changes in links, the node embeddings can be updated in real time, thus maintaining an up-to-date characterization of the network state.

[0081] Through multi-layer graph convolution iterations, typically 2-3 layers are selected to capture broad neighborhood information while avoiding over-smoothing, resulting in a set of embedded representations for each node in the entire network graph. These embedded vectors can be synthesized into a representation of the entire network state for reinforcement learning. For example, all node embeddings can be concatenated or a global state vector can be obtained through a readout function. Alternatively, the graph embeddings can be directly fed into the value network or policy network of reinforcement learning for evaluation. Node embeddings serve directly as the state input to the reinforcement learning agent, endowing the agent with topology awareness, enabling it to detect congested parts of the current network and idle parts of computational nodes.

[0082] In some embodiments, reference is made to Figure 4 Based on the observation space, reinforcement learning methods are used to generate optimal scheduling policies, including:

[0083] Step S1061: Based on the observation space, extract from the observation space , , , The current value, where Represents network bandwidth utilization; Represents the utilization rate of computing resources; This represents the time from the start to the completion of the task; This represents the energy consumption of a node.

[0084] Step S1062: Create a multi-objective reward function, wherein the function is:

[0085] ;

[0086] in, , , , These are manually set hyperparameters, where and The value is 0.5. The value is 0.1. The value is 50. , , , These are weighting coefficients that are dynamically adjusted over time or according to the state.

[0087] Step S1063: Import the multi-objective reward function into the deep Q-network algorithm to generate the objective Q-value representing the expected total reward of the current action a in state s. The calculation formula is:

[0088] ;

[0089] in, This represents the reward value calculated by the multi-objective reward function at the current moment; The discount factor, representing the balance between immediate and future rewards, has a value of 0.95. The Q value represents the action a' to be chosen in the next state s'; This represents the target network parameters that are periodically synchronized from the main network to ensure a stable training process.

[0090] Step S1064: Import the target Q-value into the reinforcement learning policy In the process, the optimal scheduling strategy is obtained. The reinforcement learning strategy formula is:

[0091] ;

[0092] in, This is the set of all scheduled actions.

[0093] Taking into account network resource utilization, computing resource utilization, task completion time, and energy consumption, a nonlinear reward function is designed to encourage agents to fully utilize available network bandwidth resources and increase link throughput. When bandwidth usage approaches saturation, the additional reward gradually decreases to avoid congestion caused by blindly pursuing high utilization. For example, a convex function of bandwidth utilization can be used so that the reward increase slows down and stagnates as utilization increases from medium to high. It can adjust based on network traffic conditions: when the network is busy, it increases the weight, making the agent pay more attention to alleviating network bottlenecks.

[0094] Encourage the full utilization of heterogeneous computing power to increase server load. When computing resources are close to full capacity, additional rewards tend to be moderated to prevent agents from over-exploiting computing power, which could lead to increased task queuing or system instability. It can be increased during peak computing demand periods, allowing agents to prioritize making fuller use of computing resources.

[0095] The agent is forced to focus on task latency metrics, with higher rewards for faster task completion, especially for latency-sensitive tasks that need to be executed as quickly as possible. Therefore, the reward function is set to have high sensitivity to task completion time, prompting the agent to make shortening task time one of its core objectives. In scenarios with high real-time requirements, such as critical tasks or service level agreements with strict latency requirements, a larger value is assigned to further highlight the low latency goal.

[0096] While ensuring performance, an energy consumption index is introduced to encourage energy-saving scheduling. To avoid the agent from exhibiting extreme behaviors, such as not performing tasks to save energy, due to simple linear penalties for energy consumption, a negative reward that increases slowly with energy consumption is designed to drive the agent to choose the lower-energy-consumption solution overall. It can be used to increase the weight of energy consumption items during predetermined energy-saving priority periods, such as at night or when there is a power shortage, thereby guiding the agent to favor energy-saving strategies during these periods.

[0097] Through the above design, the reward function can dynamically balance multi-objective conflicts: under different system states and policies, the agent will receive different comprehensive rewards, thus learning to balance the relationship between network throughput, computational efficiency, task latency, and energy consumption. The dynamic weight mechanism allows system operators to adjust the optimization focus according to policy preferences or external conditions, achieving adaptive multi-objective optimization.

[0098] The agent employs the Deep Q-Network (DQN) algorithm from deep reinforcement learning to learn the optimal scheduling policy under the aforementioned rewards. DQN is a value function method that estimates the value Q(s,a) of each action in a given state by training an approximate function. In this embodiment, the Q-Network takes the state representation extracted by the GNN as input and outputs the Q-value of each possible scheduling action. During training, the agent repeatedly interacts with the environment (network, computing system): selecting actions according to the current policy, issuing the corresponding schedule for execution, observing the execution results to obtain new states and immediate rewards, and then using this experience to update the Q-Network parameters. The discount factor balances the importance of immediate rewards and future long-term rewards. A value of 0.95 is chosen to ensure that the agent prioritizes both immediate gains and long-term performance. To improve training stability, DQN uses a target network to provide... The target network parameters are periodically synchronized from the main network or softly updated to avoid estimation oscillations during training.

[0099] An experience replay buffer is introduced to store the agent's interaction experiences, and a small batch of samples is randomly sampled in each training step to update the network, breaking sequence correlation and improving learning efficiency. Simultaneously, an ε-greedy strategy is employed for action selection: the agent randomly explores non-optimal actions with probability ε, and selects the action with the highest current Q-value with probability (1-ε). In the early stages of training, ε is set to a higher value to encourage exploration, for example, 1.0. Later, ε is gradually reduced to converge to a deterministic policy, for example, linearly or exponentially decreasing to 0.1 or even 0.01 according to the training progress, to ensure later policy convergence. After sufficient training iterations, DQN can approximate the optimal Q-function, ensuring that the action selected in any state maximizes the long-term cumulative reward. At this point, we obtain an approximate optimal policy π. The execution rule of policy π is: in state s, select action a with the highest Q(s,a) value. In this way, the agent learns how to schedule actions to achieve the best performance in multi-objective integration, given the current global network and computing power state.

[0100] Introducing a graph neural network to reduce the dimensionality of the state representation alleviates the input dimensionality of the Q-network, making DQN easier to train and converge. Simultaneously, the multi-objective reward design and DQN optimization enable the policy to gradually approach the globally optimal resource scheduling scheme, achieving a dynamic trade-off between performance, latency, and energy consumption. The trained agent can adaptively adjust scheduling in changing network environments, always operating towards the direction of optimal overall benefit.

[0101] In some embodiments, reference is made to Figure 5 Based on the optimal scheduling strategy, adjust the data flow path and computing power allocation, including:

[0102] Step S1081: The SDN controller will use the optimal scheduling strategy. This is converted into forwarding rules that can be used by the corresponding P4 switches, so that the P4 switches can update the forwarding rules to generate network control actions. and based on network control actions Adjust the data flow path.

[0103] Step S1082: The abstraction layer interface is based on the optimal scheduling strategy. Generate computing power control actions And control actions based on computing power Distribute computing tasks to heterogeneous computing nodes.

[0104] Each decision made by the agent produces a joint action, which consists of two parts:

[0105] Network control actions This is used to adjust network path selection for data flows. The agent selects a suitable combination of forwarding paths based on the current network state to optimize traffic distribution or avoid congestion. During execution, the SDN controller translates this decision into corresponding P4 switch forwarding rule updates. For example, when the bandwidth utilization of a link is detected to be close to saturation, the agent may choose to forward subsequent traffic via an alternative path. The SDN controller then issues a new P4 flow table entry, enabling real-time rerouting of traffic to the alternative link, thereby alleviating pressure on hotspot links. Similarly, for service traffic with high real-time requirements, the agent can decide to select the path with the lowest latency in the network to improve service response speed. Leveraging the high-speed forwarding and flexible matching capabilities of the P4 programmable data plane, combined with the centralized scheduling of the SDN controller, path adjustment policies can be issued and take effect within milliseconds, achieving near real-time, fine-grained control of network traffic.

[0106] Computing power control actions The HAL (Hardware Allocation Array) is used to adjust the distribution of tasks among heterogeneous computing nodes. Based on the load and task requirements of each node, the agent decides which node to deploy a new task to, or whether to migrate existing tasks to rebalance the load. Upon receiving the decision, the computing scheduler distributes the task to the designated target node for execution via the HAL interface. For example, for computationally intensive tasks that are not very sensitive to latency, the agent might choose a node with a GPU cluster to utilize the parallel computing capabilities of GPUs to shorten execution time; while for tasks with extremely low latency requirements, such as real-time signal processing, the agent might assign them to edge nodes deployed with FPGA accelerators to leverage the hardware parallelism advantages of FPGAs. If a node is overloaded, the agent can also decide to transfer some tasks to other idle nodes, thereby reducing queuing latency. By providing a unified abstraction of the underlying hardware details through the HAL, the heterogeneity of different computing units is shielded, enabling the computing scheduler to uniformly manage various computing resources and decoupling task scheduling from the hardware platform.

[0107] The aforementioned network and computing power actions are generated collaboratively and executed synchronously. The agent simultaneously determines the direction of traffic transmission and the computing method based on the global state, thereby jointly optimizing network transmission and computing allocation at the source. At the execution level, the SDN controller and computing power scheduler work independently yet in cooperation: path adjustment on the network side reduces communication latency and congestion, while task allocation on the computing side improves processing efficiency and load balancing; together, they achieve end-to-end performance improvement. For example, in a data center scenario, a policy execution might manifest as: distributing a newly submitted batch of computing tasks to multiple servers with lower loads (denoted as a computing power action), while simultaneously adjusting the traffic routing between these servers and the data source to avoid passing through already heavily loaded switches (denoted as a network action). Through real-time decision-making of reinforcement learning policies and rapid deployment via the P4 / HAL interface, collaborative optimization of network paths and computing power allocation is achieved, ensuring the system can respond promptly to state changes. After each round of action execution, the system re-enters the state data acquisition phase, and the new network and computing power states are fed back to the agent for the next decision, forming an adaptive closed loop.

[0108] Another embodiment of the present invention provides a programmable network resource and heterogeneous computing power adaptive scheduling device based on reinforcement learning, comprising:

[0109] The acquisition module is used to collect status data from network devices and heterogeneous computing nodes to obtain a network and computing power status dataset after preprocessing.

[0110] The first generation module is used to generate an observation space for reinforcement learning by processing the network topology through a GNN based on the network and computing power state dataset.

[0111] The second generation module is used to generate the optimal scheduling strategy based on the observation space and employing reinforcement learning methods.

[0112] The adjustment module is used to adjust the data flow path and computing power allocation according to the optimal scheduling strategy.

[0113] Another embodiment of the present invention provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the computer program is executed by the processor, it implements a reinforcement learning-based adaptive scheduling method for programmable network resources and heterogeneous computing power.

[0114] The processor can be a Central Processing Unit (CPU), or other general-purpose processors, Digital Signal Processors (DSPs), Application Specific Integrated Circuits (ASICs), Field-Programmable Gate Arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor can be a microprocessor or any conventional processor. The processor is the control center of the gateway's data processing device, connecting various parts of the gateway's data processing device through various interfaces and lines.

[0115] Another embodiment of the present invention provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements a reinforcement learning-based adaptive scheduling method for programmable network resources and heterogeneous computing power.

[0116] The memory may include a program storage area and a data storage area. The program storage area may store the operating system and applications required for at least one function; the data storage area may store data created by the processor, etc. Furthermore, the memory is preferably, but not limited to, high-speed random access memory (RAM). For example, it may also be non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, the memory may also optionally include memory remotely located relative to the processor, which can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks (LANs), mobile communication networks, and combinations thereof.

[0117] Those skilled in the art will understand that implementing all or part of the processes in the above embodiments can be accomplished by a computer program instructing related hardware, and can be stored in a computer-readable storage medium. When executed, the program can include the processes of the embodiments of the above methods. The storage medium can be a magnetic disk, optical disk, read-only memory (ROM), random access memory (RAM), flash memory, hard disk drive (HDD), or solid-state drive (SSD), etc.; the storage medium can also include combinations of the above types of memory.

[0118] In summary, this invention provides a method, apparatus, electronic device, and storage medium for adaptive scheduling of programmable network resources and heterogeneous computing power based on reinforcement learning, which has the following beneficial effects: It achieves an end-to-end joint optimization closed loop: Under a unified observation space, joint actions for network path and computing power allocation are simultaneously generated and executed via P4 and HAL respectively, forming a closed loop of acquisition—modeling—decision—execution—re-acquisition. This mechanism helps reduce cross-domain mismatches such as "network accessibility but computation queuing" or "computation idleness but link congestion," improving overall throughput and reducing end-to-end completion time and tail latency. It enhances adaptability to time-varying environments: Through dynamic topology modeling and time-aware graph convolutional networks, node embeddings are incrementally updated when events such as node addition / deletion, link quality fluctuations, and service bursts occur; based on the new node embeddings, a deep Q-network is used to recalculate the Q-value, generating a new scheduling strategy. This combination enables the strategy to remain effective with environmental changes and has excellent resilience to latency jitter and sudden congestion. A multi-objective adaptive trade-off is achieved: network utilization, computing power utilization, task completion time, and energy consumption are incorporated into the non-linear reward, and the optimization focus at different stages is adjusted through dynamic weights. This avoids the side effects caused by over-optimization of a single indicator, helping to improve resource utilization efficiency and control energy consumption levels while meeting latency and SLA requirements. The stability of adaptive scheduling is improved: a sequence of "time alignment—unit unification—anomaly handling—missing word imputation—smoothing—normalization—feature construction" is adopted to provide the GNN with a time-aligned network and computing power state dataset; the readout layer embeds and aggregates all network nodes into a fixed-dimensional observation vector, preserving the topological context while controlling the input size, improving the stability of training and online inference, and supporting expanded deployment on larger-scale topologies.

[0119] While the present invention has been disclosed above with reference to preferred embodiments, it is not intended to limit the invention. Those skilled in the art can make various modifications and refinements without departing from the spirit and scope of the invention. Therefore, the scope of protection of the present invention shall be determined by the claims.

Claims

1. A method for adaptive scheduling of programmable network resources and heterogeneous computing power based on reinforcement learning, characterized in that, include: Collect status data of network devices and heterogeneous computing nodes to obtain a network and computing power status dataset after preprocessing; Based on the network and computing power status dataset, the network topology is processed by GNN to generate an observation space for reinforcement learning; Based on the observation space, a reinforcement learning method is used to generate the optimal scheduling strategy; Based on the optimal scheduling strategy, adjust the data flow path and computing power allocation.

2. The adaptive scheduling method for programmable network resources and heterogeneous computing power based on reinforcement learning according to claim 1, characterized in that, The collected network device and heterogeneous computing node status data, after preprocessing, yields a network and computing power status dataset, including: The system collects network data monitored in real time by the SDN controller and P4 switch, and heterogeneous computing node status data provided by the hardware abstraction layer interface. The network data includes at least node load, link bandwidth, and latency, and the heterogeneous computing node status data includes at least the current CPU and GPU utilization. The link bandwidth is updated once per second in high-traffic scenarios, and the node load is updated once per minute in low-load scenarios. After performing outlier filtering, data interpolation, and normalization on the network data and the computing power status data in sequence, the network and computing power status dataset that can be used by the graph neural network is obtained.

3. The adaptive scheduling method for programmable network resources and heterogeneous computing power based on reinforcement learning according to claim 1, characterized in that, The step of processing the network topology using a GNN to generate an observation space for reinforcement learning based on the network and computing power state dataset includes: Dynamic topology modeling is performed based on the network and computing power state dataset to generate a network graph G. t =(V t E t ), where V t E represents a set of nodes, including at least the currently active physical network nodes and virtual computing nodes. t This represents the set of links connecting the physical network nodes and the virtual computing nodes. Each edge carries the link state that changes over time, and the link state includes at least available bandwidth, latency, and packet loss rate. A graph neural network is used to extract topological features from the network graph to obtain a set of node embedding vectors {h} for reinforcement learning. v,t In the node embedding vector set, each graph node v at time t corresponds to a low-dimensional embedding vector, which is used to capture the topological features and dynamic attributes of node v at time t. When a topology change event occurs, a time-aware graph convolution algorithm is used to update the node embedding vector set in real time, where the node state update formula is: ； in Represents a node At any moment The set of neighboring nodes; For the first The learnable weight matrix of the layer; For activation functions; It is a time decay function; Neighboring nodes Recent active timestamp; The observation space for reinforcement learning is generated based on the updated set of node embedding vectors.

4. The adaptive scheduling method for programmable network resources and heterogeneous computing power based on reinforcement learning according to claim 2, characterized in that, The step of generating an optimal scheduling strategy based on the observation space using a reinforcement learning method includes: Based on the observation space, extract from the observation space , , , The current value, where Represents network bandwidth utilization; Represents the utilization rate of computing resources; This represents the time from the start to the completion of the task; Represents node energy consumption; Create a multi-objective reward function, where the function is: ； in, , , , For hyperparameters, where and The value is 0.

5. The value is 0.

1. The value is 50. , , , These are weighting coefficients that are dynamically adjusted over time or according to the state. The multi-objective reward function is imported into the deep Q-network algorithm to generate the objective Q-value representing the expected total reward of the current action a in state s. The calculation formula is: ； in, This represents the reward value calculated by the multi-objective reward function at the current moment; The discount factor, representing the balance between immediate and future rewards, has a value of 0.

95. The Q value represents the action a' to be chosen in the next state s'; This represents the target network parameters that are periodically synchronized from the main network to ensure stability during the training process; Import the target Q-value into the reinforcement learning policy In the process, the optimal scheduling strategy is obtained. The reinforcement learning strategy formula is: ； in, This is the set of all scheduled actions.

5. The adaptive scheduling method for programmable network resources and heterogeneous computing power based on reinforcement learning according to claim 4, characterized in that, The step of adjusting the data flow path and computing power allocation according to the optimal scheduling strategy includes: The SDN controller will use the optimal scheduling strategy This is converted into forwarding rules that can be used by the corresponding P4 switches so that the P4 switches can update their network control actions using these forwarding rules. and based on the network control actions Adjust the data flow path; The abstraction layer interface is based on the optimal scheduling strategy. Generate computing power control actions And control actions based on the computing power. Distribute computing tasks to heterogeneous computing nodes.

6. A programmable network resource and heterogeneous computing power adaptive scheduling device based on reinforcement learning, characterized in that, include: The acquisition module is used to collect status data of network devices and heterogeneous computing nodes, so as to obtain a network and computing power status dataset after preprocessing. The first generation module is used to generate an observation space for reinforcement learning by processing the network topology structure through a GNN based on the network and computing power status dataset. The second generation module is used to generate an optimal scheduling strategy based on the observation space using a reinforcement learning method. The adjustment module is used to adjust the data flow path and computing power allocation according to the optimal scheduling strategy.

7. An electronic device, characterized in that, It includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the computer program is executed by the processor, it implements the adaptive scheduling method for programmable network resources and heterogeneous computing power based on reinforcement learning as described in any one of claims 1 to 5.

8. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the adaptive scheduling method for programmable network resources and heterogeneous computing power based on reinforcement learning as described in any one of claims 1 to 5.

Images