Network dynamic flow scheduling method, device, equipment and product

Abstract

The invention discloses a network dynamic flow scheduling method, device, equipment and product, and relates to the technical field of network scheduling, and the method comprises the steps: monitoring the change of target parameters of the equipment in real time through obtaining the state data and historical data of each piece of equipment in real time, and analyzing the behavior and flow trend of the equipment. The method comprises the following steps: calculating a Q value of each device through a Q learning algorithm to obtain an optimal Q value of each device under a current task, calculating an optimal path and a time slot allocation result of each device under the current task according to the optimal Q value, and scheduling a network dynamic flow according to the optimal path and the time slot allocation result to ensure that the path is shortest and the resource allocation is most reasonable, thereby improving the network transmission efficiency. And the risks of time delay and network congestion are reduced. The dynamic flow scheduling mechanism can flexibly adjust the flow load of each path in the network according to the real-time data and the historical trend, thereby ensuring the efficient utilization of network resources, and improving the certainty of equipment in the dynamic network.

Description

Technical Field

[0001] This application relates to the field of network scheduling technology, and in particular to network dynamic flow scheduling methods, apparatus, equipment and products. Background Technology

[0002] With the rapid development of network devices and applications, network traffic scheduling has become increasingly complex. Currently, most flow scheduling methods for deterministic networks are based on static network parameters, which differ from the dynamic changes in actual networks and cannot adapt to changes in network parameters in real-world environments. For example, the port rates of devices may change, or devices may suddenly go offline, rendering the previously calculated scheduling results inapplicable in the current network environment and failing to guarantee network determinism.

[0003] Therefore, how to improve the determinism of network devices in a dynamic network environment is a problem that urgently needs to be solved. Summary of the Invention

[0004] The main purpose of this application is to provide a network dynamic flow scheduling method, apparatus, device and product, which aims to solve the technical problem of low determinism of network devices in dynamic network environments.

[0005] To achieve the above objectives, this application proposes a network dynamic flow scheduling method, the method comprising:

[0006] The status data and historical data of each device are acquired in real time. The status data is used to monitor changes in the target parameters of the devices in real time, and the historical data is used to analyze device behavior and traffic trends.

[0007] Based on the state data and historical data, the Q value of each device is calculated using the Q-learning algorithm to obtain the optimal Q value of each device under the current task.

[0008] Calculate the optimal path and time slot allocation results for each device under the current task based on the optimal Q value;

[0009] The network dynamic flow is scheduled based on the optimal path and time slot allocation results.

[0010] In one embodiment, the step of calculating the Q-value of each device using a Q-learning algorithm based on the state data and historical data to obtain the optimal Q-value for each device under the current task includes:

[0011] Based on the state data and historical data, the state space and action space of Q-learning are obtained, wherein the state space includes the available space of the current task and the final available space.

[0012] Select the next operation in the action space. If the random value is less than the preset exploration rate, then randomly select an action.

[0013] If the random value is greater than the preset exploration rate, then the action corresponding to the maximum Q value is selected;

[0014] Based on the selected action, the Q value is updated according to the reward mechanism until the selected action corresponds to the target node, and the Q value corresponding to the target node is the optimal Q value.

[0015] In one embodiment, the step of calculating the optimal path and time slot allocation results for each device under the current task based on the optimal Q value includes:

[0016] Calculate the optimal path for each device under its current task based on the optimal Q value;

[0017] Obtain the path node sequence corresponding to the optimal path based on the optimal path;

[0018] The time slot allocation result is obtained by calculating the time slot allocation of each device based on the path node sequence.

[0019] In one embodiment, the step of scheduling network dynamic flows based on the optimal path and time slot allocation results includes:

[0020] The scheduled flow tasks are obtained from the queue of flows to be scheduled based on the optimal path and time slot allocation results.

[0021] Based on the path node sequence, the stream tasks to be scheduled are scheduled sequentially, starting from the source node.

[0022] In one embodiment, the step of scheduling the stream tasks to be scheduled sequentially from the source node according to the path node sequence includes:

[0023] When a sudden event or a historical periodic event occurs in the network status, the path and time slot allocation results are recalculated.

[0024] The suboptimal time slot result is obtained based on the recalculated path and time slot allocation results, and the suboptimal time slot result is sent to the flow path device of the current task;

[0025] The streaming tasks to be scheduled are scheduled based on the suboptimal time slot results.

[0026] In one embodiment, before the step of calculating the Q-value of each device using the Q-learning algorithm based on the state data and historical data to obtain the Q-value calculation result of each device under the current task, the following steps are included:

[0027] Based on the aforementioned status data and historical data, the adjusted network topology and device information are obtained;

[0028] Construct a topology adjacency matrix based on the adjusted network topology and device information;

[0029] Based on the aforementioned topological adjacency matrix, path initialization is performed using a path algorithm;

[0030] Based on the initialized path, define the function for the next action, which is used to ensure the validity of the selected path node.

[0031] Furthermore, to achieve the above objectives, this application also proposes a network dynamic flow scheduling device, which includes:

[0032] The data acquisition module is used to acquire the status data and historical data of each device in real time. The status data is used to monitor the changes in the target parameters of the devices in real time, and the historical data is used to analyze the device behavior and traffic trends.

[0033] The Q-value optimization module is used to calculate the Q-value of each device based on the state data and historical data using the Q-learning algorithm, and obtain the optimal Q-value of each device under the current task.

[0034] The result calculation module is used to calculate the optimal path and time slot allocation results for each device under the current task based on the optimal Q value;

[0035] The network scheduling module is used to schedule the dynamic flow of the network based on the optimal path and time slot allocation results.

[0036] In addition, to achieve the above objectives, this application also proposes a network dynamic flow scheduling device, the device comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the network dynamic flow scheduling method as described above.

[0037] In addition, to achieve the above objectives, this application also proposes a storage medium, which is a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, it implements the steps of the network dynamic flow scheduling method described above.

[0038] In addition, to achieve the above objectives, this application also provides a computer program product, which includes a computer program that, when executed by a processor, implements the steps of the network dynamic flow scheduling method described above.

[0039] One or more technical solutions proposed in this application have at least the following technical effects:

[0040] By acquiring real-time status and historical data from each device, the system monitors changes in device target parameters and analyzes device behavior and traffic trends. Using a Q-learning algorithm, it calculates the Q-value for each device under its current task, obtaining the optimal Q-value. Based on the optimal Q-value, it calculates the optimal path and time slot allocation for each device's current task. The system then schedules network dynamic flows based on these optimal path and time slot allocation results, ensuring the shortest paths and most reasonable resource allocation. This improves network transmission efficiency and reduces the risk of latency and network congestion. The dynamic flow scheduling mechanism can flexibly adjust the traffic load of each path in the network based on real-time data and historical trends, ensuring efficient utilization of network resources and improving device determinism in dynamic networks. Attached Figure Description

[0041] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0042] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0043] Figure 1 A schematic diagram illustrating the selection of the optimal path using the existing algorithm in this application;

[0044] Figure 2 This is a schematic diagram of the overall architecture of the dynamic flow scheduling system for a deterministic network according to an embodiment of this application.

[0045] Figure 3 This is a flowchart illustrating the first embodiment of the network dynamic flow scheduling method of this application;

[0046] Figure 4 This is a flowchart illustrating step S10 of an embodiment of this application;

[0047] Figure 5 This is a schematic diagram illustrating the network environment initialization process in an embodiment of this application.

[0048] Figure 6 This is a flowchart illustrating the second embodiment of the network dynamic flow scheduling method of this application;

[0049] Figure 7 This is a schematic diagram of the process for obtaining the Q value in an embodiment of this application;

[0050] Figure 8 This is a flowchart illustrating the third embodiment of the network dynamic flow scheduling method of this application;

[0051] Figure 9This is a flowchart illustrating the fourth embodiment of the network dynamic flow scheduling method of this application;

[0052] Figure 10 This is a schematic diagram of the flow scheduling and distribution process in a real-time environment according to an embodiment of this application;

[0053] Figure 11 This is a schematic diagram of the network environment obtained in the implementation method of this application;

[0054] Figure 12 This is a schematic diagram of the device information for APP1 and APP3 in the implementation method of this application;

[0055] Figure 13 This is a schematic diagram illustrating the device information of Test and testend devices as end switches in the implementation method of this application.

[0056] Figure 14 This is a schematic diagram showing the device information of devices 182 and SW18 as switches in the implementation method of this application.

[0057] Figure 15 This is a schematic diagram illustrating the specific information of the flow in the implementation method of this application;

[0058] Figure 16 This is a schematic diagram of the available state space in the implementation method of this application;

[0059] Figure 17 This is a table showing the Q-value information in the implementation method of this application;

[0060] Figure 18 This is a schematic diagram of the module structure of the network dynamic flow scheduling device according to an embodiment of this application;

[0061] Figure 19 This is a schematic diagram of the device structure of the hardware operating environment involved in the network dynamic flow scheduling method in this application embodiment.

[0062] The purpose, features, and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0063] It should be understood that the specific embodiments described herein are merely illustrative of the technical solutions of this application and are not intended to limit this application.

[0064] To better understand the technical solution of this application, a detailed description will be provided below in conjunction with the accompanying drawings and specific implementation methods.

[0065] Currently, most flow scheduling in deterministic networks suffers from the following problems: all flow scheduling is a simulation of the current network parameters under static conditions, which differs from the actual network situation and cannot be applied to flow scheduling planning in dynamic deterministic network environments. For example, if the device port rate is calculated as 1000Mbps during scheduling, but after the actual deployment, the device port rate becomes 100Mbps, or the device suddenly goes offline, the previous scheduling results are no longer applicable to the current network environment, and network determinism cannot be guaranteed. Existing scheduling algorithms first calculate the optimal path, then allocate all time slot resources of that optimal link or prevent the allocation of time slot resources for other flows as a condition for scheduling failure. However, in real networks, there are other path resources that can be successfully scheduled. For example, ... Figure 1 As shown, there are multiple paths from App1 to App3, and flow 1 is a non-redundant flow from App1 to App3. Under normal circumstances, the path App->dev-3->App3 is optimal. However, if dev3 is busy, although the determinism of flow 1 can be achieved by following the conventional optimal path, the time slot bandwidth left for other deterministic flows or non-time-sensitive flows is severely insufficient, increasing the load on the device. At this time, there may be idle device nodes in the network. Although conventional flow path congestion selection can also choose other paths for calculation, it requires recalculation and adjustment after scheduling. Therefore, it can be seen that the current flow planning result is a one-time calculation result for a static network. It cannot dynamically adjust paths and time slots according to the actual network conditions, and cannot guarantee device determinism in a dynamic network.

[0066] This application provides a solution that acquires real-time status and historical data of each device, monitors changes in device target parameters in real time, analyzes device behavior and traffic trends, calculates the Q-value of each device using a Q-learning algorithm, obtains the optimal Q-value for each device under the current task, calculates the optimal path and time slot allocation results for each device under the current task based on the optimal Q-value, and schedules the network dynamic flow according to the optimal path and time slot allocation results, ensuring the shortest path and the most reasonable resource allocation, improving network transmission efficiency, reducing latency and network congestion risks. The dynamic flow scheduling mechanism can flexibly adjust the traffic load of each path in the network according to real-time data and historical trends, ensuring efficient utilization of network resources and improving device determinism in dynamic networks.

[0067] It should be noted that this application can be applied to dynamic flow scheduling systems in deterministic networks (such as...). Figure 2As shown, the system consists of three main parts: First, the statistics and event reporting system is responsible for monitoring the operational status of the defense network and devices, as well as real-time monitoring of the network and device status. When a network fault or device anomaly is detected, the system generates an alarm and reports the relevant event to ensure timely response. Next, the scheduling and planning system, based on the identification of network device structure, constructs knowledge of network topology and node paths. Through the network feature module, it dynamically analyzes network characteristics to provide real-time traffic planning capabilities, enabling traffic planning and dynamic adjustment based on the actual situation of the current network. Finally, the network execution system, based on the decisions of the scheduling and planning system, optimizes the network through time slot intervention and network structure configuration, ensuring that instructions are correctly issued and executed, thereby achieving real-time intervention and adjustment of the network. These parts are closely linked through dynamic information transmission, ensuring that the network can be optimized and scheduled according to actual needs to maintain efficient and stable network operation.

[0068] Based on this, embodiments of this application provide a network dynamic flow scheduling method, referring to... Figure 3 , Figure 3 This is a flowchart illustrating the first embodiment of the network dynamic flow scheduling method of this application.

[0069] In this embodiment, the network dynamic flow scheduling method includes steps S10 to S40:

[0070] Step S10: Obtain the status data and historical data of each device in real time.

[0071] It should be noted that status data is used to monitor changes in device target parameters in real time, while historical data is used to analyze device behavior and traffic trends. For example, ... Figure 4As shown, each bridge device adds an agent client to acquire key parameters such as port speed, inbound and outbound traffic, and processing latency. The network parameter data collected by the agent client is uploaded to a time-series database, such as InfluxDB. The agent server on the deterministic network controller filters, statistically analyzes, and draws conclusions through scheduled tasks, storing the results in a MySQL database. Statistical analysis of historical device data yields historical network parameters and behavioral conclusions. These will serve as factors in selecting the optimal Q-value for network parameters in the Q-learning reinforcement learning environment and real-time scheduling. The agent client obtains details of the device's port operations from the network interface card (NIC), including port bandwidth, current speed, average port processing latency, inbound port traffic, outbound port traffic, number and size of port queues, port up / down status, and port priority. A channel exists between the agent client and the agent server for reporting events and specific real-time port data. The agent server periodically filters and statistically analyzes data in the time-series database, updates network parameters, and combines events to statistically determine port status and behavior. For example, it can statistically determine the total number of devices and ports in the network, the port connections between devices, and port throughput. Based on 7 consecutive days of historical data, the throughput of port 1 was more than 30% higher than other time periods between 18:00 and 20:00, and accounted for more than 80% of the total throughput. Therefore, it can be assumed that this port will also become busy between 18:00 and 20:00 in the future.

[0072] Step S20: Based on the status data and historical data, calculate the Q value of each device using the Q-learning algorithm to obtain the optimal Q value of each device under the current task.

[0073] It should be noted that the Q-value represents the expected reward (i.e., return) of choosing a certain action in a specific state. The Q-learning algorithm gradually finds the optimal strategy that maximizes the cumulative reward by continuously updating the Q-value table. The optimal Q-value can be the best reward that a device can achieve under the current conditions after using a specific path and time slot allocation.

[0074] Step S30: Calculate the optimal path and time slot allocation results for each device under the current task based on the optimal Q value.

[0075] It should be noted that the optimal path can be selected based on device status data, historical data, and the Q-value calculated by the Q-learning algorithm, choosing the best transmission path for network traffic under the current task. This path can maximize network performance requirements (such as minimum latency and maximum bandwidth utilization) and avoid congestion during scheduling. The time slot allocation result indicates how much time each device or link is allocated for data transmission under the current path.

[0076] Step S40: Schedule the network dynamic flow based on the optimal path and time slot allocation results.

[0077] For example, because the network environment is dynamic—such as changes in port rates of certain devices, increased link latency, or temporary device offline—the system can dynamically adjust flow scheduling based on real-time collected status data. When it finds that the current optimal path is no longer applicable, or that time slot allocation does not meet real-time requirements, the system will recalculate the path and time slot allocation, selecting a new optimal path and time slot. The scheduler will promptly adjust the data flow transmission path and resource allocation based on the new results to ensure the dynamic determinism of the network.

[0078] In this embodiment, by acquiring real-time status data and historical data of each device, the changes in the target parameters of the devices are monitored in real time, and the device behavior and traffic trends are analyzed. The Q-value of each device is calculated using the Q-learning algorithm to obtain the optimal Q-value for each device under the current task. Based on the optimal Q-value, the optimal path and time slot allocation results for each device under the current task are calculated. The network dynamic flow is scheduled according to the optimal path and time slot allocation results to ensure the shortest path and the most reasonable resource allocation, thereby improving network transmission efficiency and reducing the risk of latency and network congestion. The dynamic flow scheduling mechanism can flexibly adjust the traffic load of each path in the network according to real-time data and historical trends, ensuring the efficient utilization of network resources and improving the determinism of devices in dynamic networks.

[0079] In one implementation, before step S20, the process includes: obtaining adjusted network topology and device information based on state data and historical data; constructing a topology adjacency matrix based on the adjusted network topology and device information; initializing paths using a path algorithm based on the topology adjacency matrix; and defining a function for the next action based on the initialized paths, wherein the next action is used to ensure the legality of the selected path nodes.

[0080] It should be noted that this implementation method can be considered as the initialization of the network environment. For example, such as... Figure 5As shown, the system obtains the network topology and device information from the deterministic network controller and, combined with the network parameters of the device ports obtained from the MySQL database, further adjusts this network topology and device information to ensure that the data reflects the actual network state. Subsequently, based on the adjusted network topology and device information, the system constructs a topological adjacency matrix, where the value from each node to itself is set to 0, the value for a target node that the source node can directly reach is set to 1, and the value for a node that cannot be directly reached is set to -1. This provides the basis for subsequent path selection. After constructing the adjacency matrix, the system initializes all paths between all nodes in the network using a graph path algorithm. For example, the path from node A to node B can be initialized as [A,B][A,B][A,B]. These paths are ordered according to priority, and the subsequent reinforcement learning process will adjust and optimize the weights of these paths based on the actual state of the network. In addition, the time slot resources of device ports in the network are also initialized. The initial time slot resources of each port are set to a range of values ​​in microseconds, such as [0,1000000][0,1000000][0,1000000]. The specific time slot allocation will be dynamically adjusted in subsequent processes. Next, the system initializes the action space, including the action of rolling back to the previous node (prev) and the action of continuing to the next node (next_i). At the same time, the available state space and the final state space are initialized for the management of state during path and resource scheduling.

[0081] Understandably, to ensure the accuracy and effectiveness of path selection, the system defines a function for the next action. This function primarily checks the legality of the next action and determines whether the destination has been reached. Action legality checks consider three main aspects: first, ensuring the path to the node is reachable; second, checking whether the node's time slot resources can meet the current traffic demand; and finally, detecting the node's current busy status, especially whether the traffic load has exceeded 80%. Through this series of initializations and checks, the system lays the foundation for subsequent path optimization and time slot allocation, ensuring the legality and effectiveness of node and path selection during the scheduling process.

[0082] In this implementation, by acquiring adjusted network topology and device information based on status data and historical data, the system can dynamically reflect the actual operating status of the network. Compared to a static network topology, the dynamically adjusted topology is more adaptable, effectively responding to unexpected situations in the network and ensuring that scheduling results always meet actual needs. A topology adjacency matrix is ​​constructed based on the adjusted network topology, making the connection status between each node clear and intuitive. This matrix structure enables rapid path planning and initialization. Based on the initialized paths, the system defines the functions for the next action, ensuring that the selection of each path node undergoes a validity check, and guaranteeing the rationality of the selected path nodes in terms of reachability, time slot resource availability, and node load.

[0083] Reference Figure 6 , Figure 6 This is a flowchart illustrating the second embodiment of the network dynamic flow scheduling method of this application, based on the above. Figure 3 The first embodiment shown is followed by a second embodiment of the network dynamic flow scheduling method of this application.

[0084] In the second embodiment, step S20 includes:

[0085] Step S201: Based on the state data and historical data, obtain the state space and action space of Q-learning.

[0086] It's important to note that in network scheduling, the state space can be understood as a collection of available resources and network conditions related to the current task. The state space includes the available space for the current task and the final available space. The available space represents the availability status of nodes or ports in the current network (e.g., idle, busy, reachable, etc.), while the final available space represents the final target state that can be achieved after the task is completed, such as the endpoint node on a path or a path that meets certain transmission requirements. For example, if the number of next nodes is 2, the action space is updated to [prev, next_1, next_2], where prev represents returning to the previous node, and next_1 and next_2 represent the possibilities of reaching different next nodes, respectively.

[0087] Step S202: Select the next operation in the action space. If the random value is less than the preset exploration rate, then randomly select an action.

[0088] It's important to note that the action space can be used for selecting the next-hop node between network devices or for allocating time slots for data streams. Selecting the next operation within the action space typically involves a transfer from one node to the next, or choosing a specific path to transmit data. The preset exploration rate determines whether the algorithm explores (i.e., randomly selects an action) or utilizes (selects the action with the highest current Q-value) when choosing an action. When the random value is less than the preset exploration rate, the algorithm will randomly select an action, allowing the system to explore more possible paths and preventing the algorithm from getting trapped in local optima.

[0089] Step S203: If the random value is greater than the preset exploration rate, then select the action corresponding to the maximum Q value.

[0090] It should be noted that when the random value is greater than the preset exploration rate, the algorithm selects the action corresponding to the currently known maximum Q value. This means that the algorithm will use existing experience to select the optimal path. The maximum Q value can be understood as the Q value corresponding to the action with the highest profit in the current state.

[0091] For example, when selecting the next action, the system sets the exploration rate epsilon, which ranges from 0, 10, 10, to 1. During the learning phase, this value is set to a larger value (e.g., 0.8) to allow the system to explore more different paths; during the actual deployment phase, this value is set to a smaller value (e.g., 0.1) so that the system makes better use of the best paths it has already learned. When the randomness of the randomly generated action is less than the exploration rate epsilon, the system enters exploration mode and randomly selects an action; if the randomness is greater than the exploration rate epsilon, the system uses the learned Q-value to select the best action for the current node corresponding to the maximum Q-value.

[0092] Step S204: Based on the selected action, update the Q value according to the reward mechanism until the selected action corresponds to the target node and the Q value corresponding to the target node is the optimal Q value.

[0093] It's important to note that the reward mechanism defines the immediate feedback the system receives after performing a specific action in a given state. In network scheduling, the reward mechanism can be defined based on factors such as transmission latency, time slot utilization, and network load. When the system takes an action (e.g., choosing a path or allocating a time slot to a port), it will reward or penalize the action based on its effect. Rewards can be reduced transmission latency or improved bandwidth utilization, while penalties are negative consequences caused by network congestion or unreasonable resource allocation. Understandably, the design of the reward mechanism directly affects the update of the Q-value; therefore, after each action, the system updates the Q-value of that action in the current state based on the reward received. The target node is typically the destination node of the data flow or the final state upon task completion. Guided by the reward mechanism, the algorithm finds the optimal Q-value through multiple iterations and Q-value updates, ensuring that every decision during the scheduling process optimizes the overall network performance. For example, if the next node's time slot meets the requirements and it is not the destination node, the reward is 1; if the time slot does not meet the requirements, the penalty is -1000; if the next node is not the destination node and there are no other reachable nodes after it, the reward is -1000; if the next node's time slot meets the requirements and it is the destination node, a higher reward of 1000 is obtained. If the path is long, the system will discount the reward. For paths with a length of 1-4 nodes, the reward remains at 100%; for each additional node, the reward is discounted by 20%. For paths with low latency, the system further sets a tiered discount rate for the reward based on the specific latency to ensure that low-latency paths receive greater rewards. After each operation, the system updates the Q-value of the current node based on the reward mechanism and the learning algorithm. The formula for updating the Q-value is as follows:

[0094] Q new = (1 - learning_rate) × Q old +learning_rate

[0095] ×(reward+discount_factor×Q next_max )

[0096] Among them, Q new This represents the updated Q value, Q. old This represents the current Q-value, learning_rate represents the learning rate (set to 0.1), discount_factor represents the discount rate (set to 0.9), and reward represents the immediate reward obtained from the current action. next_maxThis is the maximum Q-value for the next node. Through this update method, the system can continuously optimize the Q-value of each node, thereby finding the optimal path in continuous training iterations. If the currently found node is the target node, it indicates that the training is complete, and the system will update the path list for the current task and continue the cyclical training for the next task. This process continues until the predetermined number of training iterations (e.g., 1000) is used up. The entire method for obtaining the optimal Q-value is as follows: Figure 7 As shown, during this process, the Q value gradually converges, eventually representing the optimal choice for each node under a specific task, thereby achieving optimization of dynamic network scheduling.

[0097] In this embodiment, by acquiring the state space and action space of Q-learning based on state data and historical data, the system can reflect the actual network status in real time and flexibly adjust resource allocation and path selection according to task requirements. By introducing a preset exploration rate, the system can randomly select actions for exploration at different stages. This mechanism effectively avoids the system from prematurely converging to a local optimum, thereby expanding the search space and discovering better scheduling schemes. Simultaneously, when the exploration rate is low, the system can prioritize currently known optimal actions, ensuring scheduling efficiency. This balance between exploration and utilization significantly improves the intelligence and robustness of the scheduling algorithm. By continuously updating the Q-value through the Q-learning algorithm, the system can continuously optimize the action selection at each step based on historical experience and reward mechanisms. By calculating the maximum Q-value, it ensures that the selected action brings the greatest cumulative benefit, thus ensuring the efficiency of network scheduling.

[0098] Reference Figure 8 , Figure 8 This is a flowchart illustrating the third embodiment of the network dynamic flow scheduling method of this application, based on the above. Figure 2 The second embodiment shown presents a third embodiment of the network dynamic flow scheduling method of this application.

[0099] In the third embodiment, step S30 includes:

[0100] Step S301: Calculate the optimal path for each device under the current task based on the optimal Q value.

[0101] It should be noted that the optimal path is selected based on the optimal Q value, that is, under the current task, the system finds the most efficient transmission path after path exploration and Q value optimization.

[0102] Step S302: Obtain the path node sequence corresponding to the optimal path based on the optimal path.

[0103] It should be noted that the path node sequence can be all intermediate nodes that connect the source node to the target node in the optimal path. Each node in the path node sequence can represent a device or port in the network. Through these nodes, the task flows from the source to the destination.

[0104] Step S303: Calculate the time slot allocation for each device based on the path node sequence to obtain the time slot allocation result.

[0105] It should be noted that the time slot allocation result is an allocation scheme calculated based on the path node sequence and network device status, representing the communication time slots that each device can use within a specific time period.

[0106] In this embodiment, the optimal path for each device under its current task is calculated based on the optimal Q-value. This ensures that the system automatically selects the most efficient and lowest-latency path after considering factors such as network conditions, load status, and time slot availability. The corresponding path node sequence is obtained based on the optimal path, ensuring that every intermediate device node traversed by the task from the source node to the target node is optimally selected. This allows for better control of the data flow's transmission order and resource allocation within the network, thereby improving overall network performance. The time slot allocation for each device is calculated based on the path node sequence, ensuring that each device has reasonable time slot resources during task transmission.

[0107] Reference Figure 9 , Figure 9 This is a flowchart illustrating the fourth embodiment of the network dynamic flow scheduling method of this application, based on the above. Figure 8 The third embodiment shown presents a fourth embodiment of the network dynamic flow scheduling method of this application.

[0108] In the fourth embodiment, step S40 includes:

[0109] Step S401: Obtain the scheduled flow tasks from the queue of flows to be scheduled based on the optimal path and time slot allocation results.

[0110] It should be noted that the queue of pending data streams refers to the collection of all data stream tasks waiting to be scheduled and processed in the current network. These tasks may not have been allocated a path or time slot due to network load, time slot resources, or other reasons. The system will schedule these tasks in a certain order according to the task priority, network conditions, etc.

[0111] Step S402: Based on the path node sequence, schedule the stream tasks to be scheduled sequentially, starting from the source node.

[0112] It should be noted that the source node can be the starting node of a streaming task, responsible for sending data packets into the network and transmitting them to the target node along the optimal path. The path node sequence defines each node that the data stream passes through in sequence during transmission. The scheduling process ensures that each task follows the plan of the path node sequence and time slot allocation, starting from the source node, passing through intermediate nodes, and finally reaching the target node.

[0113] In this embodiment, the system rapidly extracts the required streaming tasks from the queue of tasks to be scheduled by utilizing pre-calculated optimal paths and time slot allocation results. Since the scheduling of streaming tasks is based on previously optimized optimal paths and time slot allocation results, the system can ensure that the resources of each node and each time slot are used rationally, avoiding the complex process of recalculating paths and resources for each task scheduling, thus greatly improving scheduling efficiency. Streaming tasks are scheduled sequentially from the source node using a path node sequence, ensuring that streaming tasks can smoothly pass through each node along the optimal path. This guarantees high determinism in the transmission process from the source node to the target node, reducing the risk of latency and interruption.

[0114] In one implementation, based on the fourth embodiment described above, step S402 includes: when a sudden event or a historical periodic characteristic event occurs in the network state, recalculating the path and time slot allocation results; obtaining the suboptimal time slot result based on the recalculated path and time slot allocation results, and sending the suboptimal time slot result to the flow path device of the current task; and scheduling the flow task to be scheduled according to the suboptimal time slot result.

[0115] It should be noted that sudden events can be unexpected situations that occur suddenly during network operation, potentially having a significant impact on network performance or device status. Examples include sudden changes in the port speed of certain devices, device outages, link failures, and network congestion. Historical periodic characteristic events refer to regular behaviors or periodic phenomena in the network based on historical data. For example, certain device ports may experience load surges at specific times of the day (such as peak periods), or certain paths may experience increased latency during fixed time periods. Flow path devices refer to the various node devices that a flow task passes through in the network. When sudden events or periodic characteristic changes occur in the network, the system needs to recalculate the path and time slot allocation results. The purpose of recalculation is to ensure that tasks can continue under the new network conditions without being affected by changes in the current network state. Suboptimal time slot results can be understood as a relatively optimal, but not globally optimal, time slot allocation scheme calculated by the system after recalculating the path and time slot allocation.

[0116] In this embodiment, by recalculating the path and time slot allocation results, the system can respond to changes in the network environment in real time, making the scheduling process more flexible and adaptable. By quickly adjusting the scheduling strategy, the smooth transmission of tasks is ensured. Even when the optimal result cannot be obtained immediately, the use of suboptimal time slot results can still provide a reasonable scheduling scheme for streaming tasks and maintain the stable operation of tasks.

[0117] To make the embodiments and implementations of this application more specific, based on the above embodiments and implementations, this application proposes another implementation for stream scheduling and delivery in a real-time environment.

[0118] In one implementation, the parameters of Q-learning reinforcement learning are first adjusted, including the exploration rate ε (epsilon) to control the balance between exploration and exploitation when selecting actions, the reward discount rate to represent the importance of future rewards, and the learning rate to control the trade-off between new and old valuations; the macro-period is calculated from the queue of pending scheduling flows, and can be expressed as:

[0119] T 宏周期 =LCM 最小公倍数 (t peroid1 ,t peroid2 ,...,t peroidn )

[0120] Among them, the macro period can be expressed as all the flow periods (t) period The least common multiple of (). A flow task is obtained from the queue of flows to be scheduled. Based on the basic information of the flow, the corresponding network parameters in the network environment are obtained, including available space, final available space, and action space. Next, the system obtains the path information from the source node to the target node related to this task. Based on this path, scheduling begins from the source node, and the network environment parameters are dynamically adjusted using historical and real-time data of the nodes. For the selection of the next node, the system must ensure that the time slot meets the requirements. For example, the time slot formula is expressed as follows:

[0121]

[0122] Among them, t 拓扑传输时延 For example, t 拓扑传输时延 The link delay represents the topology transmission latency; the meanings of the other time slots are primarily based on their subscripts. The link delay, port forwarding delay, and port precision synchronization delay are all device or network capabilities. The specific data initialization is indicator data, which can be dynamically adjusted based on historical data analysis. Port preemption time can be expressed as:

[0123] t 端口帧抢占时间 = (Frame Interval + Preemption + Pre-position) * 8 / portSpeed

[0124] Here, portSpeed ​​represents the outgoing port speed. RR 流在宏周期内的循环在数 =T 宏周期 / t peroid This means that the time slots occupied within the macro period will cycle RR times, and the total time slot size for the next node is T. 宏周期 The transmission time of a streaming frame can be expressed as:

[0125]

[0126] Furthermore, the next node must be a node in the path, and load balancing prioritizes the node with the highest load balance. Load balancing can be represented as:

[0127] α*Q value + β*K 负载均衡

[0128] α+β=1, β = 0.5

[0129] Load balancing is a step function. If no suitable node is found on the current path, the system will fall back to the previous node and recalculate until a suitable target node is found. After the calculation is complete, the system will recalculate the optimal time slot and the backup suboptimal time slot. Then, the optimal time slot result is sent to the devices on the flow path, and the time slot status of the devices in the network environment is updated. The entire flow scheduling process also needs to consider sudden events and historical periodic characteristic events during the sending process. Sudden events may change network environment parameters, so the path and time slot need to be recalculated. Historical periodic characteristic events predict the busy status of the flow path in a specific time period and send the suboptimal time slot result to the backup path devices in advance to ensure the determinism of flow scheduling in sudden and predicted situations. The suboptimal time slot result is obtained in two ways: one is by adjusting the time slot of the device under the same path, and the other is by changing the time slot allocation due to path changes. If the time slot sending fails, the system will continue to execute the original time slot scheme. The entire process is as follows: Figure 10 As shown. The Q-value of this flow scheduling will also be saved as material for future reinforcement learning training. This algorithm helps the flow quickly plan paths and time slots using reward experience values ​​during actual scheduling by performing reinforcement learning and prediction of node historical data in advance in a static network environment; at the same time, the selection of path nodes, combined with traffic balancing and historical data, makes the scheduling results more in line with the actual situation and improves the system's adaptability and efficiency in complex network environments.

[0130] In order to provide better support when the above embodiments and implementation methods are applied to the system, Figure 11 The network environment shown and Figures 12 to 14 Using the device information shown as an example, this paper illustrates the specific implementation of the network dynamic flow scheduling method of this application. Figure 12For device information of APP1 and APP3, Figure 13 This provides device information for the Test and testend devices as end switches. Figure 14 This provides device information for devices 182 and SW18 as switches. When scheduling a flow A using the Q_learning method, the specific flow information is as follows: Figure 15 As shown. The main steps include:

[0131] (1) Initialize the environment’s device information, device port timeslots, topology link information, etc., and represent the topology using a matrix.

[0132] Device list: ['APP1','testend','device182','SW18','test','APP3']

[0133] Device link: {"APP1_1":"testend_1","testend_1":"APP1_1","device182_2":

[0134] "testend_2","testend_2":"Device 182_2","testend_4":"SW18_4",

[0135] "SW18_4":"testend_4","SW18_2":"test_2","test_2":

[0136] "SW18_2","device182_1":"test_1","test_1":"device182_1",

[0137] "test_3":"APP3","APP3":"test_3"}

[0138] Device port timeslot: {“APP1_1”:[(0,1000000)]......}

[0139] The topological matrix is ​​represented in Table 1:

[0140] Table 1

[0141]

[0142] (2) Initialize the action space [prev, next] and the available state space (points in the matrix that are greater than 0, such as...). Figure 16 (As shown). The first column represents the sequence of rows, and the second column represents the sequence of columns.

[0143] (3) Set the learning rate to 0.1, the reward discount rate to 0.9, the number of learning iterations to 1000, and the epsilon (a parameter used to control the balance between exploration and exploitation in the agent's action selection) to 0.1. Use reinforcement learning to obtain the path from any point to other points and the Q-value of each point. The Q-value of other nodes in the path is -1000, specifically expressed as... Figure 17 The table shown.

[0144] (4) Adjust reinforcement learning parameters: The number of learning iterations is 2, and the epsilon value is 0.9. Start scheduling the flow in a real network environment. The flow is from APP1 to APP3. According to step 3, there are two paths in total, and the Q values ​​are basically the same. Therefore, a path is randomly selected. The following simulates the scheduling process of reinforcement learning. The randomly selected path is represented as APP1->testend->182->test->APP3. The frame preemption time is (96+12176+64) / 1000000=0.012336us, and the flow transmission time is (1000*8) / 1000=8us.

[0145] (5) Figure 15 As shown, if device 182 reports traffic exceeding 80% of the total bandwidth, the Q value of device 182 in the next scheduling will be 0.5*1+0*0=0.5, which is less than SW18. Therefore, the path will become APP1>testend->SW18->test->APP.

[0146] (6) If the time slot of the next stream scheduling on device 182 does not meet the requirements, the Q value on device 182 will be updated to -1000. Therefore, the streams scheduled in batches need to be sorted according to frame length and period to ensure that if the previous stream cannot be scheduled, the subsequent streams will not meet the conditions either.

[0147] (7) If the statistics of equipment operation show that equipment 182 is very busy between 18:00 and 20:00 every day, then the newly scheduled flow will no longer be considered to pass through equipment 182, and the Q value of the equipment will become smaller during that time period.

[0148] (8) If a device is found to be offline during operation, a backup path that meets the conditions will be immediately activated. If no backup path can be found, the original state will be maintained.

[0149] In summary, this application identifies the operational patterns of devices through static data analysis and derives all point-to-point paths and device Q-values ​​through advance exploration of the network environment. In actual network scheduling within a deterministic network environment, it fully considers the actual conditions, operational patterns, and load balancing of devices. Utilizing prior knowledge, it dynamically plans network resources through reinforcement learning, reducing actual scheduling time and resolving algorithm failure issues caused by insufficient consideration of the dynamic network environment in conventional scheduling methods.

[0150] It should be noted that the above examples are only for understanding this application and do not constitute a limitation on the network dynamic flow scheduling method of this application. Any simple modifications based on this technical concept are within the protection scope of this application.

[0151] This application also provides a network dynamic flow scheduling device, please refer to... Figure 18 The network dynamic flow scheduling device includes:

[0152] The data acquisition module 10 is used to acquire the status data and historical data of each device in real time. The status data is used to monitor the changes in the target parameters of the devices in real time, and the historical data is used to analyze the device behavior and traffic trends.

[0153] Q-value optimization module 20 is used to calculate the Q-value of each device based on the state data and historical data using the Q-learning algorithm, and obtain the optimal Q-value of each device under the current task.

[0154] The result calculation module 30 is used to calculate the optimal path and time slot allocation results for each device under the current task based on the optimal Q value;

[0155] The network scheduling module 40 is used to schedule the dynamic flow of the network according to the optimal path and time slot allocation results.

[0156] The network dynamic flow scheduling device provided in this application, employing the network dynamic flow scheduling method described in the above embodiments, can solve the technical problem of low determinism of network devices in dynamic network environments. Compared with the prior art, the beneficial effects of the network dynamic flow scheduling device provided in this application are the same as those of the network dynamic flow scheduling method described in the above embodiments, and other technical features in the network dynamic flow scheduling device are the same as those disclosed in the methods of the above embodiments, and will not be repeated here.

[0157] This application provides a network dynamic flow scheduling device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the network dynamic flow scheduling method in the above embodiment 1.

[0158] The following is for reference. Figure 19 This document illustrates a structural diagram of a network dynamic flow scheduling device suitable for implementing embodiments of this application. The network dynamic flow scheduling device in the embodiments of this application may include, but is not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Description), PMPs (Portable Media Players), in-vehicle terminals (e.g., in-vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers. Figure 19 The network dynamic flow scheduling device shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of this application.

[0159] like Figure 19 As shown, the network dynamic flow scheduling device may include a processing unit 1001 (e.g., a central processing unit, a graphics processing unit, etc.) that can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1002 or a program loaded from a storage device 1003 into a random access memory (RAM) 1004. The RAM 1004 also stores various programs and data required for the operation of the network dynamic flow scheduling device. The processing unit 1001, ROM 1002, and RAM 1004 are interconnected via a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Typically, the following systems can be connected to the I / O interface 1006: input devices 1007 including, for example, a touchscreen, touchpad, keyboard, mouse, image sensor, microphone, accelerometer, gyroscope, etc.; output devices 1008 including, for example, a liquid crystal display (LCD), speaker, vibrator, etc.; storage devices 1003 including, for example, magnetic tape, hard disk, etc.; and communication devices 1009. Communication device 1009 allows the network dynamic flow scheduling device to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 19 Network dynamic flow scheduling devices with various systems are shown; however, it should be understood that implementation or having all of the systems shown is not required. More or fewer systems may be implemented alternatively.

[0160] Specifically, according to the embodiments disclosed in this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments disclosed in this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device, or installed from storage device 1003, or installed from ROM 1002. When the computer program is executed by processing device 1001, it performs the functions defined in the methods of the embodiments disclosed in this application.

[0161] The network dynamic flow scheduling device provided in this application, employing the network dynamic flow scheduling method described in the above embodiments, can solve the technical problem of low determinism of network devices in dynamic network environments. Compared with the prior art, the beneficial effects of the network dynamic flow scheduling device provided in this application are the same as those of the network dynamic flow scheduling method described in the above embodiments, and other technical features of this network dynamic flow scheduling device are the same as those disclosed in the previous embodiment method, and will not be repeated here.

[0162] It should be understood that the various parts disclosed in this application can be implemented using hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics can be combined in any suitable manner in one or more embodiments or examples.

[0163] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0164] This application provides a computer-readable storage medium having computer-readable program instructions (i.e., a computer program) stored thereon, the computer-readable program instructions being used to execute the network dynamic flow scheduling method in the above embodiments.

[0165] The computer-readable storage medium provided in this application may be, for example, a USB flash drive, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, system, or device. The program code contained on the computer-readable storage medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination thereof.

[0166] The aforementioned computer-readable storage medium may be included in the network dynamic flow scheduling device; or it may exist independently and not be assembled into the network dynamic flow scheduling device.

[0167] The aforementioned computer-readable storage medium carries one or more programs. When these programs are executed by the network dynamic flow scheduling device, the network dynamic flow scheduling device: acquires real-time status data and historical data of each device, wherein the status data is used to monitor changes in device target parameters in real time, and the historical data is used to analyze device behavior and traffic trends; calculates the Q-value of each device using a Q-learning algorithm based on the status data and historical data, and obtains the optimal Q-value for each device under its current task; calculates the optimal path and time slot allocation result for each device under its current task based on the optimal Q-value; and schedules the network dynamic flow based on the optimal path and time slot allocation result.

[0168] Computer program code for performing the operations of this application can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a Local Area Network (LAN) or a Wide Area Network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0169] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0170] The modules described in the embodiments of this application can be implemented in software or hardware. The names of the modules do not necessarily limit the functionality of the unit itself.

[0171] The readable storage medium provided in this application is a computer-readable storage medium that stores computer-readable program instructions (i.e., a computer program) for executing the above-described network dynamic flow scheduling method, which can solve the technical problem of low determinism of network devices in dynamic network environments. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in this application are the same as the beneficial effects of the network dynamic flow scheduling method provided in the above embodiments, and will not be repeated here.

[0172] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the network dynamic flow scheduling method described above.

[0173] The computer program product provided in this application can solve the technical problem of low determinism of network devices in dynamic network environments. Compared with the prior art, the beneficial effects of the computer program product provided in this application are the same as those of the network dynamic flow scheduling method provided in the above embodiments, and will not be repeated here.

[0174] The above description is only a part of the embodiments of this application and does not limit the patent scope of this application. All equivalent structural transformations made under the technical concept of this application and using the contents of the specification and drawings of this application, or direct / indirect applications in other related technical fields, are included in the patent protection scope of this application.

Claims

1. A network dynamic flow scheduling method, characterized in that, The method includes: The status data and historical data of each device are acquired in real time. The status data is used to monitor changes in the target parameters of the devices in real time, and the historical data is used to analyze device behavior and traffic trends. Based on the state data and historical data, the Q value of each device is calculated using the Q-learning algorithm to obtain the optimal Q value of each device under the current task. Calculate the optimal path and time slot allocation results for each device under the current task based on the optimal Q value; The network dynamic flow is scheduled based on the optimal path and time slot allocation results.

2. The method as described in claim 1, characterized in that, The step of calculating the Q-value of each device using the Q-learning algorithm based on the state data and historical data, and obtaining the optimal Q-value for each device under the current task, includes: Based on the state data and historical data, the state space and action space of Q-learning are obtained, wherein the state space includes the available space of the current task and the final available space. Select the next operation in the action space. If the random value is less than the preset exploration rate, then randomly select an action. If the random value is greater than the preset exploration rate, then the action corresponding to the maximum Q value is selected; Based on the selected action, the Q value is updated according to the reward mechanism until the selected action corresponds to the target node, and the Q value corresponding to the target node is the optimal Q value.

3. The method as described in claim 1, characterized in that, The step of calculating the optimal path and time slot allocation results for each device under the current task based on the optimal Q value includes: Calculate the optimal path for each device under its current task based on the optimal Q value; Obtain the path node sequence corresponding to the optimal path based on the optimal path; The time slot allocation result is obtained by calculating the time slot allocation of each device based on the path node sequence.

4. The method as described in claim 3, characterized in that, The step of scheduling network dynamic flows based on the optimal path and time slot allocation results includes: The scheduled flow tasks are obtained from the queue of flows to be scheduled based on the optimal path and time slot allocation results. Based on the path node sequence, the stream tasks to be scheduled are scheduled sequentially, starting from the source node.

5. The method as described in claim 4, characterized in that, The step of scheduling the stream tasks to be scheduled sequentially from the source node according to the path node sequence includes: When a sudden event or a historical periodic event occurs in the network status, the path and time slot allocation results are recalculated. The suboptimal time slot result is obtained based on the recalculated path and time slot allocation results, and the suboptimal time slot result is sent to the flow path device of the current task; The streaming tasks to be scheduled are scheduled based on the suboptimal time slot results.

6. The method according to any one of claims 1 to 5, characterized in that, Before the step of calculating the Q-value of each device using the Q-learning algorithm based on the state data and historical data to obtain the Q-value calculation result of each device under the current task, the following steps are included: Based on the aforementioned status data and historical data, the adjusted network topology and device information are obtained; Construct a topology adjacency matrix based on the adjusted network topology and device information; Based on the aforementioned topological adjacency matrix, path initialization is performed using a path algorithm; Based on the initialized path, define the function for the next action, which is used to ensure the validity of the selected path node.

7. A network dynamic flow scheduling device, characterized in that, The device includes: The data acquisition module is used to acquire the status data and historical data of each device in real time. The status data is used to monitor the changes in the target parameters of the devices in real time, and the historical data is used to analyze the device behavior and traffic trends. The Q-value optimization module is used to calculate the Q-value of each device based on the state data and historical data using the Q-learning algorithm, and obtain the optimal Q-value of each device under the current task. The result calculation module is used to calculate the optimal path and time slot allocation results for each device under the current task based on the optimal Q value; The network scheduling module is used to schedule the dynamic flow of the network based on the optimal path and time slot allocation results.

8. A network dynamic flow scheduling device, characterized in that, The device includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the network dynamic flow scheduling method as described in any one of claims 1 to 6.

9. A storage medium, characterized in that, The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, it implements the steps of the network dynamic flow scheduling method as described in any one of claims 1 to 6.

10. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, implements the steps of the network dynamic flow scheduling method as described in any one of claims 1 to 6.

Images