Resource intelligent joint optimization method based on deep reinforcement learning

Abstract

The invention discloses a resource intelligent joint optimization method based on deep reinforcement learning, and the method comprises the following steps: 1, building a communication network topology, and detecting the data flow in the network topology; 2, randomly generating a task flow, and simulating analysis and sending of tasks in the network topology; 3, the switch receives the task and searches n simple paths of the task; 4, setting constraint conditions, and screening candidate paths from the n simple paths; step 5, taking the minimum total time delay as a target, selecting an optimal path from the candidate paths, and distributing optimal computer resources and communication resources for the optimal path; and step 6, issuing the communication resources allocated to each link of the optimal path to a switch to realize communication resource allocation. According to the method provided by the invention, more concurrent tasks can be effectively accommodated on the premise of ensuring the task delay requirement, so that the resource utilization rate is maximized.

Description

Technical Field

[0001] The present invention belongs to the field of communication technology, and in particular relates to a resource intelligent joint optimization method based on deep reinforcement learning. Background Art

[0002] In today's complex and ever-changing communication environment, communication networks not only need to efficiently manage and allocate communication resources, but also must consider the joint allocation of computing resources. The dynamic situation in the communication environment is often unpredictable, which places higher demands on decision makers. They need to make quick and accurate judgments in ever-changing situations and formulate corresponding strategies. However, it is difficult to ensure the accuracy of decisions based solely on the wisdom and experience of decision makers. Therefore, the introduction of computer-aided decision-making systems has become an urgent problem to be solved.

[0003] In recent years, with the rapid development of artificial intelligence technology, researchers have begun to explore how to use intelligent algorithms to assist decision makers in making decisions. This can not only improve the scientificity and accuracy of decision-making, but also effectively reduce the consumption of human resources. In this context, the joint allocation of communication and computing resources is particularly important. At this stage, there are mainly three common methods: game theory methods, heuristic algorithms, and artificial intelligence methods.

[0004] As a mathematical tool for studying the interaction and competition between decision makers, game theory has been widely used in resource allocation problems in the field of communications. In recent years, with the development of information technology, game theory has been gradually introduced into resource allocation, especially in the context of information communication, where game theory is used to analyze the resource allocation and strategy selection of both parties. In view of the shortage of communication and computing resources, the existing technology has constructed a new multi-leader single-follower Stackelberg game model, aiming to minimize the delay and cost of task processing. At the same time, a resource allocation algorithm based on selection optimization is proposed, and the Karush-Kuhn-Tucker algorithm is used to jointly optimize the task allocation and bandwidth allocation ratio to minimize the task processing delay. In addition, the binary search method is used to fine-tune the optimization of computing resources, aiming to minimize the total cost while ensuring that the task delay limit is met. However, the envisioned scenario is relatively simple, and its effectiveness in complex communication networks cannot be fully verified. There is also a method that uses game theory and Lagrangian functions to solve the optimization problem of minimizing the energy consumption of all users and obtain the optimal resource allocation solution, but the application scenario is also ideal, and the impact of factors such as possible emergencies is not considered, and the result lacks practicality.

[0005] Heuristic algorithms have shown good performance in solving complex optimization problems due to their flexibility and efficiency, especially in the dynamic allocation of communication network resources. In recent years, domestic and foreign scholars have proposed a variety of heuristic methods for problems such as task scheduling and spectrum allocation, such as genetic algorithms and particle swarm optimization. These algorithms can not only quickly solve approximate optimal solutions, but also maintain good robustness under uncertain conditions. In uncertain networks, due to the randomness of node movement and the time-varying nature of wireless channel states, uncertain queuing delays, network device connection times and other network environment characteristics appear, which greatly affect the utilization of computing and communication resources. With the goal of minimizing the total energy consumption of the system, a multi-stage random programming optimization algorithm for joint resource allocation based on random simulation (SS-MSSP) is proposed. The genetic algorithm is used to obtain the optimal allocation strategy of transmission power and computing resources. However, this method only considers modeling random dynamic factors from the time dimension, and other dimensions have not been considered, making it difficult to adapt to more application scenarios. In addition, most existing strategies do not consider the dependencies between computing tasks, or are usually based on heuristic search algorithms, which is unacceptable for delay-sensitive services. A new graph-based workflow strategy is proposed. This strategy can handle complex workflow applications with nonlinear structures, and at the same time uses graph-based partitioning technology to find the decision-making plan with the lowest energy consumption under delay constraints. It does not consider the allocation of multi-hop bandwidth.

[0006] The development of artificial intelligence technology has provided new ideas for communication network decision-making, especially in the allocation and management of resources. In recent years, the application of AI technologies such as deep learning and reinforcement learning in communication networks has received increasing attention. Some research institutions at home and abroad have begun to explore how to integrate AI systems into communication networks to improve response speed and allocation quality. Research on allocation strategies based on artificial intelligence has been carried out, and attempts have been made to achieve intelligent allocation and dynamic adjustment of resources through machine learning algorithms, aiming to improve overall communication efficiency. For example, for the problem of insufficient node computing power and uneven bandwidth allocation in computing power networks, a deep reinforcement learning algorithm based on incentive mechanism and combined with proximal policy optimization (PPO) is proposed. Under the constraints of computing resources and transmission power, a system benefit-maximizing allocation scheme is calculated by jointly optimizing computing resources and transmission power. However, this scheme only considers the overall benefit and does not take into account the needs of delay-sensitive tasks. There is also a deep Q network to solve the bandwidth allocation problem, and a deep deterministic policy gradient algorithm to solve the continuous power allocation problem. A DRL algorithm based on meta-resources is also proposed to enhance the rapid adaptability of resource allocation strategies in dynamic environments. However, the problem of complex and variable link bandwidth brought about by different transmission modes is not considered. There is also an existing technology that constructs the optimization problem of latency and resource consumption through mathematical modeling and describes it as a Markov decision process. It improves the traditional soft behavior critical deep reinforcement learning framework through priority experience replay and proposes a resource allocation algorithm based on SACPER to minimize the total cost of the system. However, this method does not consider the scenario of a large number of user devices accessing at the same time, and the designed solution does not estimate the constraints of dependencies between tasks. Summary of the invention

[0007] Purpose of the invention: In order to solve the problems existing in the above-mentioned prior art, the present invention provides a resource intelligent joint optimization method based on deep reinforcement learning.

[0008] Technical solution: The present invention discloses a resource intelligent joint optimization method based on deep reinforcement learning, which comprises the following steps:

[0009] Step 1: Build a communication network topology, detect the data traffic in the network topology, record the link bandwidth, remaining bandwidth, and usage of computing resources of each communication node;

[0010] Step 2: Randomly generate task flow T and simulate task parsing and sending in the network topology;

[0011] Step 3: The switch receives the task in the task flow After parsing the data packet header, check whether there is a corresponding flow rule for the data packet in its own flow table. If there is a corresponding flow rule, forward the task directly to the corresponding port set in the flow rule, otherwise find n simple paths for the task;

[0012] Step 4: Task-based Minimum required transfer rate and nodes and The link between the tasks Assigned transfer rate , set constraints to screen candidate paths from n simple paths; nodes i and j are nodes at both ends of the link, i=1,2,...,n; j=1,2,...,n, n represents the total number of nodes;

[0013] Step 5: With the goal of minimizing the total delay, select the optimal path from the candidate paths and allocate optimal computing resources and communication resources to the optimal path;

[0014] Step 6: Send the communication resources allocated to each link of the optimal path to the switch to implement communication resource allocation.

[0015] Furthermore, the calculation formula of the remaining bandwidth free_bw in step 1 is as follows:

[0016] free_bw = capability – speed

[0017] Among them, capability represents the total link bandwidth, and speed represents the flow rate.

[0018] Furthermore, step 3 uses a breadth-first algorithm to find n simple paths.

[0019] Furthermore, the constraints in step 4 are:

[0020] ;

[0021] in, For Node and The available transmission rate of the link between For Node For the task Allocated computing resources, For Node Available computing resources, The expression is:

[0022] ;

[0023] in, For Node and The link between the tasks The allocated bandwidth, For Node and The link between the tasks the allocated transmission power; Represents the background noise of the channel.

[0024] Furthermore, step 5 uses the D3QN algorithm to select the optimal path from the candidate paths, and allocates optimal computer resources and communication resources to the optimal path, specifically:

[0025] Set the objective function: Set a leaky bucket controller with a capacity of b and a flow rate of b per second. The peak velocity of the fluid is , using leaky bucket controller to calculate the task The end-to-end delay bound :

[0026] ;

[0027] in, For the task The size of the largest packet in; Indicates that if , ,otherwise , Indicates the length of the largest data packet in all task flows in the router. Indicates the length of the maximum data packet in the leaky bucket controller; Representation Node Send to node rate;

[0028] Task-based The end-to-end delay bound Calculate the total delay of task transmission and take the minimum total delay as the objective function:

[0029] ;

[0030] in, is the total delay of task transmission, For analysis tasks The delay generated, The expression is:

[0031] ;

[0032] in, For Node For the task Allocated computing resources, For Node of Number of cycles, For the task The total amount of data that needs to be calculated;

[0033] Setting up the state space for:

[0034] ;

[0035] Among them, h represents the current moment, M represents the computational data sequence of the task, and M ,in Indicates the task The amount of computing data, X represents the total number of tasks, represents the minimum required transmission rate sequence of the task, , represents the maximum end-to-end delay sequence allowed by the task, , Indicates the task The maximum allowed end-to-end delay; Represents the sequence of available computing resources of the node at the current moment, , Indicates the available computing resources of the i-th node at the current moment; Represents a sequence of available transmission rates, , For Node and The available transmission rate of the link between

[0036] Setting up the action space :

[0037] ;

[0038] Agents on discrete numerical sets Select a value from the list and assign it to , Y represents the total number of discrete values, Indicates the preset parameters. Indicates the allocation of communication resources. It represents the use of non-uniform bandwidth allocation algorithm to solve communication resources. The representative adopts the uniform allocation algorithm to solve the communication resources;

[0039] Set up the reward function:

[0040] .

[0041] Furthermore, the agent adopts a greedy learning strategy to select a value from the discrete value set and assign it to .

[0042] A computer device comprises a memory, a processor and a computer program stored in the memory and capable of running on the processor, wherein the resource intelligent joint optimization method is implemented when the processor executes the computer program.

[0043] A computer-readable storage medium is used to store a program, and the program is executed to implement the resource intelligent joint optimization method.

[0044] Beneficial effect: From the perspective of intelligent resource allocation in the communication network, the present invention proposes a resource intelligent joint optimization method based on deep reinforcement learning. Under the premise of ensuring the task delay requirements, it can effectively accommodate more concurrent tasks, thereby maximizing resource utilization and reducing computing time consumption, effectively improving the accuracy and real-time performance of resource allocation. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 This is a flowchart of the resource intelligent joint optimization algorithm based on deep reinforcement learning;

[0046] Figure 2 It is a schematic diagram of the architecture of the communication network resource joint allocation system;

[0047] Figure 3 It is a flowchart of randomly generated task flows;

[0048] Figure 4 The meter chart is delivered using the OpenFlow protocol;

[0049] Figure 5 This is a comparison chart of the computational time required for each method;

[0050] Figure 6 The following is a comparison chart of the success rates of various methods. DETAILED DESCRIPTION

[0051] The accompanying drawings, which constitute a part of the present invention, are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute improper limitations on the present invention.

[0052] like Figure 1 As shown, the present invention provides a resource intelligent joint optimization method based on deep reinforcement learning, comprising the following steps:

[0053] Step 1: Build the communication network resource allocation system architecture, construct the communication network topology, use SDN technology to detect the data traffic in the network topology, record the link bandwidth and the usage of computing resources of each communication node for the Dual Duel Deep Q Network (D3QN) algorithm;

[0054] Step 2: Randomly generate task flow T in the network topology to simulate task parsing and sending;

[0055] Step 3: Use the path-finding algorithm to find the n shortest paths from node to node;

[0056] Step 4: Based on the n shortest paths found, determine whether the available computing resources in the path are not empty, and whether the available communication resources can meet the minimum transmission rate requirements of the task. If not, discard the path and determine the path in order. Find the candidate path. If not found, the task transmission is considered to have failed;

[0057] Step 5: The intelligent agent selects the best computing resource allocation scheme and the most suitable communication resource allocation scheme in the current network resource environment;

[0058] Step 6: Use the meter table in OpenFlow to send the communication resources allocated to each link of the optimal path to the switch to implement communication resource allocation.

[0059] In step 1, the present invention first constructs a communication network resource joint allocation system architecture.

[0060] Communication network resource joint allocation system architecture Figure 2 As shown. The core function of the system is to obtain the data flow in the network topology in real time through the data acquisition module, and calculate the available bandwidth of each link. This available bandwidth information is stored in the controller in the form of a graph. At the same time, the controller also stores the computing resource data of all nodes, and can dynamically adjust resource allocation according to task changes. The data acquisition module regularly sends commands to the switch through the SDN controller to obtain statistical information on the switch port traffic, thereby ensuring that the system continuously monitors the network status and provides an accurate traffic basis. After obtaining the traffic statistics information, the computing storage module is responsible for calculating the remaining available bandwidth of the link, taking into account the current traffic and past trends to ensure the effectiveness of resource allocation. Two formulas are required for calculation:

[0061] Flow rate formula: speed = (s(t1)-s(t0)) / (t1-t0).

[0062] Where: s(t1) represents the flow at time t1, and s(t0) represents the flow at time t0.

[0063] Remaining bandwidth formula: free_bw = capability – speed.

[0064] Where: capability represents the total link bandwidth.

[0065] In step 2, two different hosts are randomly selected, and one of the three set task types is selected as the generated task to be sent through Iperf. Iperf is a network performance testing tool that can test the maximum TCP and UDP bandwidth performance. Iperf is implemented based on the Server-Client mode. When measuring network parameters, Iperf distinguishes between the two roles of listener and speaker. The speaker sends a certain amount of data to the listener, and the listener counts and records the speaker's bandwidth usage, delay jitter and other parameters. After the speaker's data is all sent, the listener will send a data packet to the speaker to inform the speaker of the measurement data.

[0066] The flowchart for step 2 is as follows Figure 3 As shown, specifically:

[0067] (1) Randomly generate a client host IP as the speaker in Iperf, responsible for sending data to the listener;

[0068] (2) A random server-side host IP is generated as the listener in Iperf, which is responsible for counting the data sent by the listener and recording parameters such as bandwidth, delay jitter, etc.

[0069] (3) Determine whether the host IP generated in step (1) and step (2) are the same. If they are the same, go to step (2); otherwise, go to step (4);

[0070] (4) Randomly select one of multiple types of tasks as the data parameter to be sent;

[0071] (5) Determine whether the total amount of generated tasks meets the conditions. If so, end the process; otherwise, go to step (1).

[0072] Examples of the generated task types are shown in Table 1 below:

[0073] Table 1 Task Type Calculate the total amount of data for the task Minimum required transfer rate Maximum end-to-end delay 1-1 4Mbits 4Mbits 400ms 1-2 2Mbits 4Mbits 400ms 1-3 1Mbits 4Mbits 400ms 2-1 4Mbits 2Mbits 25ms 2-2 2Mbits 2Mbits 25ms 2-3 1Mbits 2Mbits 25ms 3-1 4Mbits 1Mbits 15ms 3-2 2Mbits 1Mbits 15ms 3-3 1Mbits 1Mbits 15ms

[0074] In step 3, after the switch receives the data packet and parses the packet header, it queries its own flow table for the corresponding flow rule for the data packet. By matching the flow rule corresponding to the data packet, it mainly compares the IP addresses of the source host and the destination host, as well as the computing resources and the bandwidth and delay parameters of the transmission. If the match is successful, the data packet is directly forwarded to the corresponding port set in the flow rule; if the match fails, a Packet-in event is generated according to the corresponding packet, and the Packet-in data packet is sent to the controller. After receiving the Packet-in data packet, the controller uses the breadth-first algorithm to find n simple paths from end to end of the task.

[0075] In step 4, the communication network resources are jointly optimized in the environment, and the network topology can be stored as a graph collection ,in represents the edge set, O represents the total number of edges, represents the Oth edge, represents the vertex set, n represents the total number of nodes, represents the nth node, and the task set is represented as , X represents the total number of tasks; since resource joint optimization needs to consider both the communication resources and computing resources required by the task, each task can be represented by a triple as =( , , ),in For the task The total amount of data that needs to be calculated, For the task The minimum available transfer rate required, For the task The maximum tolerable end-to-end delay.

[0076] Shannon's formula is used to simplify the bandwidth and power resources in the communication network. The mathematical problem modeling is shown in the following formula:

[0077] ;

[0078] in, For Node and The link between the tasks The allocated transmission power, For Node and The link between the tasks allocated bandwidth; For Node and The link between the tasks The allocated transfer rate; To represent the background noise of the channel, the above formula can simplify the bandwidth resources and power resources in the communication network from a two-dimensional problem to a one-dimensional transmission rate problem.

[0079] In a communication network, all tasks must go through a series of computational analysis before being sent to a designated node. This process is not only an important step to ensure that the task can be executed correctly, but also causes the consumption of computing resources and time delays.

[0080] In order to achieve efficient task processing, it is necessary to reasonably allocate computing resources. In the case of multi-task parallel processing, it is necessary to ensure that as many tasks as possible can be accepted and processed. In addition, the delay generated by each task must be controlled within an acceptable range to minimize the impact on the delay of the entire task link. This is crucial for the real-time and reliability of the communication network. The present invention models the delay generated by the analysis of computing tasks, as shown in the formula:

[0081] ;

[0082] in, For Node For the task Allocated computing resources, For Node of Number of cycles, For the task The total amount of data that needs to be calculated; For computing tasks The delay caused by parsing.

[0083] The above two formulas can be used to obtain the constraints for selecting the path:

[0084] ;

[0085] in, For Node and The available transmission rate of the link between, nodes i and j are the nodes at both ends of the link; For Node available computing resources; i=1,2,...,n; j=1,2,...,n. and Guaranteed Node and The transmission rate of the link between them cannot be less than the minimum transmission rate required by the task, nor can it exceed the available transmission rate of the link. and Guaranteed Node The available computing resources allocated to the task cannot be zero, nor can they exceed the maximum available computing resources of the current node.

[0086] In step 5, assume that the variable bit data stream in the packet switching passes through a shaper composed of leaky buckets, and the shaped data stream obeys the arrival curve , also known as traffic specification-service flow T-SPEC, where h represents time, is the length of the largest data packet in the vulnerability, is the peak rate, is the sustainable rate, is the burst tolerance. In the terminology of the integrated services Internet, this quadruple It is a traffic characteristic specification adopted by the Internet Engineering Task Force. It is widely used to illustrate the business traffic characteristics in integrated service networks. The definition of the leaky bucket controller obtained based on network calculation is as follows:

[0087] Definition 1 The leaky bucket controller is used to define a standard model that matches the traffic in the integrated service network. It is a controller that analyzes the flow data in the following way: Assume there is a device with a capacity of The fluid bucket (pool) is initially empty. There is a small hole at the bottom of the bucket. When the bucket is not empty, the fluid flows at a rate of Units of leakage. The data in the leaky bucket must be filled with a fluid equal to the amount of data. Data that causes the leaky bucket to overflow is called non-compliant data, otherwise it is called compliant data. The fluid in the leaky bucket model does not represent data, but it uses the same unit of measurement as data. A flow constrained by a leaky bucket satisfies T-SPEC , when passing through a node that provides a rate-delay service curve, the delay bound of the flow is as follows:

[0088] ;

[0089] In the formula, Indicates the service rate provided by the node to the data flow, Indicates the delay in the system. According to the comprehensive service model of Internet routers, the service curve provided by the router that guarantees the rate to the flow is the rate-delay function. and delay The following relationship exists as shown below:

[0090] ;

[0091] in, is the maximum packet length of the stream, is the maximum packet length of all flows in the router, is the total rate of the scheduler.

[0092] Get the end-to-end delay bound As shown below:

[0093] ;

[0094] in, Indicates that if , ,otherwise .

[0095] Therefore, the task is obtained through the above formula The end-to-end delay bound As shown below:

[0096] ;

[0097] in, Representation Node Send to node The rate of Indicates that all links are tasks The minimum assigned transmission rate;

[0098] In a communication network environment, tasks usually include data parsing and information transmission. Parsing tasks require a certain amount of computing resources, while sending tasks have specific requirements for communication resources. The two are interdependent and have far-reaching impacts. Therefore, when optimizing resources, it is necessary to comprehensively evaluate the needs of both aspects to achieve the optimization of overall performance. Since the delays in the parsing and sending processes will overlap, the response time of the final task will increase, so the total delay It has become the core goal of the optimization of the present invention, as shown in the following formula:

[0099] .

[0100] The SDN controller is deployed as an agent to globally control the resources in the entire communication network and dynamically formulate resource allocation plans for each task. The command node generates some tasks for the current environmental conditions. The information required for these tasks is sent to the agent as state information s. The agent allocates computing and communication resources based on the state information. The allocation plan is a continuous process with consistency. First, Y action strategies are used for the allocation of computing resources; in this embodiment, Y=10, and the Y action strategies are 10 discrete computing resource values; then two action strategies are used for the allocation of communication resources, namely the non-uniform bandwidth reservation algorithm and the uniform bandwidth reservation algorithm, and a continuous communication resource allocation plan is obtained by the selected algorithm. The agent returns the plan generated by the action strategy to the node; the node will return a reward r to the agent. In order to evaluate the pros and cons of the current action strategy, the agent interacts continuously with the environmental state, and the optimal resource allocation strategy is obtained by comparing the accumulated rewards between different strategies and returning it to the node.

[0101] The settings of the state space, action space, and reward function in the D3QN algorithm are as follows:

[0102] (1) State space: In the definition of state space, the agent needs to select the corresponding action strategy based on the acquired environmental state information, so it is necessary to comprehensively consider the two parts of the environmental state. The first part is the characteristics of the task, that is, the agent needs to obtain the task set , the total amount of data calculated by each task , the minimum required transmission rate sequence and the maximum end-to-end delay allowed by the task ,in Indicates the task The second part is the current environment information of the communication network, that is, At this moment, all nodes in the entire network have available computing resources and the available transmission rate in the link .

[0103] The state space is set up as follows:

[0104] ;

[0105] Action space: In the definition of action space, the agent will have its own action set. Whenever a task is generated, the agent will judge according to the current environment state and select a corresponding action strategy. After executing the corresponding action strategy, it will enter a new environment state and obtain the reward of the feedback of the corresponding action strategy. In order to improve the convergence speed and obtain a better action strategy faster, this paper introduces a greedy learning strategy, using the greedy factor . When the random number value between 0 and 1 is less than When , a value will be randomly selected from the action space. When it is greater than or equal to The invention aims to jointly optimize the allocation of computing resources and communication resources. Therefore, there are two parts of value schemes in the action space. The first part is Y discrete values ​​of the number of CPU cycles of computing resources. The intelligent agent takes the action of the current task from Select a value in The first part is the preset parameters, which are used as the action strategy of the computing resources of the current task; the second part is two value schemes for communication resources ,in Representatives use non-uniform bandwidth allocation algorithm to solve, The representative uses the uniform distribution algorithm to solve.

[0106] The action space is set up as follows:

[0107]

[0108] Reward function: The reward function is the feedback signal that the agent obtains from the environment after selecting and executing a certain action strategy. This signal is used to evaluate the effectiveness of the strategy and the correctness of the execution. Therefore, the design of the reward function must be closely related to the optimization goal. The proposed optimization goal needs to be able to meet limited constraints and ensure that all related tasks can be completed within the tolerable delay constraint, while accommodating as many concurrent tasks as possible. In order to achieve this optimization goal, the reward function should reflect several key factors. First, the timeliness of the task is very important. Therefore, if the task can be completed within its maximum tolerable delay If the task is completed within the time limit, a positive reward should be provided to the agent. On the contrary, if the task is not completed within the time limit, a negative penalty should be imposed to encourage the agent to seek more efficient decisions.

[0109] The reward function is set as shown below. This design concept not only enables the agent to make effective decisions in a dynamic environment, but also continuously optimizes the resource allocation strategy to meet the changing task requirements and network status. Through a reasonable reward mechanism, the agent will be more motivated to explore the optimal action strategy, thereby improving the overall system performance.

[0110] .

[0111] In step 6, if Figure 4 As shown, the communication resources allocated to each link of the optimal path are sent to the switch using the meter table in OpenFlow, and the Drop attribute is selected in the action to implement the allocation of communication resources. The specific flow table is shown in Table 2 below, and the meter table is shown in Table 3 below: Table 2 Source Address 10.0.0.1 Destination Address 10.0.0.4 Ingress Port 1 instruction Instrument serial number: 1, output port: 4 Table 3 Instrument serial number 1 type throw away rate 2000 bits per second

[0112] Figure 5 This is a bar chart comparing the time consumption of various methods. This embodiment uses the deep deterministic policy gradient algorithm (DDPG), the double deep Q network algorithm (DDQN) and the multi-agent deep reinforcement learning algorithm (MADRL) as the comparison method of this embodiment; the horizontal axis represents the number of nodes in different topologies, and the vertical axis represents the time spent on the calculation of different algorithms, in ms. Different network topologies send the same number of task flows. Each time the controller uses the agent to solve the optimal computing resource and communication resource allocation scheme, it records the time spent. Finally, the time spent by different methods is averaged to obtain the graph. It can be seen from the figure that the algorithm proposed by the present invention always has a lower computing time overhead, because the present invention can select the optimal computing resource and communication resource allocation scheme more quickly.

[0113] Figure 6 The bar chart compares the number of successful task allocations in various network topologies using different methods. The horizontal axis represents the number of nodes in different topologies, and the vertical axis represents the number of successful task allocations by the algorithm. Different network topologies send the same number of task streams, and the controller records the required resources for each successful task allocation as 1. Finally, the total number of successful allocations for different network topologies is averaged to obtain this graph. It can be seen from the figure that the number of successful tasks of all algorithms is proportional to the size of the topological nodes. This is because as the size of the topological nodes increases, the number of nodes inside increases, and the corresponding number of links also increases. As a result, the number of options for allocating computing resources and communication resources also increases. Figure 6 It can be seen that in different topological networks, the D3QN algorithm proposed in this embodiment is always the best compared with the other three algorithms, so it has better resource utilization and better communication service quality. Compared with the DDQN algorithm, which is also deep reinforcement learning, D3QN introduces value functions and action value functions to make the learning of the agent more stable, so that the Q value of the evaluation network can be closer to the Q value calculated by the target network, so that a better strategy can be solved.

[0114] In summary, this embodiment analyzes the advantages and disadvantages of several resource joint allocation algorithms that are currently used from the perspective of intelligent resource allocation in communication networks. On this basis, a resource intelligent joint optimization method based on deep reinforcement learning is proposed, which solves some shortcomings of the current resource joint allocation algorithm to a certain extent. Under the premise of ensuring the task delay requirements, it effectively accommodates more concurrent tasks, thereby maximizing resource utilization and the algorithm has a lower computing time overhead, effectively improving the accuracy and real-time performance of resource allocation. Therefore, the resource intelligent joint optimization method based on deep reinforcement learning proposed in the present invention is feasible and effective.

[0115] It should also be noted that the various specific technical features described in the above specific embodiments can be combined in any suitable manner without contradiction. In order to avoid unnecessary repetition, the present invention will not further describe various possible combinations.

Claims

1. A resource intelligent joint optimization method based on deep reinforcement learning, characterized in that: The method comprises the following steps: Step 1: Build a communication network topology, detect the data traffic in the network topology, record the link bandwidth, remaining bandwidth, and usage of computing resources of each communication node; Step 2: Randomly generate task flow T and simulate task parsing and sending in the network topology; Step 3: The switch receives the task in the task flow After parsing the data packet header, check whether there is a corresponding flow rule for the data packet in its own flow table. If there is a corresponding flow rule, forward the task directly to the corresponding port set in the flow rule, otherwise find n simple paths for the task; Step 4: Task-based Minimum required transfer rate and nodes and The link between the tasks Assigned transfer rate , set constraints to select candidate paths from n simple paths; nodes i and j are nodes at both ends of the link, i=1,2,...,n; j=1,2,...,n, n represents the total number of nodes; Step 5: With the goal of minimizing the total delay, select the optimal path from the candidate paths and allocate optimal computing resources and communication resources to the optimal path; Step 6: Send the communication resources allocated to each link of the optimal path to the switch to implement communication resource allocation.

2. According to claim 1, a resource intelligent joint optimization method based on deep reinforcement learning is characterized in that: The calculation formula for the remaining bandwidth free_bw in step 1 is as follows: free_bw = capability – speed; Among them, capability represents the total bandwidth of the link, and speed represents the flow rate.

3. The resource intelligent joint optimization method based on deep reinforcement learning according to claim 1 is characterized in that: Step 3 uses the breadth-first algorithm to find n simple paths.

4. According to claim 1, a resource intelligent joint optimization method based on deep reinforcement learning is characterized in that: The constraints in step 4 are: ； in, For Node and The available transmission rate of the link between For Node For the task Allocated computing resources, For Node Available computing resources, The expression is: ； in, For Node and The link between the tasks The allocated bandwidth, For Node and The link between the tasks the allocated transmission power; Represents the background noise of the channel.

5. The method for intelligent joint optimization of resources based on deep reinforcement learning according to claim 1, characterized in that: Step 5 uses the D3QN algorithm to select the optimal path from the candidate paths and allocates optimal computer resources and communication resources to the optimal path, specifically: Set the objective function: Set a leaky bucket controller with a capacity of b and a flow rate of b per second. The peak velocity of the fluid is , using leaky bucket controller to calculate the task The end-to-end delay bound : ； in, For the task The size of the largest packet in; If ,but ,otherwise , Indicates the length of the largest data packet in all task flows in the router. Indicates the length of the maximum data packet in the leaky bucket controller; Representation Node Send to node rate; Indicates that all links are tasks The minimum assigned transmission rate; Task-based The end-to-end delay bound Calculate the total delay of task transmission and take the minimum total delay as the objective function: ； in, is the total delay of task transmission, For analysis tasks The delay generated, The expression is: ； in, For Node For the task Allocated computing resources, For Node of Number of cycles, For the task The total amount of data that needs to be calculated; Setting up the state space for: ； Among them, h represents the current moment, M represents the computational data sequence of the task, and M ,in Representation Task The amount of computing data, X represents the total number of tasks, represents the minimum required transmission rate sequence of the task, , represents the maximum end-to-end delay sequence allowed by the task, , Representation Task The maximum allowed end-to-end delay; Represents the sequence of available computing resources of the node at the current moment, , Indicates the available computing resources of the i-th node at the current moment; Represents a sequence of available transmission rates, , For Node and The available transmission rate of the link between Setting up the action space : ； Agents on discrete numerical sets Select a value from the list and assign it to , Y represents the total number of discrete values, Indicates the preset parameters. Indicates the allocation of communication resources. It represents the use of non-uniform bandwidth allocation algorithm to solve communication resources. The representative adopts the uniform allocation algorithm to solve the communication resources; Set up the reward function: 。 6. The method for intelligent joint optimization of resources based on deep reinforcement learning according to claim 5, characterized in that: The agent uses a greedy learning strategy to select a value from a discrete value set and assign it to .

7. A computer device comprising a memory, a processor, and a computer program stored in the memory and capable of running on the processor, characterized in that: When the processor executes the computer program, it implements a resource intelligent joint optimization method based on deep reinforcement learning as described in any one of claims 1 to 6.

8. A computer-readable storage medium for storing a program, characterized in that: Execute the program to implement a resource intelligent joint optimization method based on deep reinforcement learning as described in any one of claims 1-6.

Images