Industrial internet-oriented network slice resource allocation method and related equipment

Abstract

The invention provides a network slice resource allocation method oriented to an industrial internet and related equipment. Comprising the following steps: designing an SDN-based network slice resource allocation management and control architecture which comprises a control plane and a data plane; an intelligent agent is deployed for each switch node in the control plane, and each switch node is provided with an IT slice queue and an OT slice queue; designing a path planning and network slice resource allocation algorithm; modeling joint path planning and network slice resource allocation as a multi-agent Markov decision process; defining a state space and an action space; designing a reward function; constructing and training a multi-agent deep reinforcement learning model; and inputting a network global view collected in the SDN-based network slice resource allocation management and control architecture and demand information of each slice into the deep reinforcement learning agent obtained by training, and outputting a traffic forwarding path and a network slice resource allocation scheme. And the bearable flow quantity of the network can be improved to the maximum extent.

Description

Technical Field

[0001] The present application relates to the field of network communication and machine learning technology, and in particular to a network slicing resource allocation method and related equipment for the Industrial Internet. Background Art

[0002] The core issue of network slicing is the need to provide high-quality differentiated services for traffic with different demands, that is, the path planning and resource allocation problem in the physical network. At present, the research and standardization activities of network slice resource allocation with the goal of minimizing traffic transmission delay attempt to use the IEEE TSN (Time-Sensitive Network) protocol as a means to achieve deterministic communication in the transmission network. However, existing studies rarely consider the impact of TSN key technologies such as CBS traffic shapers on network slice resource allocation, and only consider the transmission delay of traffic in the link. In addition, in the research on using deep reinforcement learning models to solve the slice resource allocation problem, traffic is forwarded according to a fixed strategy, and fixed forwarding strategies are likely to cause network congestion, further leading to the degradation of network services and network paralysis. Summary of the invention

[0003] In view of this, the purpose of this application is to propose a network slicing resource allocation method and related equipment for the Industrial Internet.

[0004] Based on the above purpose, the present application provides a network slice resource allocation method for the industrial Internet, which may include:

[0005] Design an SDN-based network slice resource allocation management and control architecture, including a control plane and a data plane; wherein the data plane is configured to provide a network environment and deploy resource allocation and routing policies issued by the controller; the control plane is configured to calculate traffic forwarding paths and slice resource allocation policies, and perform traffic scheduling and resource allocation; in the control plane, an intelligent agent is deployed for each switch node, and each switch node has an IT slice queue and an OT slice queue;

[0006] Design path planning and network slice resource allocation algorithms;

[0007] Model joint path planning and network slice resource allocation as a multi-agent Markov decision process; define state space and action space; design a reward function; wherein the state space includes the global state space of the environment and the partial observation space of the agent; the action space includes the joint action space of all agents;

[0008] Build and train multi-agent deep reinforcement learning models;

[0009] The global network view and demand information of each slice collected in the SDN-based network slice resource allocation management and control architecture are input into the trained deep reinforcement learning agent to output the traffic forwarding path and network slice resource allocation plan.

[0010] In some embodiments, the control plane includes a controller, a central controller and each slice controller; wherein the controller is used to monitor the network status of each slice; the central controller is used to centrally control and monitor network devices, perform scheduling and resource allocation, and delegate slice resource adjustment strategies to each slice controller;

[0011] The data plane includes industrial Internet terminal equipment, switches and full-duplex physical links; the industrial Internet terminal equipment is used to generate data traffic, collect data at a set frequency and forward the data traffic through the switch; the switch egress port adopts the TSN / CBS architecture, with IT slice queues and OT slice queues; the priority of the OT slice is higher than the priority of the IT slice.

[0012] In some embodiments, the design of path planning and network slice resource allocation algorithm includes:

[0013] The network model includes an undirected graph G = (V, E); where V is a node set, V = {V1, V2, ..., v V}; E is the link set, E = {e1, e2, ..., e E}; the link is a bidirectional link e = (u, v) ∈ E; u, v ∈ V; is the link from node u to node v; edge e i The weight W i Link e i The total link bandwidth of M network slices is the aggregate bandwidth of the shared network;

[0014] The traffic entering the network includes the source node of the traffic, the destination node of the traffic, the bandwidth requirement and the maximum end-to-end delay requirement; where F is the set of all traffic in the network; f = {v s , V d , f b , f d , size, f q},v s , v d ∈V;f b is the required bandwidth of flow f, f d is the maximum tolerable end-to-end delay of flow f; size is the size of the flow f frame, fq is the queue to which the flow belongs; the forwarding path of the flow v1 is the network node included in the forwarding path; p f The bandwidth reserved for the corresponding queues along the road in, Reserve bandwidth for node v1;

[0015] The queue corresponding to the slice is q = {prio, load, credit, loCredit, idleSlope}, where prio is the priority of the queue; load is the load of the queue; credit is the credit of the queue; loCredit is the minimum credit of the queue; idleSlope is the rate of credit accumulation; the reserved bandwidth ratio is R; R = idleSlope / C, where C is the total bandwidth; sendSlope = idleSlope-C; sendSlope is the rate of credit consumption;

[0016] The end-to-end transmission delay of the traffic meets the maximum end-to-end delay constraint: in, is the propagation delay; is the transmission delay; fi is the flow rate; Size is the byte value corresponding to the data frame in the flow rate; is the queuing delay; It is the waiting delay in the process of credit recovering from the maximum frame size loCredit to 0 at the idleSlope rate; is the maximum end-to-end delay constraint; hop i +1 is the flow rate f i The number of hops experienced on the corresponding path + 1; hop i is the flow rate f i The number of hops experienced on the corresponding path; To process delay.

[0017] In some embodiments, the global state space of the environment includes S t = {N t , M t , F t}; Put the arriving traffic into the corresponding slice queue according to the type, S t is the environmental state observed at time t; N t is network resource information; t is the slice information; F t Request information for traffic; F t ={f,δ t}, f is the flow rate, δ t is the accumulated delay of the traffic;

[0018] The partial observation space of the agent includes O={o1, o2, ..., o n}; where on is part of the observation space of agent n;

[0019] The joint action space of all agents consists of A = {a1, a2, ..., a n}; where a n is the independent action of agent n; including A=A n ×A b Among them, A n Select the next hop for the traffic; A b A is the allocation ratio of the slice bandwidth of the node where the traffic is located; b =p1,p2,...,p m ; A b is another part of the action space of each agent; each p j is the bandwidth percentage allocated to the jth slice by the node where the traffic is located at time t, and satisfies m is the type of slice.

[0020] In some embodiments, the reward function is R = αR throughput +βR utilization -γR delay -δR loop ; Among them, R throughput is the network throughput; To maximize the amount of dispatchable traffic; R utilization is the network resource utilization; Where ρ(e) is the bandwidth utilization on the link; R delay is a penalty for traffic scheduling failure caused by loops; R loop is the penalty for traffic scheduling failure due to violation of deadline satisfaction criteria; R loop =-C loop ×|| loop ,|| loop ∈[0,1];C loop and || loop are weight parameters respectively; α, β, γ, δ are weight parameters respectively.

[0021] In some embodiments, constructing and training the multi-agent deep reinforcement learning model includes:

[0022] Create a network topology model; each node has an IT slice queue and an OT slice queue;

[0023] Initialize iteration variables, determine the maximum number of iterations, and set the length of the experience replay pool;

[0024] Create an agent object instance and initialize network parameters;

[0025] Randomly generate traffic entering the network;

[0026] Set the initial global state space;

[0027] Input the initial global state space to the multi-agent deep reinforcement learning model, and each agent Actor network generates actions separately;

[0028] Recombining the actions generated by each Actor network into joint actions to be applied to the network as traffic forwarding paths and network slice resource allocation schemes;

[0029] Generate traffic requests, obtain the number of successfully scheduled traffic in the network, network resource utilization, traffic transmission delay and queuing delay at each node, calculate network throughput, resource utilization, end-to-end delay and path constraints, and calculate rewards;

[0030] Storing the initial global state space, reward and next global state space in the experience replay pool for iterative learning of the multi-agent deep reinforcement learning model;

[0031] Repeat the steps of randomly generating traffic entering the network to storing the initial global state space, reward and next global state space in the experience replay pool until the length of the experience replay pool reaches a threshold;

[0032] According to the strategy update ratio, the strategy of each agent Actor network is updated, and the parameters of the agent Actor network are updated;

[0033] Update the parameters of the Critic network shared by all agents based on the updated parameters of the Actor network;

[0034] Determine the number of iterations; in response to determining that the number of iterations has not reached the maximum number of iterations, return to the step of randomly generating traffic entering the network until the maximum number of iterations is reached here.

[0035] In some embodiments, the loss function of each agent Actor network is: Among them, G is the set of agents g; S is the state space; is the generalized advantage function; k gt (θ) is the importance weight; σ and v are constants; θ is the parameter of the agent Actor network, o gt is the observation value of agent g at time step t;

[0036] The loss function of the Critic network shared by all agents is: Among them, φ is the parameter of the Critic network shared by all agents; is the state value; R gt is the cumulative discount income; C φ The clipping function.

[0037] An embodiment of the present application also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements any of the methods described above when executing the program.

[0038] An embodiment of the present application also provides a non-transitory computer-readable storage medium, which stores computer instructions, and the computer instructions are used to enable a computer to execute any of the methods described above.

[0039] An embodiment of the present application further provides a computer program product, comprising computer program instructions, which, when executed on a computer, enable the computer to execute any of the methods described above.

[0040] From the above, it can be seen that the slicing resource method for the industrial Internet based on deep reinforcement learning provided in this application is oriented to the industrial Internet scenario, uses the SDN architecture for global control, adopts a deep reinforcement learning model, takes the global network state as the input of the evaluation network, and takes the state value as the output of the evaluation network. The algorithm takes maximizing the expected reward as the optimization goal and continuously updates the network; by selecting the optimal forwarding node and the optimal slice resource allocation ratio to specify the forwarding of traffic and the resource adjustment of each slice, combined with the CBS mechanism, the traffic forwarding path and resource allocation strategy are calculated dynamically in real time, which can maximize the network's ability to carry traffic while ensuring the deterministic transmission of the industrial Internet. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] In order to more clearly illustrate the technical solutions in the present application or related technologies, the drawings required for use in the embodiments or related technical descriptions are briefly introduced below. Obviously, the drawings described below are only embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0042] Figure 1 A flowchart of a network slice resource allocation method for the industrial Internet according to an embodiment of the present application;

[0043] Figure 2 A schematic diagram of an SDN-based network architecture according to an embodiment of the present application;

[0044] Figure 3 This is a structural diagram of the industrial Internet slice resource allocation algorithm based on MAPPO in an embodiment of the present application;

[0045] Figure 4 This is a flow chart of a network slice resource allocation algorithm based on MAPPO in an embodiment of the present application;

[0046] Figure 5 A schematic diagram of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0047] In order to make the objectives, technical solutions and advantages of the present application more clearly understood, the present application is further described in detail below in combination with specific embodiments and with reference to the accompanying drawings.

[0048] It should be noted that, unless otherwise defined, the technical terms or scientific terms used in the embodiments of the present application should be understood by people with ordinary skills in the field to which the present application belongs. The "first", "second" and similar words used in the embodiments of the present application do not represent any order, quantity or importance, but are only used to distinguish different components. "Including" or "comprising" and similar words mean that the elements or objects appearing in front of the word cover the elements or objects listed after the word and their equivalents, without excluding other elements or objects. "Connect" or "connected" and similar words are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect.

[0049] The intranet of the industrial Internet factory is divided into industrial control network (OT) and industrial information network (IT). The different requirements of the two lead to mixed transmission of multiple business flows in the network, which has the problem of mutual interference and makes it difficult to guarantee the transmission requirements of specific traffic. OT has put forward more stringent requirements on network performance, such as bounded latency and jitter. The CBS traffic shaper in TSN provides deterministic transmission guarantee for OT network through bandwidth reservation. The development of network function virtualization technology (NFV), Internet of Things and software-defined network has made intelligent services a reality. The "one-size-fits-all" design concept of traditional networks cannot adapt to emerging intelligent services, and network slicing technology has emerged. Network slicing can divide the same physical network into multiple independent virtual logical networks according to different business requirements such as latency, throughput, security, etc. to provide differentiated services and effectively utilize public infrastructure.

[0050] Among them, TSN is one of the key technologies of deterministic networks. It can use technologies such as clock synchronization, bandwidth reservation and traffic shaping to provide real-time, high-quality and highly reliable data transmission for the industrial Internet. With the advent of the Industrial 4.0 era, industrial production control systems are becoming more and more intelligent, which puts forward higher requirements for the real-time and deterministic transmission of information, requiring lower end-to-end latency and extremely low latency jitter. The "best effort" architecture of traditional Ethernet can no longer meet the requirements of deterministic transmission quality, which has led to the emergence of proprietary network protocols such as Profinet in OT, hindering the open flow of information to IT. Therefore, the time-sensitive network (TSN) technology that can achieve microsecond-level deterministic services was born.

[0051] Among them, in order to achieve deterministic transmission services in the network, the IEEE 802.1Qav standard, also known as the forwarding and queuing enhancements for time-sensitive streams (FQTSS), aims to solve the bandwidth and latency challenges faced by time-sensitive data in network transmission. This standard pays special attention to ensuring that the quality of service of time-sensitive streams is effectively guaranteed in a network environment where ordinary asynchronous Ethernet data streams and time-sensitive data streams coexist. To achieve this goal, IEEE 802.1Qav introduces the CBS algorithm, which provides deterministic guarantees by reserving the maximum required bandwidth for traffic and performing fine queue scheduling for different types of traffic transmission.

[0052] Existing research rarely considers the impact of TSN key technologies such as CBS traffic shapers on network slice resource allocation. It only considers the transmission delay of traffic in the link, ignoring the impact of queuing delay at the node on the deterministic transmission of traffic. TSN reserves bandwidth to meet deterministic delay and jitter, but the current protocol standards and research papers do not clearly specify the calculation method of bandwidth reservation value. Most of them use empirical values. This approach lacks flexibility in scheduling traffic with different latency requirements. Reserving too little bandwidth will cause traffic transmission to fail to meet latency requirements and cause packet loss. Reserving too much bandwidth and occupying it for a period of time will lead to resource waste. In the research on using deep reinforcement learning models to solve the problem of slice resource allocation, traffic is forwarded according to a fixed strategy, and fixed forwarding strategies are likely to cause network congestion, further leading to the degradation of network services and network paralysis.

[0053] Therefore, in the existing technology, the slice resource allocation method based on latency constraints rarely considers the characteristics of the TSN key technology CBS traffic shaper, only considers the transmission latency of the traffic, and most of them forward the data stream with a fixed routing strategy, which is likely to cause network congestion. The research related to the DRL-based CBS mechanism focuses on solving the traffic scheduling problem in a single TSN device under predefined constraints, which is difficult to adapt to the dynamic network environment. The heuristic algorithm used is prone to fall into the local optimum and it is difficult to ensure the global optimum.

[0054] Based on this, the embodiment of the present application provides a slice resource allocation method for the industrial Internet. For the industrial Internet factory intranet business scenario, path planning and slice resource allocation decisions are made based on the real-time network status and the traffic demand of each slice. The multi-agent deep reinforcement learning algorithm is combined with the TSN CBS traffic shaper mechanism to send the optimal path of the traffic and the optimal resource allocation plan of the slice to the data layer to guide forwarding, thereby ensuring the deterministic service of the OT slice while improving the throughput of the IT slice to maximize the utilization of network resources. It is more adaptable to the dynamic network environment and can respond to dynamic traffic in a timely manner. In the SDN-based network architecture, each controller can collect the global view of the network and the demand information of each slice and input it into the deep reinforcement learning agent. The agent calculates the traffic transmission path and resource allocation plan based on the state information, which is sent by the controller to the forwarding plane, thereby achieving the differentiated service goal. It can guarantee the deterministic transmission requirements of the OT slice to a certain extent, that is, bounded jitter and end-to-end delay, and at the same time can improve the throughput of the IT slice and maximize the overall resource utilization of the network.

[0055] like Figure 1 As shown, it is a flow chart of a network slice resource allocation method for the industrial Internet provided by an embodiment of the present invention. The network slice resource allocation method for the industrial Internet may include:

[0056] S100, design a network slice resource allocation management and control architecture based on SDN, including a control plane and a data plane; wherein the data plane is configured to provide a network environment and deploy resource allocation and routing policies issued by the controller; the control plane is configured to calculate traffic forwarding paths and slice resource allocation policies, and perform traffic scheduling and resource allocation; in the control plane, an intelligent agent is deployed for each switch node, and each switch node has an IT slice queue and an OT slice queue;

[0057] S200, design path planning and network slice resource allocation algorithms;

[0058] S300, modeling joint path planning and network slice resource allocation as a multi-agent Markov decision process; defining a state space and an action space; and designing a reward function; wherein the state space includes a global state space of the environment and a partial observation space of the agent; and the action space includes a joint action space of all agents;

[0059] S400, builds and trains multi-agent deep reinforcement learning models;

[0060] S500, input the network global view and the demand information of each slice collected in the SDN-based network slice resource allocation management and control architecture into the trained deep reinforcement learning agent, and output the traffic forwarding path and network slice resource allocation plan.

[0061] The slicing resource method for the industrial Internet based on deep reinforcement learning provided in the embodiment of the present application is oriented to the industrial Internet scenario, uses the SDN architecture for global control, adopts a deep reinforcement learning model, takes the global network state as the input for evaluating the network, and takes the state value as the output for evaluating the network. The algorithm optimizes the network with maximizing the expected reward as the optimization goal, and continuously updates the network; by selecting the optimal forwarding node and the optimal slice resource allocation ratio to specify the forwarding of traffic and the resource adjustment of each slice, combined with the CBS mechanism, the traffic forwarding path and resource allocation strategy are dynamically calculated in real time, which can maximize the network's ability to carry traffic while ensuring the deterministic transmission of the industrial Internet.

[0062] In some embodiments, in step S100, Figure 2 As shown, the SDN-based network slice resource allocation management and control architecture incorporates DRL (Deep Reinforcement Learning) into its path planning and slice resource allocation mechanism, so that a relationship can be established between network resources and service requirements. Among them, Software Defined Network (SDN) is a new type of network architecture. Its basic idea is to completely decouple the control plane and data plane of the traditional distributed network, and use a logically centralized controller to control the entire distributed data plane. Reinforcement learning is a unique branch in machine learning, which can be parallel to supervised learning and unsupervised learning, and can together constitute the three major paradigms of machine learning. Deep reinforcement learning is a combination of deep neural networks and reinforcement learning, which enables the algorithm to process high-dimensional perceptual inputs. The agent constantly interacts with the environment and can automatically learn the optimal actions (i.e., strategies) to be taken in different states to maximize the rewards.

[0063] In some embodiments, the data plane may be composed of a series of network elements, such as industrial Internet terminal devices, switches, and full-duplex physical links. Among them, the industrial Internet terminal devices continuously generate massive data traffic, collect data from related network devices at a set frequency, and forward it through the switch. The switch egress port can adopt the TSN / CBS architecture, and can adopt a dual queue structure for traffic scheduling, which is used for OT slices and IT slices respectively, and the traffic of the slices can enter the corresponding queues. That is, the OT slice can enter the OT slice queue, and the IT slice can enter the IT slice queue. Generally, the bandwidth between queues can be set to be strictly isolated, and the priority of the OT slice is higher than that of the IT slice. That is, the data plane includes industrial Internet terminal devices, switches, and full-duplex physical links; the industrial Internet terminal devices are used to generate data traffic, collect data at a set frequency, and forward data traffic through the switch; the switch egress port adopts the TSN / CBS architecture, has an IT slice queue and an OT slice queue; the priority of the OT slice is higher than that of the IT slice. In this way, the data plane can provide a stable and deterministic network environment for the entire SDN-based network slice resource allocation and control architecture, and deploy the resource allocation and routing strategies issued by the controller.

[0064] In some of the embodiments, the control plane may adopt a multi-layer control architecture, which may include a controller, a central controller, and a network slice controller, etc. Among them, the controller is responsible for monitoring the network status of each network slice, and the central controller is responsible for centralized control and monitoring of network equipment, execution of scheduling and resource allocation, etc., and entrusting the slice resource adjustment strategy to each network slice controller. Usually, in the control plane, a DRL agent can be deployed for each switch entity, and the DRL agent can calculate the traffic forwarding path and slice resource allocation strategy based on the network status and traffic request. The control plane sends the strategy output by the agent, such as bandwidth adjustment, to the switch on the forwarding plane through the southbound interface. The control plane carries a topology discovery module, a link monitoring module, a network status statistics module, a routing management module, and a slice management module. The control plane can use the collaborative functions of these modules to complete traffic scheduling and resource allocation.

[0065] In this way, when the controller receives a deterministic service request from a user, it can analyze the characteristic information of this deterministic flow and calculate the explicit path and resource reservation information based on the network topology and status information and the deterministic network capabilities. If the allocation is successful, it will respond to the service request. Therefore, combined with SDN technology, deterministic networks will become more flexible and agile in ensuring deterministic services. It can achieve centralized network management and configuration, improve the efficiency of network management and reduce the complexity of network configuration. By incorporating deep reinforcement learning, also known as deep neural networks, the powerful representation ability of DNN (Deep Neural Network) can fully fit the optimal strategy and adapt well to complex environments.

[0066] In some embodiments, in step S200, Figure 3 As shown, the design path planning and network slice resource allocation algorithm may include defining the system model as a network model N t , Traffic Model F t , Slice Model M t and delay model. Among them, the network model can be described as an undirected graph. That is, the network model can include an undirected graph G = (V, E). Among them, G is an undirected graph. V is a node set, which represents a set of physical switches. Specifically, V = {v1, v2, ..., v V}. E is a link set, which represents the physical link between devices. Specifically, E = {e1, e2, ..., e E}. e E It can be a link. The link can be a bidirectional link e = (u, v) ∈ E; u, v ∈ V represents the link from node u to node v. The communication network is abstracted as a graph, and the edge e can be set j The weight W i Link e i The total link bandwidth of M network slices is the aggregate bandwidth of the shared network.

[0067] In some embodiments, the deterministic network flow pattern is complex but fixed. The traffic entering the network can be set to include the source node of the traffic, the destination node of the traffic, the bandwidth requirement and the maximum end-to-end delay requirement. Specifically, it can include {SourceID, DestinationID, Bandwidth Requirement, MaxEndtoEnd Delay Requirement}. SourceID can be the source node of the traffic. DestinationID can be the destination node of the traffic. BandwidthRequirement can be the bandwidth requirement. Max EndtoEnd Delay Requirement can be the maximum end-to-end delay requirement. Among them, F can be the set of all business flows in the network. For a traffic f = {v s , v d , f b , f d , size, f q},v s , v d ∈V. Where v s and v d are nodes respectively. b is the required bandwidth of flow f. d is the maximum tolerable end-to-end delay of flow f. size is the size of the flow f frame. q is the queue to which the traffic belongs. Usually, the multi-agent decision module on the control plane can assign a feasible forwarding path to it according to the network status and business flow characteristics. The forwarding path of the traffic can be Used to represent all network nodes included in the path. v1, v2 and They can be network nodes included in the forwarding path respectively. f The bandwidth reserved for the corresponding queues along the road can be in, The bandwidth size can be reserved for node v1. The bandwidth size can be reserved for node v1. Can be a node The reserved bandwidth size.

[0068] In some of the embodiments, the network can be divided into IT network slices and OT network slices. Usually, each network slice at each network node can correspond to a queue, and the resource usage of the links in the network slice can be periodically monitored by the SDN controller. The queue corresponding to the slice can be q = {prio, load, credit, loCredit, idleSlope}, where prio is the priority of the queue. Load is the load of the queue. Credit is the credit of the queue. In the CBS mechanism, credit can determine whether the switch can send messages from a queue. LoCredit is the minimum credit of the queue, which is used to indicate the minimum value to which the credit can be reduced. IdleSlope is the rate at which the credit value accumulates, which is a key parameter for the operation of the CBS mechanism. The reserved bandwidth ratio can be R. Set R = idleSlope / C, where C can be the total bandwidth. A key parameter for the operation of the CBS mechanism can be sendSlope, which can be the rate at which the credit value is consumed.

[0069] In some of the embodiments, the end-to-end delay of the traffic may mainly include transmission delay, propagation delay, queuing delay and processing delay. The end-to-end transmission delay of the traffic may satisfy the maximum end-to-end delay constraint: in, It can be the maximum end-to-end delay constraint; hop i +1 can be the flow rate f i The number of hops experienced on the corresponding path + 1; hop i is the flow rate f i The number of hops experienced on the corresponding path; is the propagation delay, which usually depends on the device medium. To handle latency, which usually depends on the device medium, assume that all links have the same length and medium, so all data flows f i The propagation delay and processing delay on the link are the same. It can be the transmission delay, which can be the time from the first bit of the data frame to the end of the transmission of the last bit at the output port, which mainly depends on the size of the data flow and the link bandwidth R reserved by the network slice. Specifically, f i is the traffic; Size is the byte value corresponding to the data frame in the traffic, that is, the data flow size. R is the link bandwidth reserved for the network slice. The queuing delay can be the delay caused by the data frames waiting to be transmitted in the buffer of the network device. Usually, the queuing delay cannot be calculated exactly, so the queuing delay in the worst case is considered. The waiting delay during the process of credit recovering from the maximum frame size loCredit to 0 at the idleSlope rate, in order to conform to the characteristic that in the CBS mechanism, only when credit ≥ 0 can the traffic obtain the opportunity to transmit, that is, to enable the traffic to be transmitted in the CBS mechanism.

[0070] In some of these embodiments, in step S300, each node in the network can be regarded as an agent to process the corresponding formulated problem. For the agent in the algorithm, in a certain environmental state, when issuing a certain action, it will obtain the feedback of the environment, that is, the reward, and the state of the environment will also change accordingly. In the new state, the agent continues to issue actions and obtain feedback, continuously interacting with the environment. The Markov decision process MDP can be formalized as a five-tuple <S, A, O, P, R>. Among them, S can be the global state space of the environment, and O can be the partial observation space of the agent. A can be the joint action space of all agents. The joint actions of all agents can cause the global state to transfer, usually from the current state s ∈ S following the state transition probability P to the next state s′ ∈ S. R can be the single-step reward, R(r t |s t , a t ): S × A → R.

[0071] In some of these embodiments, the global state space of the environment can include S t = {N t , M t , F t}. Usually, the arriving traffic can be placed in the corresponding slice queue according to the type. S t is the environmental state observed at time t. Among them, N t can be the network resource information. M t can be the slice information. F t can be the traffic request information. F t = {f, δ t}, f can be the traffic, and δ t can be the cumulative delay of the traffic.

[0072] In some of these embodiments, the joint action space of all agents can include A = {a1, a2,..., a n}}. Usually, the actions of multiple agents can determine the traffic forwarding path and the slice resource allocation scheme. Among them, A is the joint action space of all agents. a n is the independent action of agent n. A = A n × A b . Among them, A nUsually, each node needs to make two joint decisions when forwarding traffic: One joint decision can be the next hop node selection: Each node needs to select one of the neighboring nodes as the next hop of the traffic. The action space of this part can be Each represents a possible next hop node. Another joint decision can be to allocate the percentage of slice bandwidth. Usually each agent also needs to decide the percentage of bandwidth allocated to each slice. Assuming there are m different slices, the other part of each agent's action space can be represented as A b A is the allocation ratio of the slice bandwidth of the node where the traffic is located; b =p1,p2,...,p m ; A b is another part of the action space of each agent; each p j is the bandwidth percentage allocated to the jth slice by the node where the traffic is located at time t, and satisfies m is the type of slice.

[0073] In some embodiments, the optimization goal is to maximize the overall network throughput and resource utilization while satisfying various constraints. A reward function can be designed accordingly. The reward function can be R = αR throughput +βR utilization -γR delay -δR loop Among them, R throughput The network throughput can be maximized by maximizing the amount of schedulable traffic. To measure the throughput. To maximize the amount of dispatchable traffic. utilization It can be the network resource utilization, which can be expressed as the ratio of the actually used bandwidth resources to the available bandwidth resources. Usually, to simplify the calculation, the resource utilization can be defined as the average bandwidth utilization of all links in the network. Where E is the link set, e is a bidirectional link, and ρ(e) is the bandwidth utilization on the link. delay It is the penalty for traffic scheduling failure caused by loop. in, is the maximum end-to-end delay constraint, f d is the maximum tolerable end-to-end delay of flow f. loop is the penalty for traffic scheduling failure due to violation of deadline satisfaction criteria. loop =-C loop ×|| loop ,|| loop ∈[0, 1]. loop and ||loop are weight parameters respectively. Among them, C loop Can be a positive constant, equivalent to the penalty size, to ensure that the impact of the loop penalty on the total reward function R is as expected. || loop 0 means no loop occurs and no penalty is imposed, 1 means a loop occurs and a penalty is imposed. α, β, γ, and δ are weight parameters. delay and R loop These two penalty items can guide the agent to calculate the forwarding path that meets the delay requirements.

[0074] In some embodiments, in step S400, Figure 4 As shown, the construction and training of the multi-agent deep reinforcement learning model may include:

[0075] 1) Create a network topology model; each node has an IT slice queue and an OT slice queue. In a specific application, there may be n = 15 nodes in the network topology, and each node retains Q n (Q n =2) queues are used for OT slices and IT slices respectively, and the link bandwidth in the topology can be set to a unified value of 100MB / s.

[0076] 2) Initialize the iteration variable, determine the maximum number of iterations, and set the length of the experience replay pool. In specific applications, the initial number of iterations can be set to i = 0, and the maximum number of iterations can be i max ,i max ≥5000,i max It can be set based on actual needs. Set the experience replay pool, the length can be set to n, n>=5000.

[0077] 3) Create an agent object instance and initialize network parameters. In a specific application, 15 agent object instances can be created. In the MAPPO model, each agent corresponds to an Actor network, and all agents share a Critic network. They all use a three-layer fully connected neural network and randomly initialize network parameters.

[0078] 4) Randomly generate traffic entering the network. Usually, the information of each deterministic network traffic f can be expressed as a six-tuple f = {v s , v d , f b , f d , size, f q}, respectively describe the characteristics of the deterministic flow: including source and destination ports, bandwidth requirements, end-to-end delay limit, and the queue to which it belongs. Among them, v s Can be a source port. d Can be the destination port. bis the required bandwidth of flow f. d is the maximum tolerable end-to-end delay of flow f. size is the size of the flow f frame. q The queue to which the traffic belongs. In specific applications, the source node and the destination node can be created by selecting 2 values ​​from the vertex set with equal probability. The length of the OT slice data packet can be 100-400B, the period can be randomly generated in {1, 2, 4, 8} ms, the sending time is randomly distributed, and the upper limit of the delay follows the normal distribution with a minimum value of 20ms and a maximum value of 50ms. The bandwidth of the traffic follows the normal distribution with a minimum value of 5MB / s and a maximum value of 20MB / s. The size of the IT traffic data frame follows a uniform distribution, the mean value can be 500Byte, and the sending time is randomly distributed.

[0079] 5) Set the initial global state space. For example, the initial state can be set to a combination of network information, slice information, and traffic information S t = {N t , M t , F t}.

[0080] 6) Input the initial global state space to the multi-agent deep reinforcement learning model, and each agent generates an action. In this step, the iteration starts, the value of i increases by 1, and each agent generates an action. Specifically, the agent can select node A from the neighboring nodes of the action set through the Actor network. n =v i , v i ∈V and the resource allocation ratio A of each slice b ={p OT , p IT}. Multi-Agent Deep Reinforcement Learning (MADRL) is a branch of reinforcement learning. It can apply the ideas and algorithms of deep reinforcement learning to the learning and control of multi-agent systems, and study the decision-making problems of collaboration or competition among multiple agents. The strategy of each agent in a multi-agent system depends not only on its own strategy and environmental feedback, but is also affected by the behavior and cooperative relationship of other agents, involving more complex state space and action space. The multi-agent deep reinforcement learning model can adopt the existing mappo model. The MAPPO multi-agent reinforcement learning algorithm adopts the Actor-Critic architecture, in which the actor network is responsible for strategy optimization, the critic network is responsible for estimating the value function, and the actor network is responsible for generating actions.

[0081] 7) Reorganize the actions of each agent into a joint action Action = A n ×A b, which can be applied to the network as a traffic forwarding path and network slice resource allocation solution.

[0082] 8) Generate traffic requests, obtain the number of successfully scheduled traffic in the network, network resource utilization, traffic transmission delay and queuing delay at each node, calculate network throughput, resource utilization, end-to-end delay and path constraints, and calculate rewards. Specifically, the network throughput R can be calculated. throughput , Resource utilization R utilization , end-to-end delay R delay and path constraint R loop , according to the weight index That is Reward.

[0083] 9) The initial global state space, reward and next global state space are stored in the experience replay pool for iterative learning of the multi-agent deep reinforcement learning model. In specific applications, State' can be set to a combination of network information, slice information, and traffic information S t = {N t ,M t ,F t}. Set State, Action = A n ×A b , Reward, State' is stored in the experience replay pool for reference in the iterative learning of the intelligent agent's deep reinforcement learning model.

[0084] 10) Repeat the steps of randomly generating traffic entering the network to storing the initial global state space, reward and next global state space in the experience replay pool until the length of the experience replay pool reaches the threshold. That is, repeat steps 4) to 9) until the experience replay pool is full.

[0085] 11) According to the policy update ratio, the policy of each agent deep reinforcement learning model is updated, and the parameters of the agent deep reinforcement learning model (that is, the parameters of the actor network) are updated. In specific applications, the policy gradient update rule of the PPO algorithm can be used to maximize the expected return. The network parameters can be updated by back-propagating the loss function to the Critic network / Actor network.

[0086] 12) Update the parameters of the network shared by all agent deep reinforcement learning models based on the updated parameters of the agent deep reinforcement learning model. That is, repeat step 11 and update the actor-old network with the parameters of the actor-new network.

[0087] 13) Determine the number of iterations; in response to determining that the number of iterations has not reached the maximum number of iterations, return to the step of randomly generating traffic entering the network until the maximum number of iterations is reached. That is, determine whether the number of iterations i exceeds the maximum number of iterations i max If it does not exceed, return to step 4 and continue iterating. If it exceeds, the algorithm ends.

[0088] In some embodiments, the loss function of each agent actor network can be: Among them, G is the set of agents g; S is the state space; is the generalized advantage function; k gt (θ) is the importance weight; σ and v are constants; θ is the parameter of the actor network, o gt is the observation value of agent g at time step t.

[0089] In some embodiments, the loss function of the Critic network shared by all agents is: Among them, φ is the parameter of the Critic network shared by all agents; is the state value; R gt is the cumulative discount income; C φ The clipping function.

[0090] The SDN-based slice resource allocation and control architecture designed in the TSN network in the embodiment of the present application performs resource planning and traffic scheduling through the SDN centralized controller to ensure the deterministic service quality (Quality of Service, QoS) of the business and effectively improve the resource utilization and network throughput of the physical network.

[0091] The embodiment of the present application is aimed at the intranet of the industrial Internet factory. Taking into account different traffic characteristics and CBS traffic shaping mechanism, the complex joint path planning and network slice resource allocation problems are modeled as a multi-agent Markov decision process to achieve fine-grained resource allocation. By providing the definition of state, operation and reward in the SDN-based slice resource allocation management and control framework under the introduced TSN network, the traffic calculation forwarding next hop and the slice resources at each node along the path are divided, forming a network slice resource allocation method, which expands the boundary of network performance optimization to maximize network resource utilization.

[0092] This application is aimed at the industrial Internet, and uses a multi-agent reinforcement learning algorithm based on MAPPO to implement a network slice resource allocation scheme. The network nodes are set as agents, and each agent is responsible for calculating the next-hop node of the traffic and the resource allocation ratio of each slice at the node. Multiple agents share the reward of joint actions. After multiple iterative learning, while ensuring the deterministic service of OT slices, the throughput of IT slices is improved, avoiding the risk of excessive computing node load or network congestion causing a decline in the overall user experience. In addition, the slice resource allocation method for the industrial Internet in the embodiment of the present application can adapt to the dynamic nature of the environment, without the need to manually establish a complex static model, and can adjust the scheduling strategy in real time to adapt to the new environment.

[0093] It is understandable that before using the technical solutions of each embodiment of the present disclosure, the type, scope of use, usage scenarios, etc. of the personal information involved will be informed to the user in an appropriate manner, and the user's authorization will be obtained.

[0094] For example, in response to receiving an active request from a user, a prompt message is sent to the user to clearly remind the user that the operation requested to be performed will require obtaining and using the user's personal information. Thus, the user can independently choose whether to provide personal information to software or hardware such as an electronic device, application, server, or storage medium that performs the operation of the technical solution of the present disclosure according to the prompt message.

[0095] As an optional but non-limiting implementation, in response to receiving the user's active request, the prompt information may be sent to the user in the form of a pop-up window, in which the prompt information may be presented in text form. In addition, the pop-up window may also carry a selection control for the user to choose "agree" or "disagree" to provide personal information to the electronic device.

[0096] It is understandable that the above notification and the process of obtaining user authorization are merely illustrative and do not constitute a limitation on the implementation of the present disclosure. Other methods that meet relevant laws and regulations may also be applied to the implementation of the present disclosure.

[0097] It should be noted that the method of the embodiment of the present application can be performed by a single device, such as a computer or server. The method of this embodiment can also be applied to a distributed scenario and completed by multiple devices cooperating with each other. In the case of such a distributed scenario, one of the multiple devices can only perform one or more steps in the method of the embodiment of the present application, and the multiple devices will interact with each other to complete the described method.

[0098] It should be noted that the above describes some embodiments of the present application. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the above embodiments and still achieve the desired results. In addition, the processes depicted in the accompanying drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0099] Based on the same inventive concept, corresponding to any of the above-mentioned embodiments and methods, the present application also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the network slicing resource allocation method for the industrial Internet described in any of the above embodiments is implemented.

[0100] Figure 5 A more specific schematic diagram of the hardware structure of an electronic device provided in this embodiment is shown, and the device may include: a processor 1010, a memory 1020, an input / output interface 1030, a communication interface 1040, and a bus 1050. The processor 1010, the memory 1020, the input / output interface 1030, and the communication interface 1040 are connected to each other through the bus 1050 in the device.

[0101] The processor 1010 can be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this specification.

[0102] The memory 1020 may be implemented in the form of ROM (Read Only Memory), RAM (Random Access Memory), static storage device, dynamic storage device, etc. The memory 1020 may store an operating system and other application programs. When the technical solutions provided in the embodiments of this specification are implemented by software or firmware, the relevant program codes are stored in the memory 1020 and are called and executed by the processor 1010.

[0103] The input / output interface 1030 is used to connect the input / output module to realize information input and output. The input / output module can be configured in the device as a component (not shown in the figure), or it can be externally connected to the device to provide corresponding functions. The input device may include a keyboard, a mouse, a touch screen, a microphone, various sensors, etc., and the output device may include a display, a speaker, a vibrator, an indicator light, etc.

[0104] The communication interface 1040 is used to connect a communication module (not shown) to realize communication interaction between the device and other devices. The communication module can realize communication through a wired mode (such as USB, network cable, etc.) or a wireless mode (such as mobile network, WIFI, Bluetooth, etc.).

[0105] The bus 1050 includes a path that transmits information between the various components of the device (eg, the processor 1010, the memory 1020, the input / output interface 1030, and the communication interface 1040).

[0106] It should be noted that, although the above device only shows the processor 1010, the memory 1020, the input / output interface 1030, the communication interface 1040 and the bus 1050, in the specific implementation process, the device may also include other components necessary for normal operation. In addition, it can be understood by those skilled in the art that the above device may also only include the components necessary for implementing the embodiments of the present specification, and does not necessarily include all the components shown in the figure.

[0107] The electronic device of the above-mentioned embodiment is used to implement the corresponding network slice resource allocation method for the industrial Internet in any of the above-mentioned embodiments, and has the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0108] Based on the same inventive concept, corresponding to any of the above-mentioned embodiment methods, the present application also provides a non-transitory computer-readable storage medium, wherein the non-transitory computer-readable storage medium stores computer instructions, and the computer instructions are used to enable the computer to execute the network slice resource allocation method for the industrial Internet as described in any of the above embodiments.

[0109] The computer-readable medium of this embodiment includes permanent and non-permanent, removable and non-removable media, and information storage can be implemented by any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, read-only compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device.

[0110] The computer instructions stored in the storage medium of the above embodiment are used to enable the computer to execute the network slice resource allocation method for the industrial Internet as described in any of the above embodiments, and have the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0111] Based on the same inventive concept, corresponding to the network slice resource allocation method for the industrial Internet described in any of the above embodiments, the present disclosure also provides a computer program product, which includes computer program instructions. In some embodiments, the computer program instructions can be executed by one or more processors of a computer so that the computer and / or the processor execute the network slice resource allocation method for the industrial Internet. Corresponding to the execution subject corresponding to each step in each embodiment of the network slice resource allocation method for the industrial Internet, the processor that executes the corresponding step may belong to the corresponding execution subject.

[0112] The computer program product of the above-mentioned embodiment is used to enable the computer and / or the processor to execute the network slice resource allocation method for the industrial Internet as described in any of the above embodiments, and has the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0113] Those skilled in the art should understand that the discussion of any of the above embodiments is merely illustrative and is not intended to imply that the scope of the present application (including the claims) is limited to these examples. In line with the concept of the present application, the technical features in the above embodiments or different embodiments may be combined, the steps may be implemented in any order, and there are many other variations of the different aspects of the embodiments of the present application as described above, which are not provided in detail for the sake of simplicity.

[0114] In addition, to simplify the description and discussion, and in order not to make the embodiments of the present application difficult to understand, the known power supply / ground connection with the integrated circuit (IC) chip and other components may or may not be shown in the provided drawings. In addition, the device can be shown in the form of a block diagram to avoid making the embodiments of the present application difficult to understand, and this also takes into account the fact that the details of the implementation of these block diagram devices are highly dependent on the platform to be implemented in the embodiments of the present application (that is, these details should be fully within the scope of understanding of those skilled in the art). In the case of elaborating specific details (e.g., circuits) to describe exemplary embodiments of the present application, it is obvious to those skilled in the art that the embodiments of the present application can be implemented without these specific details or when these specific details are changed. Therefore, these descriptions should be considered to be illustrative rather than restrictive.

[0115] Although the present application has been described in conjunction with specific embodiments of the present application, many replacements, modifications and variations of these embodiments will be apparent to those skilled in the art from the foregoing description. For example, other memory architectures (e.g., dynamic RAM (DRAM)) may use the embodiments discussed.

[0116] The embodiments of the present application are intended to cover all such substitutions, modifications and variations that fall within the broad scope of the appended claims. Therefore, any omissions, modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the embodiments of the present application should be included in the scope of protection of the present application.

Claims

1. A network slice resource allocation method for the industrial Internet, characterized in that: include: Design an SDN-based network slice resource allocation management and control architecture, including a control plane and a data plane; wherein the data plane is configured to provide a network environment and deploy resource allocation and routing policies issued by the controller; the control plane is configured to calculate traffic forwarding paths and slice resource allocation policies, and perform traffic scheduling and resource allocation; in the control plane, an intelligent agent is deployed for each switch node, and each switch node has an IT slice queue and an OT slice queue; Design path planning and network slice resource allocation algorithms; Model joint path planning and network slice resource allocation as a multi-agent Markov decision process; define state space and action space; design a reward function; wherein the state space includes the global state space of the environment and the partial observation space of the agent; the action space includes the joint action space of all agents; Build and train multi-agent deep reinforcement learning models; The global network view and demand information of each slice collected in the SDN-based network slice resource allocation management and control architecture are input into the trained deep reinforcement learning agent to output the traffic forwarding path and network slice resource allocation plan.

2. The network slice resource allocation method for the industrial Internet according to claim 1 is characterized in that: The control plane includes a controller, a central controller and each slice controller; wherein the controller is used to monitor the network status of each slice; the central controller is used to centrally control and monitor network devices, perform scheduling and resource allocation, and delegate slice resource adjustment strategies to each slice controller; The data plane includes industrial Internet terminal equipment, switches and full-duplex physical links; the industrial Internet terminal equipment is used to generate data traffic, collect data at a set frequency and forward the data traffic through the switch; the switch egress port adopts the TSN / CBS architecture, with IT slice queues and OT slice queues; the priority of the OT slice is higher than the priority of the IT slice.

3. The network slice resource allocation method for the industrial Internet according to claim 1 is characterized in that: The designed path planning and network slice resource allocation algorithm includes: The network model includes an undirected graph G = (V, E); where V is a node set, V = {v1, v2, ..., v V }; E is the link set, E={e1,e2,…,e E }; the link is a bidirectional link e = (u, v) ∈ E; u, v ∈ V; is the link from node u to node v; edge e i The weight W i Link e i The total link bandwidth of M network slices is the aggregate bandwidth of the shared network; The traffic entering the network includes the source node of the traffic, the destination node of the traffic, the bandwidth requirement and the maximum end-to-end delay requirement; where F is the set of all traffic in the network; f = {v s ,v d ,f b ,f d ,size,f q },v s ,v d ∈V;f b is the required bandwidth of flow f, f d is the maximum tolerable end-to-end delay of flow f; size is the size of the flow f frame, f q is the queue to which the traffic belongs; the forwarding path p of the traffic f ={v1,v2,…,v |pf| }, v1 is the network node included in the forwarding path; p f The bandwidth reserved for the corresponding queues along the road in, Reserve bandwidth for node v1; The queue corresponding to the slice is q = {prio, load, credit, loCredit, idleSlope}, where prio is the priority of the queue; load is the load of the queue; credit is the credit of the queue; loCredit is the minimum credit of the queue; idleSlope is the rate of credit accumulation; the reserved bandwidth ratio is R; R = idleSlope / C, where C is the total bandwidth; sendSlope = idleSlope-C; sendSlope is the rate of credit consumption; The end-to-end transmission delay of the traffic meets the maximum end-to-end delay constraint: in, is the propagation delay; is the transmission delay; f i is the flow rate; Size is the byte value corresponding to the data frame in the flow rate; is the queuing delay; It is the waiting delay in the process of credit recovering from the maximum frame size loCredit to 0 at the idleSlope rate; is the maximum end-to-end delay constraint; hop i +1 is the flow rate f i The number of hops experienced on the corresponding path + 1; hop i is the flow rate f i The number of hops experienced on the corresponding path; To process delay.

4. The network slice resource allocation method for the industrial Internet according to claim 3 is characterized in that: The global state space of the environment consists of S t = {N t ,M t ,F t }; Put the arriving traffic into the corresponding slice queue according to the type, S t is the environmental state observed at time t; N t is network resource information; t is the slice information; F t Request information for traffic; F t ={f,δ t }, f is the flow rate, δ t is the accumulated delay of the traffic; The partial observation space of the agent includes O = {o1, o2, ..., o n }; Among them, n is part of the observation space of agent n; The joint action space of all agents consists of A = {a1, a2, …, a n }; where a n is the independent action of agent n; including A=A n ×A b Among them, A n Select the next hop for the traffic; A b A is the allocation ratio of the slice bandwidth of the node where the traffic is located; b =p1,p2,…,p m ; A b is another part of the action space of each agent; each p j is the bandwidth percentage allocated to the jth slice by the traffic node at time t, and satisfies m is the type of slice.

5. The network slice resource allocation method for the industrial Internet according to claim 3 is characterized in that: The reward function is R = αR throughput +βR utilization -γR delay -δR loop ; Among them, R throughput is the network throughput; To maximize the amount of dispatchable traffic; R utilization is the network resource utilization; Where ρ(e) is the bandwidth utilization on the link; R delay is a penalty for traffic scheduling failure caused by loops; R loop is the penalty for traffic scheduling failure due to violation of deadline satisfaction criteria; R loop =-C loop ×|| loop ,|| loop ∈[0,1]; C loop and || loop are weight parameters respectively; α, β, γ, δ are weight parameters respectively.

6. The network slice resource allocation method for the industrial Internet according to claim 1 is characterized in that: The constructing and training the multi-agent deep reinforcement learning model includes: Create a network topology model; each node has an IT slice queue and an OT slice queue; Initialize iteration variables, determine the maximum number of iterations, and set the length of the experience replay pool; Create an agent object instance and initialize network parameters; Randomly generate traffic entering the network; Set the initial global state space; Input the initial global state space to the multi-agent deep reinforcement learning model, and each agent Actor network generates actions separately; Recombining the actions generated by each Actor network into joint actions to be applied to the network as traffic forwarding paths and network slice resource allocation schemes; Generate traffic requests, obtain the number of successfully scheduled traffic in the network, network resource utilization, traffic transmission delay and queuing delay at each node, calculate network throughput, resource utilization, end-to-end delay and path constraints, and calculate rewards; Storing the initial global state space, reward, and next global state space in an experience replay pool for iterative learning of the multi-agent deep reinforcement learning model; Repeat the steps of randomly generating traffic entering the network to storing the initial global state space, reward and next global state space in the experience replay pool until the length of the experience replay pool reaches a threshold; According to the strategy update ratio, the strategy of each agent Actor network is updated, and the parameters of the agent Actor network are updated; Update the parameters of the Critic network shared by all agents based on the updated parameters of the Actor network; Determine the number of iterations; in response to determining that the number of iterations has not reached the maximum number of iterations, return to the step of randomly generating traffic entering the network until the maximum number of iterations is reached here.

7. The network slice resource allocation method for the industrial Internet according to claim 6 is characterized in that: The loss function of each agent Actor network is: Among them, G is the set of agents g; S is the state space; is the generalized advantage function; k gt (θ) is the importance weight; σ and v are constants; θ is the parameter of the agent Actor network, o gt is the observation value of agent g at time step t; The loss function of the Critic network shared by all agents is: Among them, φ is the parameter of the Critic network shared by all agents; is the state value; R gt is the cumulative discount income; C φ The clipping function.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the method according to any one of claims 1 to 7 when executing the program.

9. A non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to cause a computer to execute the method according to any one of claims 1 to 7.

10. A computer program product, comprising computer program instructions, which, when executed on a computer, cause the computer to execute the method according to any one of claims 1 to 7.

Images