Multi-type task offloading system and method based on double-agent deep reinforcement learning

Abstract

The present application relates to a multi-type task offloading system and method based on double-agent deep reinforcement learning, belonging to the field of communication technology. The method comprises the following steps: S1: inputting a node feature matrix into a node mapping agent; S2: the node mapping agent outputs the probability of all nodes being selected, and the node mapping with the maximum selection probability is selected; S3: determining whether all components have been mapped, yes, then executing S4, otherwise returning to S1; S4: inputting a link feature matrix into a link mapping agent; S5: the link mapping agent outputs the probability of all paths being selected, and the path mapping with the maximum selection probability is selected; S6: determining whether all links have been mapped, yes, then executing S7, otherwise returning to S4; S7: calculating the double-agent reward and loss function, and updating the relevant parameters. The present application balances the allocation of multi-type resources, improves the utilization rate of physical node resources, reduces the request blocking rate, and improves the carrying capacity of the edge cloud network and the income of the service provider.

Description

Technical Field

[0001] This invention belongs to the field of communication technology and relates to a multi-type task offloading system and method based on dual-agent deep reinforcement learning. Background Technology

[0002] Wireless communication has evolved from first-generation networks providing basic coverage only for large commercial customers to 5G offering enhanced mobile broadband, massive machine-type communications, and ultra-reliable and low-latency communication capabilities. Multi-access edge computing networks are characterized by a variety of physical devices using multiple access technologies. 5G networks can provide multi-dimensional resources belonging to a large number of geographically distributed network devices with varying hardware capabilities in computing, communication, and memory. The heterogeneity of 5G infrastructure presents challenges to optimizing multi-dimensional resources. The increasing demand for intelligent applications from users leads to more diverse user task requirements. A multi-component task request is often random and exhibits uncertain relationships between different resource requests; for example, compute-intensive tasks have lower storage and communication requirements, while other types of tasks, such as video caching, require significant storage resources. Existing work mainly focuses on tasks with fewer resource types and cannot meet the demands of diverse task requests, posing challenges to network resource management and optimization.

[0003] Furthermore, the number of resource types requested by intelligent applications is constantly increasing, and considering the complexity of wireless networks, the solution space for solving mathematical models grows significantly. Exact algorithms can no longer obtain optimal solutions, while heuristic algorithms with excessively high time complexity make finding feasible solutions very costly. Deep reinforcement learning focuses on how agents respond to environmental rewards or penalties, developing habitual behaviors, and quickly identifying the best solutions for task offloading in edge clouds. Although offloading decisions can be made by collecting all information about the system state using a single agent, as network scale continues to expand, single-agent deep reinforcement learning methods will face enormous computational pressure, with solution time increasing exponentially, making it difficult to find feasible solutions in a short time. Summary of the Invention

[0004] In view of this, the purpose of this invention is to provide a multi-type task offloading system and method based on dual-agent deep reinforcement learning, solving the technical problem that it is difficult to find a feasible solution in a short time when using single-agent deep reinforcement learning methods for multi-type task offloading. The node mapping agent module designed in this invention fully considers the local and global attributes of edge cloud network nodes when mapping multiple components of multi-type task requests: node remaining resources, node betweenness centrality, node feature vector centrality, and node connectivity centrality, as well as the net reward function for node mapping. The link mapping agent module designed in this invention fully considers the local and global attributes of edge cloud network links when mapping links of multi-type task requests: link remaining bandwidth, link packet loss rate, link betweenness centrality, and the mean difference of link remaining bandwidth, using the net benefit of link mapping as the reward function. This balanced allocation of multi-type resources in the edge cloud network improves the utilization rate of various types of resources in edge cloud network nodes, reduces task blocking rate, increases the carrying capacity of the edge cloud network for task requests, and increases the revenue of service providers.

[0005] To achieve the above objectives, the present invention provides the following technical solution:

[0006] A multi-task offloading method based on dual-agent deep reinforcement learning, which includes the following steps:

[0007] S1: The edge cloud network model module will collect various components from multiple types of task requests, use the edge cloud network node features to form a node feature matrix, and input it to the node mapping agent module.

[0008] S2: The node mapping agent module outputs the probability of all nodes in the node feature matrix being selected, selects the physical node with the highest probability for mapping, and updates the edge cloud network model module resources.

[0009] S3: The node mapping agent module determines whether all components of the multi-type task request have been mapped. If all components have been mapped, then execute S4 to map the link of the multi-type task request; otherwise, return to the next component of the multi-type task request mapped by S1.

[0010] S4: The edge cloud network model module will collect various components from multiple types of task requests, and input the link feature matrix composed of edge cloud network link features into the link mapping agent module.

[0011] S5: The link mapping agent module outputs the probability of all paths in the link feature matrix being selected, selects the path with the highest probability for mapping, and updates the edge cloud network model module resources.

[0012] S6: The link mapping agent module determines whether all links of the multi-type task request have been mapped. If all links have been mapped, S7 is executed; otherwise, S4 is returned to map the next link in the multi-task request.

[0013] S7: The environment module calculates the reward and loss functions of the node mapping agent module and the link mapping agent module, and updates the parameters of the node mapping agent module and the link mapping agent module.

[0014] Furthermore, in S2, the node mapping agent module outputs the probability of all nodes being selected in the node feature matrix, selects the physical node with the highest probability for mapping, and updates the edge cloud network model module resources, specifically:

[0015] S21: Input the node feature matrix into the input layer 1 of the node mapping agent module, and perform a normalization operation on the node feature matrix using the deviation normalization method;

[0016] S22: Input the normalized node feature matrix into the first convolutional layer of the node mapping agent module, whereby the first convolutional layer generates the available resource vector for each node;

[0017] S23: Input the available resource vector of each node into the Softmax layer 1 of the node mapping agent module, wherein the Softmax layer 1 transforms the available resource vector of each node into the probability of each node being selected;

[0018] S24: Input the probability of each node being selected into the node mapping of the agent module's filter layer one. The filter layer one filters out nodes that do not satisfy the constraints of node resource type, node resource quantity, node one-to-one constraint, and flow conservation constraint, and records the probability of filtering out nodes that do not satisfy any of the above constraints as 0.

[0019] S25: The output layer of the node mapping agent module recalculates the probability of each node in the edge cloud network model module being selected after being filtered by the filtering layer. The output layer outputs the probability corresponding to each node. The output layer selects the node with the highest output probability for mapping and updates the resources of the edge cloud network model module.

[0020] Furthermore, in S5, the link mapping agent module outputs the probability of all paths being selected in the link feature matrix, selects the path with the highest probability for mapping, and updates the edge cloud network model module resources, specifically:

[0021] S31: Input the link feature matrix into the second input layer of the link mapping agent module, and perform a normalization operation on the link feature matrix using the deviation normalization method;

[0022] S32: Input the normalized link feature matrix into the second convolutional layer of the link mapping agent module, and the second convolutional layer generates the available resource vector for each path;

[0023] S33: Input the available resource vector of each path into the Softmax layer 2 of the link mapping agent module, whereby the Softmax layer 2 transforms the available resource vector of each path into the probability of each path being selected.

[0024] S34: The probability of each path being selected is input into the second filtering layer of the link mapping agent module. The second filtering layer filters out paths that do not satisfy the link direction constraint, flow conservation constraint, bandwidth capacity constraint, and packet loss rate constraint, and records the probability of filtering out a path that does not satisfy one of the above constraints as 0.

[0025] S35: The output layer 2 of the link mapping agent module recalculates the probability of each path filtered by the filtering layer being selected. The output layer 2 of the link mapping agent module outputs all paths and their corresponding probabilities. The output layer 2 selects the path with the highest output probability for mapping and updates the edge cloud network model module resources.

[0026] A multi-task offloading system based on dual-agent deep reinforcement learning includes an environment module, a node mapping agent module, and a link mapping agent module. The environment module contains an edge cloud network model module and a multi-task request model module. The environment module inputs the collected node feature matrix into the node mapping agent module. The node mapping agent module selects the node with the highest probability for each component in each multi-task request and maps it, updating the edge cloud network model module resources. After the node mapping agent module maps all components of the multi-task requests, the environment module inputs the collected link feature matrix into the link mapping agent module. The link mapping agent module selects the path with the highest probability for each link in the multi-task request and maps it, updating the edge cloud network model module resources. After the link mapping agent module maps all links in the multi-task requests, the environment module calculates the reward and loss functions of both the node mapping agent module and the link mapping agent module, and updates their parameters.

[0027] Furthermore, the edge cloud network model module is described as: a directed graph G s (V s E s ) represents edge cloud network; V s E represents the set of physical nodes in an edge cloud network. s This represents the set of physical links in an edge cloud network. They represent the i-th element in the edge cloud network, respectively. sThe total amount and remaining amount of type t resources owned by each physical node, where T represents the set of resource types that a physical node can provide, which includes type I, type II, and type III resources, i.e., T = {I, II, III}. and Representing physical node i s The betweenness centrality, eigenvector centrality, and connectivity centrality; For physical node i s The set of neighboring nodes, Let each represent the j-th element in the edge cloud network. s The total bandwidth resources and remaining bandwidth resources of each physical link. Indicates the j-th s Packet loss rate of each physical link, of which and BD js Representing physical link j respectively s The difference between the mean difference in betweenness centrality and residual bandwidth; Indicates from source physical node a s To the destination physical node b s The kth physical path traversed, and Representing paths The remaining bandwidth and packet loss rate, P s (a s ,b s ) represents the source physical node a s To the destination physical node b s The set of acyclic paths, K = |P s (a s ,b s )|, where

[0028]

[0029] In formula (1), Indicates the i-th s The betweenness centrality of physical nodes in an edge cloud network is a measure of how many times a node acts as a bridge between two other nodes for the shortest path. The more times a node acts as a bridge, the greater its betweenness centrality. This is a binary variable representing physical node a. s to b s The number of shortest paths between them, a s ≠b s , Indicates a s to bs Does the shortest path between them pass through node i? s If the condition is met, the value is 1; otherwise, the value is 0.

[0030]

[0031] In formula (2), Represents physical node i s Eigenvector centrality indicates that the importance of a node is influenced by the importance of its neighboring nodes; that is, the more important the nodes a node is connected to, the more important the node itself is. Here, λ is a constant. Indicates if a s is i s The neighboring nodes, The value is 1 if it is not 1, otherwise the value is 0.

[0032]

[0033] In formula (3), Represents physical node i s Connectivity centrality, a measure of physical node i s The distance to all other nodes in the network, among which Indicates node i s To node a s The number of hops in the shortest path between them;

[0034]

[0035] In formula (4), Indicates physical link j s The centrality of the intermediaries, through the link j s The number of shortest paths is used as a metric to measure the importance of a link;

[0036]

[0037] In formula (5), Indicates physical link j s The difference between the remaining bandwidth and the average remaining bandwidth of all links in the network;

[0038]

[0039] In formula (6), Represents the k-th path The remaining bandwidth is equal to the minimum remaining bandwidth of all links in the path;

[0040]

[0041] In formula (7), Represents the k-th path The packet loss rate;

[0042] Furthermore, the multi-type task request model module is described as follows: R represents the set of all multi-type task requests; R fail G represents a set of multiple task types where mapping failed; r (V r E r () represents a multi-type task request, r∈R; V r E represents the set of components in the r-th multi-type task request. r This represents the set of links between components in the r-th multi-type task request; In the r-th multi-type task request, the i-th... r The number of resources of type t requested by each component, where and They represent the j-th multi-type task request in the r-th request. r The number of bandwidth resources requested for each link and the maximum tolerable packet loss rate, among which

[0043] The total revenue from unloading multiple types of task requests is the difference between the revenue and cost of successfully mapped requests, as shown in formula (8):

[0044]

[0045] Where, γ t χ represents the selling price of the t-th type of resource requested by each component. t Let φ represent the unit cost of deploying the t-th type of resource in a physical node. These represent the selling price and cost price of the bandwidth resources, respectively. It is a binary variable, if the i-th request of the task r The i-th component is placed in the edge cloud network. s Its value is 1 when it is on a physical node, otherwise its value is 0; It is a binary variable, if the i-th request of the task r The value of a component is 1 when the t-th type of resource is selected; otherwise, the value is 0. It is a binary variable, if the j-th request of the task r The j-th link is placed in the edge cloud network. s Its value is 1 when there is a physical link; otherwise, its value is 0.

[0046] The formula to maximize the total revenue (TB) from multiple task types is:

[0047] max:TB(9)

[0048] The constraints are:

[0049]

[0050]

[0051]

[0052]

[0053]

[0054]

[0055]

[0056] Formula (10) represents a one-to-one node constraint: a component in a multi-type task request can only be placed on a physical node in the edge cloud network; Formula (11) represents a node resource type constraint: a component in a multi-type task request can only request one type of resource on a physical node in the edge cloud network; Formula (12) represents a node resource quantity constraint: the total capacity of the t-th type of resource occupied by all requests does not exceed the total capacity of the t-th type of resource on physical nodes in the edge cloud network; Formula (13) represents a link direction constraint: a link in a multi-type task request can only select a physical link in one direction in the edge cloud network; Formula (14) represents a flow conservation constraint: the inflow of each physical node in the edge cloud network is equal to the outflow; Formula (15) represents a link bandwidth capacity constraint: the total bandwidth occupied by all multi-type requests in the edge cloud network does not exceed the total bandwidth of the links; Formula (16) represents a packet loss rate constraint: the packet loss rate of the selected mapped path in the edge cloud network is less than or equal to the tolerable packet loss rate of the link between components of the multi-type task request;

[0057] Evaluation metrics for various task request types include total revenue (as shown in formula (8), blocking rate, average node resource utilization, and average bandwidth utilization:

[0058]

[0059]

[0060]

[0061] Where BR represents the blocking rate of multi-type task requests; the lower the blocking rate, the more task requests the edge cloud network can handle; NU tdenoted by t, it represents the average node resource utilization rate of the edge cloud network for resource type t. A higher node resource utilization rate indicates that the node resources of the edge cloud network are being utilized more fully. LU represents the average bandwidth resource utilization rate. In multi-type task requests, the shorter the path of the link between components mapped to the edge cloud network, the smaller the number of hops, and the smaller the bandwidth occupancy rate.

[0062] Furthermore, the node mapping agent module is provided with an input layer 1, a convolutional layer 1, a softmax layer 1, a filtering layer 1, and an output layer 1 from bottom to top.

[0063] The input layer collects feature matrices of each physical node in the edge cloud network module related to t types of resources. Combining the feature matrices of all physical nodes into a node feature matrix for the t-th type of resource in the edge cloud network module. right Using the deviation standardization method, normalization is performed to obtain the normalized feature matrix. make The values ​​in the range are all between 0 and 1;

[0064] i-th s The feature matrix of each physical node related to the t-th type of resource is as follows: Includes the remaining resources of the node Node Centrality Node eigenvector centrality and node connectivity centrality

[0065]

[0066] The feature matrix of all physical nodes related to the t-th type of resource is as follows:

[0067]

[0068] The formula for the deviation standardization method is:

[0069]

[0070] Where x represents the original list, and x' represents the list after standardization of deviations. min x represents the minimum value in the original list. max This represents the maximum value in the original list;

[0071] The convolutional layer one normalizes the feature matrix. The convolution operation is performed with ω1 as the convolution kernel weight vector, b1 as the bias term, and the ReLU function as the activation function. The convolutional layer generates a vector of available resources for candidate actions. The ReLU function, or piecewise linear function, returns the value provided as input if the input is greater than 0; otherwise, it returns 0.

[0072] For the normalized node feature matrix Perform convolution operations:

[0073]

[0074] The Softmax layer will use the available resource vector of candidate actions. Transformed into the probability of each action being selected

[0075]

[0076] The first filtering layer filters out all actions that do not meet the node resource type constraint, node resource quantity constraint, node one-to-one mapping constraint and flow conservation constraint, and obtains the updated available resource vector and candidate action set for the actions.

[0077] The probability corresponding to the action of the output layer filtering out the filter layer is recorded as 0; the probability is recalculated to obtain a new probability distribution. for:

[0078]

[0079] Furthermore, the link mapping agent module is provided with input layer 2, convolutional layer 2, softmax layer 2, filtering layer 2 and output layer 2 from bottom to top;

[0080] The second input layer collects the feature matrix of each link in the edge cloud network. Then, the feature matrices of all links are used to construct the link feature matrix of the edge cloud network. right Normalization was performed using the deviation standardization method to obtain the normalized link feature matrix I. L , making I L The values ​​in the range are all between 0 and 1;

[0081] j s Feature matrix of physical links Includes remaining bandwidth of the link Packet loss rate Intermediation Centrality Difference between the average and remaining bandwidth

[0082]

[0083] The feature matrix of all physical links is:

[0084]

[0085] In the second convolutional layer, ω2 is the kernel weight vector, b2 is the bias term, and the ReLU function is the activation function. The convolutional layer generates a vector C of available resources for candidate actions. L :

[0086]

[0087] The second Softmax layer will use the available resource vector C of the candidate actions. L Transformed into the probability of each action being selected

[0088]

[0089] The second filtering layer filters out all path actions that do not meet the link direction constraint, flow conservation constraint, bandwidth capacity constraint and packet loss rate constraint, and records their probability as 0, thus obtaining the probability and action set of all actions after the update.

[0090] The second output layer recalculates the action probabilities obtained from the filtering layer to obtain a new probability distribution P. L :

[0091]

[0092] Furthermore, the reward and loss functions of the environment module computing node mapping agent module and link mapping agent module are as follows:

[0093] The total revenue from node mapping is used as the reward function for the node mapping agent module, and the formula is:

[0094]

[0095] The loss function for the node mapping agent module is:

[0096]

[0097] The total reward from link mapping is used as the reward function for the link mapping agent module, and the formula is:

[0098]

[0099] The loss function of the link mapping agent module is:

[0100]

[0101] The beneficial effects of this invention are as follows:

[0102] First, the present invention designs node mapping agents and link mapping agents to balance the allocation of various types of resources in the edge cloud network, thereby improving the utilization rate of various types of resources of edge cloud network nodes, reducing task blocking rate, increasing the carrying capacity of edge cloud network for task requests, and increasing the revenue of service providers.

[0103] Secondly, compared to a single agent, the dual agent in this invention can extract network node attributes and link attributes respectively, and can reach a stable state more quickly during the agent's learning process.

[0104] Other advantages, objectives, and features of the invention will be set forth in part in the description which follows, and in part will be apparent to those skilled in the art from the following examination, or may be learned from practice of the invention. The objectives and other advantages of the invention can be realized and obtained through the following description. Attached Figure Description

[0105] To make the objectives, technical solutions, and advantages of the present invention clearer, the preferred embodiments of the present invention will be described in detail below with reference to the accompanying drawings, wherein:

[0106] Figure 1 This is a flowchart of the multi-type task unloading method based on dual-agent deep reinforcement learning of the present invention;

[0107] Figure 2 This is a diagram illustrating the overall framework of the multi-task unloading system based on dual-agent deep reinforcement learning of this invention.

[0108] Figure 3 This is the topology diagram used in the multi-type task unloading method based on dual-agent deep reinforcement learning in this invention;

[0109] Figure 4 This is a comparison chart of the total revenue of the present invention when the number of topology graph components based on COST239 is 2-6;

[0110] Figure 5 This is a comparison chart of the blocking rates when the number of topology components based on COST239 is 2-6.

[0111] Figure 6 This is a comparison chart of the average link bandwidth resource utilization when the number of topology components based on COST239 is 2-6.

[0112] Figure 7 This is a comparison chart of the average node resource utilization rate of Type I resources when the number of COST239 topology graph components is 2-6.

[0113] Figure 8This is a comparison chart of the average node resource utilization rate of Type II resources when the number of COST239 topology graph components is 2-6.

[0114] Figure 9 This is a comparison chart of the average node resource utilization rate of Type III resources when the number of COST239 topology graph components is 2-6. Detailed Implementation

[0115] The following specific examples illustrate the implementation of the present invention. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and various details in this specification can be modified or changed based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the illustrations provided in the following embodiments are only schematic representations of the basic concept of the present invention. Unless otherwise specified, the following embodiments and features can be combined with each other.

[0116] The accompanying drawings are for illustrative purposes only and are schematic diagrams, not actual pictures. They should not be construed as limiting the invention. To better illustrate the embodiments of the invention, some parts in the drawings may be omitted, enlarged, or reduced, and do not represent the actual product dimensions. It is understandable to those skilled in the art that some well-known structures and their descriptions may be omitted in the drawings.

[0117] In the accompanying drawings of the embodiments of the present invention, the same or similar reference numerals correspond to the same or similar components. In the description of the present invention, it should be understood that if terms such as "upper," "lower," "left," "right," "front," and "rear" indicate the orientation or positional relationship based on the orientation or positional relationship shown in the drawings, they are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, the terms used to describe positional relationships in the drawings are only for illustrative purposes and should not be construed as limiting the present invention. For those skilled in the art, the specific meaning of the above terms can be understood according to the specific circumstances.

[0118] Please see Figures 1-9 This paper presents a multi-type task unloading system and method based on dual-agent deep reinforcement learning.

[0119] This invention proposes a multi-type task request offloading method based on dual-agent deep reinforcement learning. The overall algorithm flowchart is as follows: Figure 1As shown, under the same edge cloud network and multiple types of task requests, compared with the single-agent deep reinforcement learning method with node-link mapping (NF_L_SA_DRL), the dual-agent deep reinforcement learning method with node-link cross mapping (LF_L_DA_DRL), and the single-agent deep reinforcement learning method with node-link cross mapping (LF_L_SA_DRL), the average node resource utilization and net benefit of this scheme are higher than the comparison methods, and the task blocking rate of this scheme is lower than all comparison methods.

[0120] A multi-task offloading method based on dual-agent deep reinforcement learning, which includes the following steps:

[0121] Step 1: Collect the features of edge cloud network nodes to form a node feature matrix, and input it into the node mapping agent;

[0122] Step 2: The node mapping agent outputs the probability of all nodes being selected, selects the node with the highest probability for mapping, and updates the edge cloud network resources;

[0123] Step 3: Determine whether all components of the multi-type task request have been mapped. If so, proceed to Step 4 to map the multi-type task request link; otherwise, return to Step 1 to map the next component of the multi-type task request.

[0124] Step 4: Collect edge cloud network link features to construct a link feature matrix, and input it into the link mapping agent;

[0125] Step 5: The link mapping agent outputs the probability of all paths being selected, selects the path with the highest probability for mapping, and updates the edge cloud network resources;

[0126] Step 6: Determine whether all links of the multi-type task request have been mapped. If so, proceed to Step 7; otherwise, return to Step 4 to map the next link in the multi-type task request.

[0127] Step 7: Calculate the reward and loss functions for the node mapping module and the link mapping module, and update the parameters of the node mapping agent and the link mapping agent;

[0128] Furthermore, step 2 is divided into the following steps:

[0129] Step 2.1: Input the node feature matrix into the input layer of the node mapping agent, and perform normalization operation using the deviation normalization method;

[0130] Step 2.2: Input the normalized node feature matrix into the convolutional layer of the node mapping agent to generate the available resource vector for each node;

[0131] Step 2.3: Input the available resource vector of each node into the Softmax layer to transform the available resource vector of each node into the probability of each node being selected;

[0132] Step 2.4: Filter out nodes that do not meet the node resource type constraint, node resource quantity constraint, node one-to-one constraint, and flow conservation constraint, and record their probability as 0;

[0133] Step 2.5: Recalculate the probability of each node being selected in the edge cloud network, and output each node and its corresponding probability.

[0134] Furthermore, step 5 is divided into the following steps:

[0135] Step 5.1: Input the link feature matrix into the input layer of the link mapping agent and perform normalization using the deviation normalization method;

[0136] Step 5.2: Input the normalized link feature matrix into the convolutional layer of the link mapping agent to generate the available resource vector for each path;

[0137] Step 5.3: Input the available resource vector of each path into the Softmax layer, and transform the available resource vector of each path into the probability of each path being selected;

[0138] Step 5.4: Filter out paths that do not meet the link direction constraint, flow conservation constraint, bandwidth capacity constraint, and packet loss rate constraint, and record their probability as 0;

[0139] Step 5.5: Recalculate the probability of each path being selected, and output all paths and their corresponding probabilities.

[0140] The edge cloud network model is described as: a directed graph G s (V s E s ) represents edge cloud network; V s E represents the set of physical nodes in an edge cloud network. s This represents the set of physical links in an edge cloud network. They represent the i-th element in the edge cloud network, respectively. s The total amount and remaining amount of type t resources owned by each physical node, where T represents the set of resource types that a physical node can provide, which includes type I, type II, and type III resources, i.e., T = {I, II, III}. and Representing physical node i s The betweenness centrality, eigenvector centrality, and connectivity centrality; For physical node i s The set of neighboring nodes, Let each represent the j-th element in the edge cloud network. s The total bandwidth resources and remaining bandwidth resources of each physical link. Indicates the j-th s Packet loss rate of each physical link, of which and Representing physical link j respectively s The difference between the mean difference in betweenness centrality and residual bandwidth; Indicates from source physical node a s To the destination physical node b s The kth physical path traversed, and Representing paths The remaining bandwidth and packet loss rate, P s (a s ,b s ) represents the source physical node a s To the destination physical node b s The set of acyclic paths, K = |P s (a s ,b s )|, where

[0141]

[0142]

[0143]

[0144]

[0145]

[0146]

[0147]

[0148] Formula (1) represents the i-th s The betweenness centrality of physical nodes in an edge cloud network is a measure of how many times a node acts as a bridge between two other nodes for the shortest path. The more times a node acts as a bridge, the greater its betweenness centrality. This is a binary variable representing physical node a. s to b s The number of shortest paths between them, a s ≠b s , Indicates a s to bs Does the shortest path between them pass through node i? s If the node has passed through a certain point, the value is 1; otherwise, the value is 0. Formula (2) represents the value of physical node i. s Eigenvector centrality indicates that the importance of a node is influenced by the importance of its neighboring nodes; that is, the more important the nodes a node is connected to, the more important the node itself is. Here, λ is a constant. Indicates if a s is i s The neighboring nodes, The value is 1 if the physical node i is 1 otherwise the value is 0. Formula (3) represents the physical node i s Connectivity centrality, a measure of physical node i s The distance to all other nodes in the network, among which Indicates node i s To node a s The number of hops of the shortest path between; Formula (4) represents the physical link j s The centrality of the intermediaries, through the link j s The number of shortest paths is used as a metric to measure the importance of a link; Formula (5) represents the physical link j s The difference between the remaining bandwidth of the kth path and the average remaining bandwidth of all links in the network; Formula (6) represents the difference between the remaining bandwidth of the kth path and the average remaining bandwidth of all links in the network. The remaining bandwidth is equal to the minimum remaining bandwidth of all links in the path; Formula (7) represents the k-th path. The packet loss rate.

[0149] The network model for multi-component task requests is represented as follows: R represents the set of all multi-type task requests; R fail G represents a set of multiple task types where mapping failed; r (V r E r () represents a multi-type task request, r∈R; V r E represents the set of components in the r-th multi-type task request. r This represents the set of links between components in the r-th multi-type task request; In the r-th multi-type task request, the i-th... r The number of resources of type t requested by each component, where and They represent the j-th multi-type task request in the r-th request. r The number of bandwidth resources requested for each link and the maximum tolerable packet loss rate, among which

[0150] The total revenue from unloading multiple types of task requests is the difference between the revenue and cost of successfully mapped requests (as shown in Formula 8).

[0151]

[0152] Where, γ t χ represents the selling price of the t-th type of resource requested by each component. t Let φ represent the unit cost of deploying the t-th type of resource in a physical node. These represent the selling price and cost price of the bandwidth resources, respectively. It is a binary variable, if the i-th request of the task r The i-th component is placed in the edge cloud network. s Its value is 1 when it is on a physical node, otherwise its value is 0; It is a binary variable, if the i-th request of the task r The value of a component is 1 when the t-th type of resource is selected; otherwise, the value is 0. It is a binary variable, if the j-th request of the task r The j-th link is placed in the edge cloud network. s Its value is 1 when there is a physical link; otherwise, its value is 0.

[0153] The optimization objective of this invention is to maximize the total revenue (TB) from multiple types of task requests, as shown in the formula:

[0154] max:TB (9)

[0155] The constraints are:

[0156]

[0157]

[0158]

[0159]

[0160]

[0161]

[0162]

[0163] Among them, Formula (10) is a one-to-one node constraint: a component in a multi-type task request can only be placed on a physical node in the edge cloud network; Formula (11) is a node resource type constraint: a component in a multi-type task request can only request one type of resource on a physical node in the edge cloud network; Formula (12) is a node resource quantity constraint: the total capacity of the t-th type of resource occupied by all requests does not exceed the total capacity of the t-th type of resource on the physical nodes in the edge cloud network; Formula (13) is a link direction constraint: a link in a multi-type task request can only select a physical link in one direction in the edge cloud network; Formula (14) is a flow conservation constraint: the inflow of each physical node in the edge cloud network is equal to the outflow; Formula (15) is a link bandwidth capacity constraint: the total bandwidth occupied by all multi-type requests in the edge cloud network does not exceed the total bandwidth of the links; Formula (16) is a packet loss rate constraint: the packet loss rate of the selected mapped path in the edge cloud network is less than or equal to the tolerable packet loss rate of the link between the multi-type task request components.

[0164] The evaluation metrics for this method include total revenue (as shown in Formula 8), blocking rate (as shown in Formula 17), average node resource utilization (as shown in Formula 18), and average bandwidth resource utilization (as shown in Formula 19).

[0165]

[0166]

[0167]

[0168] Formula (17) represents the blocking rate of multi-type requests. The smaller the blocking rate, the more tasks the edge cloud network can carry. Formula (18) represents the average node resource utilization rate of the edge cloud network for the t-th type of resource. The higher the node resource utilization rate of the edge cloud network, the more fully the node resources of the edge cloud network are utilized. Formula (19) represents the average bandwidth resource utilization rate. The shorter the path of the link between components in multi-type task requests to the edge cloud network, the smaller the number of hops, and the smaller the bandwidth occupancy rate.

[0169] Furthermore, such as Figure 2 As shown, the overall framework of the NL_L_DA_DRL method consists of an environment module, a node mapping agent module, and a link mapping agent module. The environment module includes an edge cloud network model module and a multi-type task request model module.

[0170] Node Mapping Intelligent Agent Module

[0171] The node mapping agent module consists of, from bottom to top, an input layer 1, a convolutional layer 1, a softmax layer 1, a filter layer 1, and an output layer 1.

[0172] The input layer collects the feature matrix of each physical node in the edge cloud network related to the t-th type of resource. As shown in formula (20), the feature matrices of all physical nodes are then combined to form the node feature matrix of the edge cloud network for the t-th type of resource. As shown in formula (21), for Normalization is performed using the deviation standardization method shown in formula (21), resulting in the normalized feature matrix. make The values ​​in the range are all between 0 and 1.

[0173] i-th s The feature matrix of each physical node related to the t-th type of resource is as follows: Includes node remaining resources, node betweenness centrality, node eigenvector centrality, and node connectivity centrality:

[0174]

[0175] The feature matrix of all physical nodes related to the t-th type of resource is as follows:

[0176]

[0177] The formula for the deviation standardization method is:

[0178]

[0179] Where x represents the original list, and x' represents the list after standardization of deviations. min x represents the minimum value in the original list. max This represents the maximum value in the original list.

[0180] The function of convolutional layer 1 is to transform the normalized feature matrix The convolution operation is performed, where ω1 is the convolution kernel weight vector, b1 is the bias term, and ReLU is the activation function. The convolutional layer generates a vector of available resources for candidate actions. (As shown in Formula 23). The ReLU function, a piecewise linear function, returns the value provided as input if the input is greater than 0; if the input is 0 or less, it returns 0.

[0181] For the normalized node feature matrix Perform convolution operations:

[0182]

[0183] The Softmax layer first generates a vector of available resources for candidate actions. Transformed into the probability of each action being selected

[0184]

[0185] The first filtering layer filters out all actions that do not meet the constraints of node resource type, node resource quantity, node one-to-one mapping, and flow conservation, and obtains the updated available resource vector and candidate action set for the updated actions.

[0186] The output layer sets the probability of actions filtered out by the filter layer to 0, then recalculates the probabilities to obtain a new probability distribution.

[0187]

[0188] The total revenue from node mapping is used as the reward function for the node mapping agent module, and the formula is:

[0189]

[0190] The loss function for the node mapping agent module is:

[0191]

[0192] Link Mapping Intelligent Agent Module

[0193] Similar to the node mapping agent module, the link mapping agent module consists of an input layer 2, a convolutional layer 2, a softmax layer 2, a filtering layer 2, and an output layer 2, arranged from bottom to top.

[0194] The second input layer of the link mapping agent module collects the feature matrix of each link in the edge cloud network. (As shown in Formula 28), and then the feature matrices of all links are used to construct the link feature matrix of the edge cloud network. right Normalization is performed using the deviation standardization method (as shown in Equation 22) to obtain the normalized link feature matrix I. L , making I L The values ​​in the range are all between 0 and 1.

[0195] j s Feature matrix of physical links Includes link bandwidth remaining amount, packet loss rate, betweenness centrality, and mean difference of remaining bandwidth:

[0196]

[0197] The feature matrix of all physical links is:

[0198]

[0199] In the second convolutional layer of the link mapping agent module, ω2 is the convolutional kernel weight vector, b2 is the bias term, and the ReLU function is the activation function. The second convolutional layer generates the available resource vector C for candidate actions. L :

[0200]

[0201] The second layer of Softmax will use the available resource vector C of candidate actions. L Transformed into the probability of each action being selected

[0202]

[0203] The second filter layer filters out all path actions that do not meet the constraints of link direction, flow conservation, bandwidth capacity, and packet loss rate, and sets their probability to 0, thus obtaining the probability and action set of all actions after the update.

[0204] The output layer 2 recalculates the action probabilities obtained from the filtering layer 2, resulting in a new probability distribution P. L :

[0205]

[0206] The total revenue from link mapping is used as the reward function for the link mapping agent module, with the following formula:

[0207]

[0208] The loss function of the link mapping agent module is:

[0209]

[0210] The flowchart of the NF_L_DA_DRL algorithm is as follows: Figure 1 As shown, PyCharm is used as the simulation software to implement this scheme and verify it.

[0211] The edge cloud network model module uses a COST239 network topology diagram, such as... Figure 3 As shown in Table 1, the edge cloud network model module contains 11 nodes and 26 links. There are three types of node service resources: I, II, and III. The specific parameter settings are shown in Table 1.

[0212] Table 1 Edge Cloud Network Parameter Settings

[0213]

[0214]

[0215] The multi-type task request model module is randomly generated, with the number of components ranging from 2 to 6, and the number of links ranging from [|V] r |-1,|V r |(V r Generate within the range of -1) / 2]. There are 3 types of resource requests for tasks: I, II, III; within the same task request, each component randomly requests one type of resource, and the request types of each component are different. The range of the resource amount requested by the component is (50, 150]; the range of the request link bandwidth is (50, 150]; the range of the request link tolerable packet loss rate is [0.3, 0.5].

[0216] Table 2 Parameter Settings for Multi-Type Task Requests

[0217] Parameter name Numerical range Number of nodes [2，6] Number of links <![CDATA[[|V r |-1,|V r |(V r -1)2]]]> Request node service resource type [Ⅰ，Ⅱ，Ⅲ] Request node service resource quantity [50，150] Request link bandwidth resources [50，150] Request link to tolerate packet loss rate [0.3，0.5]

[0218] The comparison methods involved in this example include: NF_L_SA_DRL, LF_L_SA_DRL, and LF_L_DA_DRL.

[0219] NF_L_SA_DRL: It adopts a node-first, link-later mapping order and uses a single agent to complete the action selection for node mapping and link mapping.

[0220] LF_L_SA_DRL: The mapping order of node link cross-mapping is adopted. After each node is mapped, the direct links between that node and all the already mapped nodes need to be mapped. The node mapping and link mapping are implemented by the same agent.

[0221] LF_L_DA_DRL: Similar to LF_L_SA_DRL, it adopts the mapping order of node link cross-mapping, but unlike LF_L_SA_DRL which uses a single agent, the LF_L_DA_DRL algorithm uses a dual agent.

[0222] Set the number of task request components to 2-6.

[0223] Figure 4-9 The graph shows the experimental results for task request components ranging from 2 to 6. Figure 4 It can be seen that the NF_L_DA_DRL algorithm has the highest total revenue. Before 150 task requests, the total revenue of the four algorithms is not much different. However, when the number of task requests reaches 210, the total revenue of the four algorithms reaches a stable state. The total revenue of the NF_L_DA_DRL algorithm is close to 120,000, which is about 12.99% higher than the NF_SA_DRL algorithm, about 22.44% higher than the LF_L_DA_DRL algorithm, and about 26.42% higher than the LF_L_SA_DRL algorithm.

[0224] Figure 5Regarding task blocking rates, under the same conditions, the NF_L_DA_DRL and NF_L_SA_DRL algorithms, which use a node-first, link-later mapping order, have lower blocking rates than the LF_L_DA_DRL and LF_L_SA_DRL algorithms, which use node-link cross-mapping. Under the same mapping order, the blocking rate of the mapping method using two agents is lower than that of the mapping method using one agent. When the number of task requests reaches 150, the blocking rates of the NF_L_SA_DRL, LF_L_DA_DRL, and LF_L_SA_DRL algorithms begin to increase significantly. When the number of tasks reaches 300, the blocking rate of the NF_L_DA_DRL algorithm is the lowest, and its edge cloud network can handle the most task requests.

[0225] Figure 6 The NF_L_DA_DRL algorithm achieves an average bandwidth utilization rate that falls within the middle range. Theoretically, a lower average bandwidth utilization rate indicates less bandwidth usage by the edge cloud network, meaning the edge cloud network can handle more task requests given sufficient physical node service resources. This is because... Figure 5 The NF_L_DA_DRL algorithm has the lowest blocking rate and carries the most task requests in its edge cloud network. Therefore, the NF_L_DA_DRL algorithm tends to select the optimal path during mapping. As a result, the bandwidth resource utilization of the NF_L_DA_DRL algorithm is between the highest and lowest average bandwidth resource utilization among the comparison algorithms. With a certain blocking rate, the bandwidth cost is relatively small.

[0226] from Figure 7-9 From the perspective of average node resource utilization, the node-priority two-stage mapping order is superior to the node-link intersection mapping order. Furthermore, under the same mapping order, the dual-agent offloading algorithm consistently outperforms the single-agent offloading algorithm in terms of average node resource utilization. This is because, compared to a single agent, a dual-agent algorithm can extract network node attributes and link attributes separately, allowing it to reach a stable state more quickly during the agent's learning process.

[0227] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A multi-type task offloading method based on dual-agent deep reinforcement learning, characterized in that: The method includes the following steps: S1: The edge cloud network model module will collect various components from multiple types of task requests, use the edge cloud network node features to form a node feature matrix, and input it to the node mapping agent module. S2: The node mapping agent module outputs the probability of all nodes in the node feature matrix being selected, selects the physical node with the highest probability for mapping, and updates the edge cloud network model module resources. S3: The node mapping agent module determines whether all components of the multi-type task request have been mapped. If all components have been mapped, then execute S4 to map the link of the multi-type task request; otherwise, return to the next component of the multi-type task request mapped by S1. S4: The edge cloud network model module will collect various components from multiple types of task requests, and input the link feature matrix composed of edge cloud network link features into the link mapping agent module. S5: The link mapping agent module outputs the probability of all paths in the link feature matrix being selected, selects the path with the highest probability for mapping, and updates the edge cloud network model module resources. S6: The link mapping agent module determines whether all links of the multi-type task request have been mapped. If all links have been mapped, S7 is executed; otherwise, S4 is returned to map the next link in the multi-task request. S7: The environment module calculates the reward and loss functions of the node mapping agent module and the link mapping agent module, and updates the parameters of the node mapping agent module and the link mapping agent module. The edge cloud network model module is described as a directed graph. Indicates edge cloud network; This represents the set of physical nodes in an edge cloud network. This represents the set of physical links in an edge cloud network. , These represent the first and second parts of the edge cloud network, respectively. The physical node possesses the first The total amount and surplus of each type of resource, among which , , This represents the set of resource types that a physical node can provide, categorized into Type I, Type II, and Type III resources. ={Ⅰ, Ⅱ, Ⅲ}; , and Representing physical nodes The betweenness centrality, eigenvector centrality, and connectivity centrality; For physical nodes The set of neighboring nodes, ; , These represent the first and second parts of the edge cloud network, respectively. The total bandwidth resources and remaining bandwidth resources of each physical link. Indicates the first Packet loss rate of each physical link, of which ; and Representing physical links The difference between the mean difference in betweenness centrality and residual bandwidth; Indicates from the source physical node To the destination physical node The first A physical path, and Representing paths The remaining bandwidth and packet loss rate, , Represents the source physical node To the destination physical node The set of acyclic paths ,in ; （1） In formula (1), Indicates the first The betweenness centrality of physical nodes in an edge cloud network is a measure of how many times a node acts as a bridge between two other nodes for the shortest path. The more times a node acts as a bridge, the greater its betweenness centrality. A binary variable representing a physical node. arrive The number of shortest paths between them , express arrive Does the shortest path between them pass through the node? If the condition is met, the value is 1; otherwise, the value is 0. （2） In formula (2), Represents physical nodes Eigenvector centrality indicates that the importance of a node is influenced by the importance of its neighboring nodes; that is, the more important the nodes a node is connected to, the more important the node itself is. It is a constant. Indicates if yes The neighboring nodes, The value is 1 if it is 1, otherwise the value is 0. （3） In formula (3), Represents physical nodes Connectivity centrality, a measure of physical nodes The distance to all other nodes in the network, among which Represents a node To the node The number of hops in the shortest path between them; （4） In formula (4), Indicates physical link The centrality of the intermediaries, through the link The number of shortest paths is used as a metric to measure the importance of a link; （5） In formula (5), Indicates physical link The difference between the remaining bandwidth and the average remaining bandwidth of all links in the network; （6） In formula (6), Indicates the first Path The remaining bandwidth is equal to the minimum remaining bandwidth of all links in the path; （7） In formula (7), Indicates the first Path The packet loss rate.

2. The multi-task offloading method based on dual-agent deep reinforcement learning according to claim 1, characterized in that: In step S2, the node mapping agent module outputs the probability of all nodes being selected in the node feature matrix, selects the physical node with the highest probability for mapping, and updates the edge cloud network model module resources, specifically: S21: Input the node feature matrix into the input layer 1 of the node mapping agent module, and perform a normalization operation on the node feature matrix using the deviation normalization method; S22: Input the normalized node feature matrix into the first convolutional layer of the node mapping agent module, whereby the first convolutional layer generates the available resource vector for each node; S23: Input the available resource vector of each node into the Softmax layer 1 of the node mapping agent module, wherein the Softmax layer 1 transforms the available resource vector of each node into the probability of each node being selected; S24: Input the probability of each node being selected into the node mapping of the agent module's filter layer one. The filter layer one filters out nodes that do not satisfy the constraints of node resource type, node resource quantity, node one-to-one constraint, and flow conservation constraint, and records the probability of filtering out nodes that do not satisfy any of the above constraints as 0. S25: The output layer of the node mapping agent module recalculates the probability of each node in the edge cloud network model module being selected after being filtered by the filtering layer. The output layer outputs the probability corresponding to each node. The output layer selects the node with the highest output probability for mapping and updates the resources of the edge cloud network model module.

3. The multi-task offloading method based on dual-agent deep reinforcement learning according to claim 1, characterized in that: In step S5, the link mapping agent module outputs the probability of all paths being selected in the link feature matrix, selects the path with the highest probability for mapping, and updates the resources of the edge cloud network model module, specifically: S31: Input the link feature matrix into the second input layer of the link mapping agent module, and perform a normalization operation on the link feature matrix using the deviation normalization method; S32: Input the normalized link feature matrix into the second convolutional layer of the link mapping agent module, and the second convolutional layer generates the available resource vector for each path; S33: Input the available resource vector of each path into the Softmax layer 2 of the link mapping agent module, whereby the Softmax layer 2 transforms the available resource vector of each path into the probability of each path being selected. S34: The probability of each path being selected is input into the second filtering layer of the link mapping agent module. The second filtering layer filters out paths that do not satisfy the link direction constraint, flow conservation constraint, bandwidth capacity constraint, and packet loss rate constraint, and records the probability of filtering out a path that does not satisfy one of the above constraints as 0. S35: The output layer 2 of the link mapping agent module recalculates the probability of each path filtered by the filtering layer being selected. The output layer 2 of the link mapping agent module outputs all paths and their corresponding probabilities. The output layer 2 selects the path with the highest output probability for mapping and updates the edge cloud network model module resources.

4. A multi-task offloading system based on dual-agent deep reinforcement learning, characterized in that: The system includes an environment module, a node mapping agent module, and a link mapping agent module. The environment module contains an edge cloud network model module and a multi-task request model module, which are installed side-by-side. The environment module inputs the collected node feature matrix into the node mapping agent module. The node mapping agent module selects the node with the highest probability for each component in each multi-type task request and maps it, updating the edge cloud network model module resources. After the node mapping agent module has mapped all components of the multi-type task requests, the environment module inputs the collected link feature matrix into the link mapping agent module. The link mapping agent module selects the path with the highest probability for each multi-type task request link and maps it, updating the edge cloud network model module resources. After the link mapping agent module has mapped all links of the multi-type task requests, the environment module calculates the reward and loss functions for both the node mapping agent module and the link mapping agent module, and updates the parameters of both modules.

5. The multi-task offloading system based on dual-agent deep reinforcement learning according to claim 4, characterized in that: The multi-type task request model module is described as follows: This represents the set of all multi-type task requests; This represents a collection of multiple task types where mapping failed. This represents a multi-type task request. ; Indicates the first A collection of components in multiple types of task requests. Indicates the first A collection of links between components in a multi-type task request; Indicates the first In the multi-type task request, the first The first component request The number of each type of resource, among which , ; and They represent the first In the multi-type task request, the first The number of bandwidth resources requested for each link and the maximum tolerable packet loss rate, among which ; The total revenue from unloading multiple types of task requests is the difference between the revenue and cost of successfully mapped requests, as shown in formula (8): （8） in, Indicates the first request of each component The selling price per unit of each type of resource, Indicates the deployment of the first physical node The unit cost of this type of resource and These represent the selling price and cost price of the bandwidth resources, respectively. It is a binary variable, if the task request's first The first component is placed in the edge cloud network. Its value is 1 when it is on a physical node, otherwise its value is 0; It is a binary variable, if the task request's first Select the first component Its value is 1 when it is a certain type of resource, otherwise its value is 0; It is a binary variable, if the task request's first The first link is placed in the edge cloud network. Its value is 1 when there is a physical link; otherwise, its value is 0. Maximize the total revenue from multiple types of task requests The formula is: （9） The constraints are: （10） （11） （12） （13） （14） （15） （16） Wherein, formula (10) represents a one-to-one node constraint: a component in a multi-type task request can only be placed on a physical node in the edge cloud network; formula (11) represents a node resource type constraint: a component in a multi-type task request can only request one type of resource on a physical node in the edge cloud network; formula (12) represents a node resource quantity constraint: the amount of resources occupied by all requests... The total capacity of all resource types shall not exceed the number of physical nodes in the edge cloud network. The total capacity of each type of resource; Formula (13) represents the link direction constraint: a link in a multi-type task request can only select a physical link in one direction in the edge cloud network; Formula (14) represents the flow conservation constraint: the inflow of each physical node in the edge cloud network is equal to the outflow; Formula (15) represents the link bandwidth capacity constraint: the total bandwidth occupied by all multi-type requests in the edge cloud network does not exceed the total link bandwidth; Formula (16) represents the packet loss rate constraint: the packet loss rate of the selected mapped path in the edge cloud network is less than or equal to the tolerable packet loss rate of the link between multi-type task request components; Evaluation metrics for various task requests include total revenue (as shown in formula (8), blocking rate, average node resource utilization, and average bandwidth utilization: （17） （18） （19） in, BR This indicates the blocking rate of various task requests. The lower the blocking rate, the more task requests the edge cloud network can handle. Indicates the first The average node resource utilization rate of the edge cloud network is the highest among all resource types. A higher node resource utilization rate indicates that the node resources of the edge cloud network are being fully utilized. LU represents the average bandwidth resource utilization rate. In multi-type task requests, the shorter the path of the link between components mapped to the edge cloud network, the smaller the number of hops, and the lower the bandwidth utilization rate.

6. The multi-task offloading system based on dual-agent deep reinforcement learning according to claim 5, characterized in that: The node mapping agent module consists of, from bottom to top, an input layer 1, a convolutional layer 1, a softmax layer 1, a filter layer 1, and an output layer 1. The input layer collects information about each physical node in the edge cloud network module. t Feature matrix of various resource types The feature matrices of all physical nodes are combined into an edge cloud network module related to the first... t Node feature matrix of various resource types ,right Using the deviation standardization method, normalization is performed to obtain the normalized feature matrix. ,make The values ​​in the range are all between 0 and 1; No. The physical node is related to the first The feature matrix of the resource type is Includes the remaining resources of the node Node centrality Node eigenvector centrality and node connectivity centrality : （20） The relevant information about all physical nodes The feature matrix of the resource type is : （21） The formula for the deviation standardization method is: （22） in Represents the original list. This represents the list after calculating the standardized deviations. This represents the minimum value in the original list. This represents the maximum value in the original list; The convolutional layer one normalizes the feature matrix. Perform convolution operation The convolution kernel weight vector, The bias term and the ReLU function are used as activation functions. The convolutional layer generates a vector of available resources for candidate actions. The ReLU function, also known as a piecewise linear function, returns the value provided as input if the input is greater than 0. If the input is 0 or less, return 0; For the normalized node feature matrix Perform convolution operations: （23） The Softmax layer will use the available resource vector of candidate actions. Transformed into the probability of each action being selected : （24） The first filtering layer filters out all actions that do not meet the node resource type constraint, node resource quantity constraint, node one-to-one mapping constraint and flow conservation constraint, and obtains the updated available resource vector and candidate action set for the actions. The probability corresponding to the action of the output layer filtering out the filter layer is recorded as 0; the probability is recalculated to obtain a new probability distribution. for: （25）。 7. The multi-task offloading system based on dual-agent deep reinforcement learning according to claim 6, characterized in that: The link mapping agent module consists of, from bottom to top, an input layer 2, a convolutional layer 2, a softmax layer 2, a filtering layer 2, and an output layer 2. The second input layer collects the feature matrix of each link in the edge cloud network. Then, the feature matrices of all links are used to construct the link feature matrix of the edge cloud network. ,right Normalization was performed using the deviation standardization method to obtain the normalized link feature matrix. ,make The values ​​in the range are all between 0 and 1; No. Feature matrix of physical links Includes remaining bandwidth of the link Packet loss rate Intermediation centrality Difference between the average and remaining bandwidth : （26） The feature matrix of all physical links is: : （27） In the second convolutional layer The convolution kernel weight vector, The bias term and the ReLU function are used as activation functions. The convolutional layer generates a vector of available resources for candidate actions. : （28） The second Softmax layer will use the available resource vector of candidate actions. Transformed into the probability of each action being selected : （29） The second filtering layer filters out all path actions that do not meet the link direction constraint, flow conservation constraint, bandwidth capacity constraint and packet loss rate constraint, and records their probability as 0, thus obtaining the probability and action set of all actions after the update. The second output layer recalculates the action probabilities obtained from the filtering layer to obtain a new probability distribution. : （30）。 8. The multi-type task offloading system based on dual-agent deep reinforcement learning according to claim 7, characterized in that: The reward and loss functions for the environment module's computation node mapping agent module and link mapping agent module are as follows: The total revenue from node mapping is used as the reward function for the node mapping agent module, and the formula is: （31） The loss function for the node mapping agent module is: （32） The total reward from link mapping is used as the reward function for the link mapping agent module, and the formula is: （33） The loss function of the link mapping agent module is: （34）。

Images