Control device, learning method, communication system, communication method, and program

Abstract

The present disclosure is a control device that controls an allocation of a task to a modeled physical network, the control device comprising: an acquisition unit that acquires task information related to the task requested from a terminal device that can communicate, through a wireless network, with a prescribed base station within a communication range, network usage state information indicating a usage state of the physical network, and device information including a selection result of the base station by the terminal device; and a calculation unit that transmits, to the terminal device, a second learning model trained through machine learning by machine learning a first learning model by using input data that is the task information and the network usage state information, and correct answer data indicating a node to which the task is to be allocated, and machine learning the second learning model by using input data that is the network usage state information and the device information, and correct answer data that is the selection result of the base station by the terminal device.

Description

Control device, learning method, communication system, communication method and program

[0001] The present disclosure relates to network and cloud control technology, and more particularly to control for allocating tasks.

[0002] Rapid advances in communication technology have led to the emergence of numerous applications in various fields, including healthcare, smart cities, and manufacturing. These applications run on devices such as computers, smartphones, IoT devices, and vehicles. Because devices have limited computing power, application processing is often offloaded to high-performance external cloud servers. This processing is called cloud computing, and the offloaded computational requests are called tasks. Cloud computing is suitable for processing tasks that require large-scale computing power. However, it is not suitable for processing tasks that require low latency, because the geographical distance between the cloud server and the device increases latency.

[0003] To address this issue, edge computing, which offloads tasks to edge servers located close to devices, has been proposed. This reduces task latency by processing tasks on the edge server. With the increasing importance of mobile applications in recent years, multi-access edge computing (MEC) has also become increasingly important. MEC provides edge computing for mobile services by deploying computational resources within wireless networks. However, edge servers (MEC servers) are geographically distributed, and each server has limited resources and processing capabilities. Therefore, they are not suitable for tasks requiring large-scale computational resources or for processing large volumes of tasks.

[0004] To address the latency issues in cloud computing and the computational resource constraints in edge computing and multi-access edge computing, collaborative task offloading has been considered. In this approach, clouds, edge servers, and the network connecting them are considered a unified platform, providing integrated computational and communication resources. Furthermore, a task scheduler assigns the optimal server for offloading based on the characteristics of each task. For example, resources can be efficiently utilized by offloading tasks requiring large computational resources to cloud servers and tasks requiring low latency to edge servers.

[0005] Additionally, a cooperative task offloading method using cooperative multi-agent deep reinforcement learning has been proposed (Non-Patent Document 1). Reinforcement learning can immediately output efficient task offloading by learning in advance the relationship between the input network pattern and the output task offloading. In Non-Patent Document 1, an agent that has learned the optimal task offloading method is placed on each edge server. Furthermore, by providing a mechanism for multiple agents to learn cooperatively, selfish behavior of each agent can be prevented, improving the efficiency of task offloading.

[0006] A. Suzuki, M. Kobayashi, and E. Oki, "Multi-agent deep reinforcement learning for cooperative computing offloading and route optimization in multi cloud-edge networks," vol. 20, no. 4, pp. 4416-4434, 2023.

[0007] However, conventional methods are targeted at wired networks and do not take wireless networks into consideration. In wired networks, devices (terminal equipment) are connected to nearby edge servers via fixed routes, whereas in wireless networks, devices select a base station to connect to from among surrounding base stations. Conventional methods only control the efficient selection of an offload destination for tasks accepted by each edge server, but do not control the efficient selection of a base station to which each device connects. Since the nearby edge server varies depending on the base station selected, the optimal route from the device to each server and the resource load also differ. Therefore, when conventional methods are applied to networks that include wireless sections, inefficient selection of a base station to connect to can result in reduced resource utilization across the network and servers.

[0008] The present invention has been made in consideration of the above-mentioned problems, and aims to suppress a decrease in resource utilization efficiency across the entire network and server, even when communication from a device (terminal device) to a nearby edge server is via a wireless network.

[0009] In order to solve the above problem, the invention of claim 1 is a control device that controls the allocation of tasks to a modeled physical network constructed by nodes having edge nodes and cloud nodes, and includes an acquisition unit that acquires task information regarding the task requested from a terminal device that can communicate with a predetermined base station within its communication range via a wireless network, network usage information indicating the usage status of the physical network, and device information including the result of base station selection by the terminal device, and a calculation unit that machine-learns a first learning model using input data that is the task information and the network usage information and correct answer data indicating the node to which the task is to be assigned, and machine-learns a second learning model using input data that is the network usage information and the device information and correct answer data that is the result of base station selection by the terminal device, and transmits the second learning model learned by machine learning to the terminal device.

[0010] The present invention has the effect of suppressing a decrease in resource utilization efficiency across the entire network and servers, even when communication from a terminal device to a nearby edge server is via a wireless network.

[0011] 1 is a diagram illustrating an example of the overall configuration of a communication system in an embodiment of the present invention. FIG. 2 is a conceptual diagram illustrating a physical network of this embodiment. FIG. 3 is a hardware configuration diagram of a control device of this embodiment. FIG. 4 is a flowchart illustrating control of a task offload system. FIG. 5 is a flowchart illustrating control of the task offload system. FIG. 6 is a diagram illustrating definitions of notations used in a physical network model. FIG. 7 is a diagram illustrating definitions of notations used in a task model. FIG. 8 is a diagram illustrating definitions of notations for task offload problems. FIG. 9 is a diagram illustrating definitions of notations for reinforcement learning. FIG. 10 is a diagram illustrating algorithm 1. FIG. 11 is a diagram illustrating algorithm 2. FIG. 12 is a diagram illustrating each equation. FIG. 13 is a diagram illustrating each equation.

[0012] [Overview of the Embodiment] This embodiment proposes a cooperative task offloading method using cooperative multi-group multi-agent deep reinforcement learning (CMG-MADRL). Conventional methods only optimize task offloading. In contrast, this embodiment introduces two agent groups. The first is an agent group (each terminal device) that selects a base station in a wireless network, and the second is an agent group (each edge server) that selects an offload destination server. Each base station allocation agent is deployed to each terminal device (end device (also simply referred to as "device")) and aims to learn and optimize the selection of a base station to which each terminal device connects. Each server allocation agent is deployed to each edge server and aims to learn and optimize the offload destination of accepted tasks. Furthermore, by having all agents across the two groups learn cooperatively, base station selection and task offloading are optimized simultaneously.

[0013] [Overall Configuration of Communication System] An overview of a communication system that performs task offloading will be described below with reference to Figures 1 and 2. Figure 1 is a diagram showing an example of the overall configuration of a communication system according to an embodiment of the present invention.

[0014] As shown in Fig. 1, the communication system of this embodiment is constructed by a predetermined control device 50, multiple terminal devices (end devices) 10a and 10b, and a modeled physical network 140. Due to space limitations, Fig. 1 only shows a limited number of terminal devices 10a and 10b, edge nodes 121, 122, and 123, cloud nodes 131 and 132, and the control device 50, but there may be more of each than the number shown in Fig. 1. Furthermore, because the terminal devices 10a and 10b have the same functional units, the functional units of the terminal device 10b are omitted.

[0015] The control device 50 is a device that controls task allocation for a modeled physical network constructed by multiple nodes. In a real-world physical network 40, an edge server 20 (described later) generally also functions as the control device 50.

[0016] The control device 50 acquires task information and network usage status information from the modeled physical network 140 and performs task allocation control for the modeled physical network 140 .

[0017] The "task information" includes at least one of information on required computing resource demand, required traffic demand, and maximum allowable delay time.

[0018] "Network usage information" includes information regarding network topology, bandwidth, or metrics.

[0019] "Metrics" are indicators that serve as materials for evaluating candidates that can be selected as offload destinations, such as CPU (Central Processing Unit) usage rate, memory usage rate, and disk usage rate.

[0020] Specifically, the control device 50 formulates the problem of optimal task offloading and base station selection for a multi-cloud and multi-edge network, taking into account constraints on physical network usage, such as network topology and / or bandwidth. Here, "optimal offloading" is defined as a solution that maximizes server and link resource utilization efficiency and minimizes task delay while satisfying constraints on server capacity, link capacity, and task delay. The decision variables here are the allocation of task computing resources and the path between the terminal device and the assigned server. The control device 50 also proposes a task offloading and base station selection algorithm based on Cooperative Multi-agent Deep Reinforcement Learning (Coop-MADRL).

[0021] <Physical Network> The modeled physical network 140 is constructed by a plurality of terminal devices that request tasks, a plurality of edge nodes 121, 122, and 123, and a plurality of cloud nodes 131 and 132.

[0022] <Control Device> The control device 50 has an acquisition unit 51, a calculation unit 55, a control unit (determination unit) 57, and a transmission unit 59. Each of these units is a function or means realized by an operation commanded by a processor 101 (described later) of the control device 50 in accordance with a program installed in the control device 50. Note that in the overall configuration of the communication system, the <Control Device> column and the <Terminal Device> column explain the relationship between the control device 50 and the terminal device 10a, but because the terminal device 10a and the terminal device 10b have the same functional configuration, the relationship between the control device 50 and the terminal device 10b will be omitted.

[0023] The acquisition unit 51 acquires task information indicating the requested task details and network usage status information from the physical network 140, and outputs these to the calculation unit 55 and the control unit 57. That is, the acquisition unit 51 acquires task information related to a task requested by the terminal device 10a that can communicate with a predetermined base station within its communication range via a wireless network, network usage status information indicating the usage status of the physical network, and device information including the result of base station selection by the terminal device 10a.

[0024] During learning, the calculation unit 55 uses algorithms 1 to 3 described below to learn learning models M1 and M2 based on the task information and network usage status information acquired from the acquisition unit 51 and device information related to the terminal device 10a acquired from the transmission unit 19 of the terminal device 10a (e.g., control results such as the terminal device's selection of a base station), and transmits the learning model M2 trained by machine learning to the terminal device 10a. That is, the calculation unit 55 trains the learning model M1 by machine learning using input data that are the task information and network usage status information and correct answer data on nodes to be assigned tasks, and trains the learning model M2 by machine learning using input data that are the network usage status information and device information and correct answer data on the base station selection result by the terminal device 10a, thereby transmitting the learning model M2 trained by machine learning to the terminal device 10a.

[0025] During inference, the calculation unit 55 also acquires task information relating to a task requested by the terminal device 10a and network usage status information indicating the usage status of the physical network.

[0026] The control unit 57 has a (machine) learning model M1 and uses the learning model M1 to optimize task offloading during inference. That is, the control unit 57 serves as a (first) determination unit and uses the trained learning model M1 generated by machine learning using input data, which are task information and network usage information, and correct answer data for nodes to which tasks are to be assigned, to determine specific nodes to offload tasks based on the task information and network usage information. Note that the calculation unit 55 updates the model parameters of the learning model M1 by training the learning model M1.

[0027] The transmitting unit 59 controls task allocation to the physical network 140 based on the task allocation control information acquired from the control unit 57. That is, the transmitting unit 59 transmits tasks to specific nodes determined by the control unit 57. The processing of each unit will be described later.

[0028] The terminal device 10a has an acquisition unit 11, a control unit (determination unit) 17, and a transmission unit (selection unit) 19. Each of these units is a function or means realized by an operation commanded by a processor 101 (described later) of the terminal device 10a in accordance with a program installed in the terminal device 10a. The control unit 17 also acquires and stores a trained (machine) learning model M2 trained by the calculation unit 55 of the control device 50. Specific processing of each unit will be described later.

[0029] <Terminal Device> The terminal device 10a has an acquisition unit 11, a control unit (determination unit) 17, and a transmission unit 19. Each of these units is a function or means realized by an operation commanded by a processor 101 (described later) of the terminal device 10a in accordance with a program installed in the terminal device 10a.

[0030] The acquisition unit 11 acquires network usage information from the physical network 140 and outputs it to the control unit 17 .

[0031] The control unit 17 acquires and stores the trained (machine) learning model M2 trained by the calculation unit 55 of the control device 50. The control unit 17 uses the learning model M2 to optimize the selection of a base station based on the network usage status information acquired from the acquisition unit 11. That is, the control unit 17 serves as a (second) determination unit, and uses the trained learning model M2 generated by performing machine learning using input data that is the network usage status information and correct answer data that is the base station selection result by the terminal device 10a to determine a specific base station to connect to via the wireless network based on the network usage status information.

[0032] The transmitter 19 transmits task information of its own device (terminal device 10a) to the physical network 140. Furthermore, the transmitter 19 transmits base station control information to be connected to the physical network 140 based on control information of the base station (edge ​​node) to be connected, acquired from the control unit 17. That is, the transmitter 19 selects a specific base station determined by the control unit 17, and communicates with the specific base station. The specific processing of each unit will be described later.

[0033] [Explanation of Physical Network] Fig. 2 is a conceptual diagram showing a physical network of this embodiment. A physical network 40 is constructed by multiple terminal devices 10a and 10b that request tasks, multiple edge servers 21 and 22, multiple cloud servers 31 and 32, and a control device 50. The edge servers 21 and 22 are MEC (Multi-access Edge Computing) servers. The cloud servers 31 and 32 provide more sufficient computing resources than the edge servers 21 and 22, but incur additional delays compared to the edge servers 21 and 22.

[0034] Furthermore, the terminal device 10a can be connected to multiple edge servers 21 and 22 and multiple cloud servers 31 and 32 via a wireless access network an1. Similarly, the terminal device 10b can be connected to multiple edge servers 21 and 22 and multiple cloud servers 31 and 32 via a wireless access network an2. A core network cn is constructed between the edge server 21 and the edge server 22. The control device 50 is connected to the core network cn. The modeled physical network 140 shown in FIG. 1 corresponds to the physical network 40 shown in FIG. 2. Note that, due to space limitations, FIG. 2 shows only a limited number of terminal devices, edge nodes, cloud nodes, wireless access networks, and core networks, but there may be more of each than those shown in FIG. 2.

[0035] Hereinafter, the terminal devices 10a and 10b will be collectively referred to as "terminal device 10." The edge servers 21 and 22 will be collectively referred to as "edge server 20." The cloud servers 31 and 32 will be collectively referred to as "cloud server 30." The edge nodes 121, 122, and 123 will be collectively referred to as "edge node." The cloud nodes 131 and 132 will be collectively referred to as "cloud node." The edge node and cloud node will be collectively referred to as "node." Furthermore, the wireless access networks an1 and an2 will be collectively referred to as "wireless access network an."

[0036] The terminal device 10 may be a personal computer, a smartphone, a smartwatch, an IoT device, a home appliance, a communication device mounted on or installed in a mobile object, etc. Mobile objects include vehicles, aircraft, ships, robots, etc.

[0037] As shown in Fig. 2, every node has computing resources, such as an edge server 20 or a cloud server 30, that execute tasks on behalf of the terminal device 10. Every node is also connected to a base station r1 or r2, or a router r3 or r4, which forwards traffic to other nodes. The base stations r1 and r2 are base stations of a wireless network. Each edge server 20 has a control device 50 (see Fig. 1) for determining the optimal node to offload each task.

[0038] The terminal device 10 is configured by a computer and generates various tasks having various applications, each of which is configured with at least one of information on required computing resource demand, traffic demand, and maximum allowable delay.

[0039] Each terminal device 10 can calculate its own tasks within the terminal device 10 or offload tasks to an adjacent edge or cloud.

[0040] [Hardware Configuration of the Embodiment] FIG. 3 is a diagram showing the hardware configuration of the control device of the present embodiment.

[0041] 3, the control device 50 includes a processor 101, a memory 102, an auxiliary storage device 103, a connection device 104, a communication device 105, and a drive device 106. The hardware components constituting the control device 50 are connected to each other via a bus 107.

[0042] The processor 101 serves as a control unit that controls the entire control device 50, and includes various arithmetic devices such as a CPU (Central Processing Unit). The processor 101 reads various programs into the memory 102 and executes them. The processor 101 may also include a GPGPU (General-purpose computing on graphics processing unit).

[0043] The memory 102 has a main storage device such as a read-only memory (ROM) or a random access memory (RAM). The processor 101 and the memory 102 form a so-called computer, and the processor 101 executes various programs read onto the memory 102, thereby enabling the computer to realize various functions.

[0044] The auxiliary storage device 103 stores various programs and various information used when the processor 101 executes the various programs.

[0045] The connection device 104 is a connection device that connects an external device (for example, a display device 108 and an operation device 109 ) to the control device 50 .

[0046] The communication device 105 is a communication device for transmitting and receiving various types of information to and from other devices.

[0047] The drive device 106 is a device for loading a recording medium 106m. The recording medium 106m here includes media that record information optically, electrically, or magnetically, such as a CD-ROM (Compact Disc Read-Only Memory), a flexible disk, or a magneto-optical disk. The recording medium 106m may also include semiconductor memory that records information electrically, such as a ROM (Read Only Memory) or flash memory.

[0048] The various programs to be installed in the auxiliary storage device 103 are installed, for example, by setting the distributed recording medium 106m in the drive device 106 and reading the various programs recorded on the recording medium 106m by the drive device 106. Alternatively, the various programs to be installed in the auxiliary storage device 103 may be installed by being downloaded from a network via the communication device 105.

[0049] The terminal device 10, the edge server 20, and the cloud server 30 have the same hardware configuration as the control device, and therefore, a description thereof will be omitted.

[0050] [Processing of the Embodiment] <Control Procedure> Consider a discrete time process t. Assume that each device has one or more tasks during t∈[0,T]. At the beginning of each time process t, a device attempting to offload a task observes information about the status of surrounding base stations, MEC servers, cloud servers, and wired and wireless links. If only one base station is available, the task request is sent to that base station. If multiple base stations are available, the base station allocation agent selects the optimal base station and sends the task request to that base station. After the task request arrives at each base station, each base station observes information about the task requests it has received and the utilization status of all servers and wired links. Based on the observations, the server allocation agent calculates the optimal server to offload the task to. If multiple tasks arrive at each base station simultaneously during time process t, the proposed method repeats the process of determining the offload server in a first-in, first-out (FIFO) manner. Next, the proposed method calculates and updates the optimal route between servers in the wired network. Each base station then forwards the task to the optimal server via the selected route. The offloaded server executes the task and returns the results to the device. After a certain period of time, the process moves to the next time step t+1.

[0051] In this state, the following processing is executed. The control of the task offload system will be described using Figures 4 to 6. Figures 4 to 6 are flowcharts showing the control of the task offload system. Note that S1 to S23 are executed during machine learning, and S1 to S19 are executed during inference (when optimizing task offloading and base station selection).

[0052] S1: As shown in FIG. 4, at the start of t, the acquisition unit 11 of each device (terminal device 10) observes the network usage status by acquiring network usage status information indicating the network usage status (information on the status of surrounding base stations, MEC servers, cloud servers, and wired and wireless links).

[0053] S2: The control unit 17 determines whether there is one connectable base station or not.

[0054] S3: If there is more than one base station (NO), the control unit 17 calculates the optimal base station for offloading tasks based on the observation results of step S1 using the proposed method described below, which is located in each device.Then, the transmission unit 19 selects the optimal base station.Note that if there is only one base station, step S3 is omitted, and the transmission unit 19 selects this single base station.

[0055] S11: As shown in FIG. 5, at the start of each time process t, each task information arrives at a predetermined edge server 20 selected by the base station allocation agent.

[0056] S12: The acquisition unit 51 of each edge server 20 (control device 50) acquires task information and network usage information to observe task information and network usage. The task information includes at least one of information on required computing resource demand, traffic demand, and maximum allowable delay time. The network information is information on network usage, such as network topology and / or bandwidth.

[0057] S13: Based on the observation results of process S12, the calculation unit 55 of each edge server 20 (control device 50) calculates the optimal specific node for offloading the task using the proposed method deployed in the control device 50 (or the control unit 57) (for details, see [Proposed Method] below).

[0058] S14: If multiple tasks arrive at each edge server 20 simultaneously (YES), the method repeats the determination of the offload node using the first-in first-out (FIFO) method. If the tasks do not arrive simultaneously (NO), proceed to the next process.

[0059] S15: The calculation unit 55 of each edge server 20 (control device 50) aggregates traffic demand information between nodes, calculates and updates the optimal route between the nodes.

[0060] S16: The transmitter 59 of each edge server 20 (control device 50) transfers the task to the optimal node via the optimal route.

[0061] S17: Each node to which the task has been transferred executes the task and returns the result to the terminal device 10 that made the request.

[0062] S18: If a predetermined termination condition is met (YES), the control of the task offload system ends during inference, and proceeds to step S20 during learning. The predetermined termination condition is, for example, when task requests from each terminal device 10 have ended.

[0063] S19: If the predetermined termination condition is not met in the above process S18 (NO), after a certain time has elapsed (YES), the process returns to process S1 and the process is repeated at the next time process t+1.

[0064] It is assumed that the currently running task continues to consume resources of the offloaded node and the links through which the task passes until the result is returned to the terminal device 10. Therefore, in this embodiment, a task for which a request is accepted in time process t does not need to be completed by time process t+1.

[0065] S20: In process S18, if a predetermined termination condition is met (YES), the acquisition unit 51 records the task information, network usage information, control result (behavior of selecting a base station), and reward as learning data at each time from t=0 until the termination condition is met in a storage unit constructed by the memory 102, the auxiliary storage device 103, or the recording medium 106m.

[0066] S21: The calculation unit 55 randomly retrieves the learning data recorded in step S21 from the storage unit, and executes the learning algorithm to update the learning models M1 and M2 (model parameters).

[0067] S22: The calculation unit 55 determines whether the learning models M1 and M2 have sufficient performance. If the performance is not sufficient (NO), the process returns to S1. On the other hand, if the performance is sufficient (YES), the learning process ends.

[0068] <Network Model> Figure 7 defines the notation used in the physical network model. Consider a set of devices, E, and denote the total number of devices as |E|. Also consider a physical network graph, G(N,L), consisting of a set of physical nodes, N, and a set of physical links, L. Each physical node plays one of the following roles: an edge node in the edge layer, a router node in the network layer, or a cloud node in the cloud layer. The set of edge nodes is denoted as NE ⊂ N, the set of router nodes as NR ⊂ N, and the set of cloud nodes as NC ⊂ N. The numbers of these nodes are denoted as |N| for all physical nodes, |NE| for edge nodes, |NR| for router nodes, and |NC| for cloud nodes, respectively. For descriptive purposes, edge nodes and cloud nodes are collectively referred to as server nodes, i.e., NS = NE ∪ NC ⊂ N. An edge node represents an MEC server directly connected to a base station as a single unit, while a cloud node represents a cloud server. The link between the device (terminal equipment 10) and the base station is a wireless connection, and the access network between the device and the base station is excluded from this physical network graph G(N, L).

[0069] Assume that the Orthogonal Frequency Division Multiple Access (OFDMA) method is used for wireless communication between devices and base stations. The available bandwidth at each time step t is divided into |C| orthogonal channels, where C represents the set of channels available at each base station, and |C| is the number of channels available at each base station. Also, the server capacity of the i-th server node is expressed as follows:

[0070] This indicates the upper limit of the number of tasks that can be assigned. Here, the symbols shown below are natural numbers.

[0071] Here, we assign one CPU core to each task.

[0072] Let be equal to the number of CPU cores in the i-th server node. The bandwidth capacity of link (i,j) is given below.

[0073] This indicates the limit of the bandwidth resource. Here, the symbols shown below denote the set of positive real numbers.

[0074] <Task Model> Next, FIG. 8 shows definitions of notations used in the task model.

[0075] The task set is defined as K, and the total number of tasks is denoted as |K|. Each task k ∈ K is characterized as follows:

[0076] where T k ∈ [0, T] is the acceptance time of task k.

[0077] Below are the task types specific to each application.

[0078] d k comp is the required computational demand.

[0079] Below are the required uploads:

[0080] Below are the download traffic requirements:

[0081] The following is the maximum allowable delay for completing task k:

[0082] Also,

[0083] The following relationships are handled:

[0084] If task k can be completed by time t, the requirements of task k are fulfilled. Computational demand refers to the total amount of CPU cycles required to accomplish the task. Traffic demand refers to the amount of traffic generated by the task. When task k is accepted, it consumes computational and bandwidth resources on the system according to the amount of computational and traffic demand of task k. If task k is offloaded to one of the MEC servers through the wireless network, the bandwidth resources consumed on G(N,L) are considered to be 0.

[0085] <Proposed Method> Next, we will outline the proposed task offloading method. The proposed method consists of two steps: learning and control (inference). After learning, each agent continuously performs task offloading. Figure 9 shows the definition of the notation for the task offloading problem. In this method, there are four types of control variables, namely, the base station allocation variable Z, the channel allocation variable W, and the t , server assignment variable Y, routing variable X t The aim is to optimize

[0086] where the binary variable z kn ∈ {0, 1} is "1" if the device requesting task k is connected to the base station of edge node n, and "0" otherwise. The binary variable shown below is "1" if task k is assigned to channel c of edge node n at time step t, and "0" otherwise.

[0087] binary variable y kn∈ {0, 1} is "1" if the computation request of task k is assigned to server node n, and "0" otherwise.

[0088] The variables shown below represent the traffic demand m from the source node p to the destination node q. t pq indicates the proportion of times that link (i,j) passes through at t.

[0089] where Mt := {m t pq} denotes the traffic demand matrix between nodes p and q at time t, and m t pq represents the total traffic demand between nodes p and q at t.

[0090] The reward function is designed based on the objective function (Equation 1) shown in FIG.

[0091] where u t S and U t l , and , respectively denote the maximum server utilization and link utilization at time step t, and are defined as follows:

[0092]

[0093] The utilization rate of the i-th server is shown below.

[0094] The utilization rate of link (i,j) is shown below.

[0095] The following shows the end-to-end delay of task k.

[0096] λ denotes a weighting parameter that determines the importance ratio between resource efficiency and task delay.

[0097] In this proposed method, the following two agent groups are introduced:

[0098] Also, the base station allocation agent shown below learns how to determine the base station allocation for device e.

[0099] In addition, the server allocation agent described below learns how to determine the server to which an edge node n ∈ NE should offload a task accepted by that edge node.

[0100] This is expressed as (Equation 2) in FIG.

[0101] Similarly, e ∈ E and n ∈ N E Using the above, the observation set and action set of each agent group are expressed as shown in (Equation 3) in FIG.

[0102] <Modeling> The states are defined as follows:

[0103] agent

[0104] The observation for is defined as follows:

[0105] Also, agents

[0106] The observation for is defined as follows:

[0107] where K t and K. n,t denote the subset of tasks accepted at time step t and edge node n at time step t, respectively. That is, K n,t ⊆ K t ⊆ K. These are defined as (Equation 4) and (Equation 5) shown in FIG.

[0108] agent

[0109] The candidate action sets are expressed as follows:

[0110] Also, we denote the set of neighboring edge nodes of device e and the set of neighboring server nodes that can offload the accepted task at edge node n, respectively. Neighboring edge nodes are defined as base stations located within a certain distance from each device. Neighboring server nodes are defined as servers within a certain number of hops on the physical network graph G(N,L). If device e and edge node n do not receive a task at t, the agent selects the action of "do nothing."

[0111] <Learning procedure> During learning, each agent

[0112] Information i,t Based on the observation, each agent g i is action a i,t This is equivalent to determining the variables Z and Y. Specifically, the agent

[0113] determines the base station to which device e will connect, and the agent

[0114] determines the server to which the accepted task at edge node n is offloaded. Then, each base station determines the optimal channel allocation W for each task. t Then, the device transfers the task to the selected base station. As a next step, the proposed method aggregates traffic requests between nodes and calculates the optimal route between them. Then, each edge node transfers the task to the offload server through the optimal route, and the proposed method receives a reward r t By repeating these steps, the agent <o t , a t , r t Collect training samples that are combinations of o t and a t is shown as follows:

[0115]

[0116] Based on these samples, all agents learn. The learning algorithm is

[0117] The learning algorithm is the same for both. The technique used is that described in Reference 1. (Reference 1) P. Sunehag, G. Lever, A. Gruslys, WM Czarnecki, VF Zambaldi, M. Jaderberg, M. Lanctot, N. Sonnerat, JZ Leibo, K. Tuyls et al., "Value-decomposition networks for cooperative multi-agent learning based on team reward." in Proc. AAMAS, 2018, pp. 2085-2087. Algorithm 1 shows learning using CMG-MADRL. Line 1 shows the initialization of agent parameters. A series of procedures (lines 2-21) are executed repeatedly until learning is complete. Lines 3-4 show the generation of tasks and the initialization of environment parameters. A series of actions is called an episode, and each episode (lines 5-19) is executed repeatedly. In each episode, the agent <o t , a t , r t The time step of the network simulator is t. sim ∈ T sim and is reset at the beginning of each episode. Lines 6 to 12 show the observation and action procedures for the two agent groups. Each agent executes the procedures in parallel.

[0118] In lines 8 and 12, a random action is selected with probability ε, and an action that maximizes the action value function of reinforcement learning is selected with probability 1-ε. This is to avoid convergence to a local optimum. In line 10, edge n is selected as t sim If multiple tasks are accepted by the agent

[0119] selects one task in a FIFO manner. The two task subsets can be described as the following relationship:

[0120] In line 13, a t The task offloading is updated by Algorithm 2 according to the above. Line 14 shows the reward calculation. Lines 15-16 indicate the termination condition of agent learning. Lines 17-18 show the t sim If all accepted tasks are assigned in the next t sim Line 20 shows the storage in the replay memory M. Line 21 shows the history of episode transitions randomly taken from M.

[0121] will be trained.

[0122] <Environment update> Algorithm 2 shows the procedure for updating the environment. Four types of control variables are updated in Algorithm 2. Lines 1 and 2 are Z and W. t Line 2 shows the calculation of Y, and line 4 shows the calculation of U. Line 2 greedily allocates channels according to the order of arrival. If there are no channels available on the base station, the request is postponed to the next step. Line 3 shows the calculation of Y, and line 4 shows the calculation of U. t S The calculation of line 5 is M t The calculation of X is in line 6. t Line 7 shows the calculation of the delay.

[0123] <Reward calculation> Algorithm 3 is

[0124] The procedure for calculating the reward is as follows: Eff(x) is an efficiency function, which is defined as (Equation 6) in FIG.

[0125] This function is designed to be less efficient as x increases. If x < 1, it returns a positive value that depends on x, otherwise it returns a negative value. If x > 1, the efficiency decrease doubles. τ t ave indicates the average task delay efficiency and the average satisfaction of the delay at time step t, and is defined as (Equation 7) in FIG. 15.

[0126] where:

[0127] denotes the number of tasks executed at time step t.

[0128] u t+1 L > 1 or u t+1 If L > 1 is not satisfied, return "-3". Finally, return the reward clipped to the range of -3 to 3.

[0129] This concludes the explanation of the proposed method.

[0130] [Major Effects of the Embodiment] According to the present embodiment, efficient task offloading can be performed in a system including a wired network, a wireless network, cloud computing, and multi-access edge computing.

[0131] Specifically, we proposed a collaborative task offloading method based on collaborative multi-group multi-agent deep reinforcement learning. We prepared an agent group that selects a base station and an agent group that selects an offload destination server. Furthermore, by having all agents across the two groups learn collaboratively, we were able to simultaneously optimize base station selection and task offloading. This has the effect of preventing a decrease in resource utilization efficiency across the entire network and servers, even when communication from a device (terminal device 10) to a nearby edge server is via a wireless network.

[0132] [Supplementary Note] The present invention is not limited to the above-described embodiment, and may have the following configurations or processes (operations).

[0133] For example, the control device 50 can be realized by a computer and a program, and this program can be recorded on a (non-transitory) recording medium or provided via a communication network such as the Internet.

[0134] 10 Terminal device 10a Terminal device 10b Terminal device 11 Acquisition unit (second acquisition unit) 17 Control unit (determination unit, second determination unit) 19 Transmission unit (selection unit, second transmission unit) 21 Edge server 22 Edge server 31 Cloud server 32 Cloud server 40 Physical network 50 Control device 51 Acquisition unit (first acquisition unit) 55 Calculation unit 57 Control unit (determination unit, first determination unit) 59 Transmission unit (first transmission unit) 121 Edge node 122 Edge node 131 Cloud node 132 Cloud node 133 Cloud node 140 Modeled physical network

Claims

1. A control device that controls the allocation of tasks to a modeled physical network constructed by nodes having edge nodes and cloud nodes, comprising: an acquisition unit that acquires task information related to the task requested from a terminal device that can communicate with a predetermined base station within its communication range via a wireless network, network usage status information indicating the usage status of the physical network, and device information including the result of the selection of a base station by the terminal device; and a calculation unit that machine-learns a first learning model using input data that is the task information and the network usage status information and correct answer data indicating the node to be assigned a task, and machine-learns a second learning model using input data that is the network usage status information and the device information and correct answer data that is the result of the selection of a base station by the terminal device, and transmits the second learning model that has been machine-learned by the machine learning to the terminal device.

2. The control device according to claim 1, wherein the calculation unit calculates the nodes based on the observation results of the acquisition unit using a task offloading algorithm based on cooperative multi-agent deep reinforcement learning.

3. The control device according to claim 1, wherein the task information includes at least one of the following information: computing resource demand, traffic demand, and maximum allowable delay time.

4. The control device of claim 1, wherein the network usage information includes information regarding network topology, bandwidth, or metrics.

5. A learning method executed by a control device that controls the allocation of tasks to a modeled physical network constructed by nodes having edge nodes and cloud nodes, comprising: an acquisition process that acquires task information related to the task requested from a terminal device that can communicate with a predetermined base station within its communication range via a wireless network, network usage status information indicating the usage status of the physical network, and device information including the result of the selection of a base station by the terminal device; and a computation process that machine-learns a first learning model using input data that is the task information and the network usage status information and correct answer data indicating the node to be assigned the task, and machine-learns a second learning model using input data that is the network usage status information and the device information and correct answer data that is the result of the selection of a base station by the terminal device, thereby transmitting the second learning model that has been machine-learned by the machine learning to the terminal device.

6. A program for causing a computer to execute the method according to claim 5.

7. A communication system having a control device that controls task allocation to a modeled physical network constructed by nodes having edge nodes and cloud nodes, and a terminal device that can communicate with a predetermined base station within its communication range via a wireless network, wherein the control device has: a first acquisition unit that acquires task information related to the task requested from the terminal device and network usage status information indicating the usage status of the physical network; a first determination unit that determines a specific node to offload the task based on the task information and the network usage status information using a trained first learning model generated by performing machine learning using input data that is the task information and the network usage status information and ground truth data indicating a node to which a task is to be assigned; and a first transmission unit that transmits the task to the specific node, wherein the terminal device has: a second acquisition unit that acquires the network usage status information; a second determination unit that determines a specific base station to connect to via the wireless network based on the network usage information using a trained second learning model generated by performing machine learning using input data that is the network usage information and correct answer data that is a base station selection result by the terminal device; and a second transmission unit that selects the specific base station and communicates with the specific base station.

8. A communication method executed by a communication system having a control device that controls task allocation to a modeled physical network constructed by nodes having edge nodes and cloud nodes, and a terminal device that can communicate with a predetermined base station within its communication range via a wireless network, wherein the control device executes: a first acquisition process that acquires task information related to the task requested from the terminal device and network usage status information that indicates the usage status of the physical network; a first determination process that determines a specific node to offload the task based on the task information and the network usage status information, using a trained first learning model generated by performing machine learning using input data that is the task information and the network usage status information and ground truth data that indicates a node to which a task is to be assigned; and a first transmission process that transmits the task to the specific node; and the terminal device executes: a second acquisition process that acquires the network usage status information; A communication method which executes a second determination process of determining a specific base station to connect to via the wireless network based on the network usage information, using a trained second learning model generated by performing machine learning using input data which is the network usage information and correct answer data which is the result of base station selection by the terminal device, and a second transmission process of selecting the specific base station and communicating with the specific base station.

Images