Wireless network simulator based on deep reinforcement learning

Abstract

According to example embodiments, the device is configured as a Deep Reinforcement Learning, DRL, agent to simulate the behavior of the network components. The DRL agent takes as input the network state and the user traffic. It generates the next network state and the user performance. The training algorithm of the simulator is configured for the DRL agent and it is derived for handling the time-dependent properties in the network components. The simulator uses a training algorithm such that it enables robust inference with a limited number of transitions collected with real network components and users. It is derived by using state augmentation with an autoencoder architecture. It is also configured by using a local regression reward estimation algorithm, e.g. with Gaussian processes.

Description

Technical Field

[0001] This disclosure generally relates to the field of wireless communications. In particular, this disclosure relates to devices and related methods for simulating wireless networks, as well as computer programs. Background Technology

[0002] Modern wireless networks have a large number of base stations deployed in the Radio Access Network (RAN), such as eNBs in LTE and gNBs in 5G NR, to wirelessly connect mobile users to the network; distributed network functions (NFs) in the Core Network (CN) to support nationwide mobility services; and network switches in the Transport Network (TN) to connect the RAN and CN. These network components are highly complex and difficult to model mathematically accurately. For example, a base station consists of multiple protocol stacks (e.g., PHY, MAC, RLC, RRC), ranging in number from BTS and eNB in ​​LTE to gNB in ​​NR. As a result, it is difficult to formulate the radio transmission between mobile users and the BS. Because these components are interconnected—for example, radio transmission in the RAN, service transmission in the TN, and packet processing in the CN—the entire network becomes extremely complex, and formulating it mathematically in an accurate manner is very challenging.

[0003] Existing network management solutions are typically derived within network simulators, which are often constructed based on (simplified) mathematical formulas of cellular networks. For example, user scheduling algorithms can be derived in a RAN simulator to improve the throughput of the Base Station (BS), which simulates mobile user traffic arrival and channel dynamics. NS-3, for instance, is a widely used network simulator where events are inserted, executed, monitored, and decoupled to simulate transmissions across different networks. However, it abstracts the very detailed processing of transport packets at the physical layer. As a result, such simulators cannot accurately simulate real, large-scale cellular networks. These model-based approaches suffer from the significant differences between network simulators and real networks. In other words, these derived solutions may suffer from degradations in performance and effectiveness. Consequently, they fail to efficiently optimize network performance in the increasingly complex networks moving towards 5G and beyond. Summary of the Invention

[0004] The independent claims define the scope of protection sought by the various exemplary embodiments of this disclosure. Exemplary embodiments and features (if any) described in this specification that do not fall within the scope of the independent claims should be interpreted as examples useful for understanding the various exemplary embodiments of this disclosure.

[0005] An example embodiment of the simulator device includes at least one processor and at least one memory including computer program code. The at least one memory and the computer program code are configured to utilize the at least one processor to enable the simulator device to: configure deep reinforcement learning (DRL) agents. Each DRL agent is configured to simulate the operation of components of a wireless network. Each DRL agent is configured to represent a state of wireless network information and component information. The DRL agents are configured to receive and execute training data, thereby expanding the state and estimating the reward. The simulator device is configured to interconnect the DRL agents to simulate real connections between components in a wireless network and execute the DRL agents based on the state as input to simulate the wireless network online. For example, the simulator device can utilize high-fidelity simulation of end-to-end networks. A distributed model-free simulation framework with DRL technology can be obtained. Robust training algorithms for agents with state expansion and reward estimation can be obtained.

[0006] In the example embodiments, alternatively or in addition to the example embodiments described above, each DRL agent is configured to emulate an individual component in a real wireless network, wherein the component includes the individual component, and the wireless network includes a real wireless network implemented in a specific geographical area. The network and user agents can emulate the behavior of real network components and users.

[0007] In another example embodiment, alternatively or in addition to the example embodiments described above, the state includes an internal state representing technical internal information of the component, and each DRL agent is configured to receive the internal state as input. Technical characteristics of individual components can be considered for simulation.

[0008] In the example embodiments, alternatively or in addition to the example embodiments described above, the state includes an external state representing the state of wireless network users and the state of other DRL agents, and each DRL agent is configured to receive the external state as input. For example, interactions with other network components and user equipment can be considered in the simulation.

[0009] In the example embodiments, alternatively or in addition to the example embodiments described above, each DRL agent is further configured to output a next internal state based on the stated state, which represents the network configuration of the DRL agent based on the stated state. This can simulate the interaction and dynamics of network components.

[0010] In the example embodiments, alternatively or in addition to the example embodiments described above, the user agent is further configured to emulate the operation of user equipment in the wireless network, and the user agent is configured to generate data services and user performance within the wireless network. For example, devices in the network can simulate user actions.

[0011] In the example embodiments, alternatively or in addition to the example embodiments described above, the DRL proxy is configured to receive data services and user performance within the wireless network. User interactions within the network can also be simulated.

[0012] In the example embodiments, alternatively or in addition to the example embodiments described above, the user equipment includes a mobile device.

[0013] In the example embodiments, alternatively or in addition to the example embodiments described above, the device is further configured to use an autoencoder to augment the state for augmentation. Robust training algorithms can be implemented.

[0014] In the example embodiments, alternatively or in addition to the example embodiments described above, the autoencoder includes a variational autoencoder (VAE).

[0015] In the example embodiments, alternatively or in addition to the example embodiments described above, the device is further configured to use distributional regression for reward estimation. Robust training algorithms can be configured for training.

[0016] In the example embodiments, alternatively or in addition to the example embodiments described above, the device is configured for Gaussian process regression (GPR) for reward estimation.

[0017] In the example embodiments, alternatively or in addition to the example embodiments described above, the device is configured in an expanded state, such that a large number of states are obtained for the DRL agent; and

[0018] The device is configured to reward estimates of a large number of states through distributional regression based on state similarity. The simulator device enables robust training configurations for simulation.

[0019] In the example embodiments, alternatively or in addition to the example embodiments described above, the wireless network includes a mobile network.

[0020] In the example embodiments, alternatively or in addition to the example embodiments described above, the DRL agent is configured as a base station, switch, or data processor unit that emulates a wireless network.

[0021] Example embodiments of simulator devices include components for performing the following:

[0022] Configure deep reinforcement learning DRL agents, where each DRL agent is configured to simulate the operation of components of a wireless network, and each DRL agent is configured to represent the state of information about the wireless network and the components.

[0023] The DRL agent receives and executes the training data, enabling the augmentation of the state and estimation of the reward;

[0024] Interconnect DRL proxies are used to simulate real connections between components in a wireless network; and

[0025] DRL agents are executed based on the state as input to simulate wireless networks online.

[0026] Example embodiments of a method include:

[0027] Configure deep reinforcement learning DRL agents, where each DRL agent is configured to simulate the operation of components of a wireless network, and each DRL agent is configured to represent the state of information about the wireless network and the components.

[0028] The DRL agent receives and executes training data, in which the state is augmented and the reward is estimated.

[0029] Interconnect DRL proxies are used to simulate real connections between components in a wireless network; and

[0030] DRL agents are executed based on the state as input to simulate wireless networks online.

[0031] In the example embodiments, alternatively or in addition to the example embodiments described above, offline training of the DRL agent is further performed before state expansion.

[0032] In the example embodiments, alternatively or in addition to the example embodiments described above, each DRL agent is configured to emulate an individual component in a real wireless network, wherein the component includes the individual component, and the wireless network includes a real wireless network implemented in a specific geographic area.

[0033] In the example embodiments, alternatively or in addition to the example embodiments described above, the method is configured for model-free simulation.

[0034] Example embodiments of a computer program product include program code configured to perform the methods according to any of the example embodiments above when the computer program product is executed on a computer. Attached Figure Description

[0035] The accompanying drawings, which are included to provide a further understanding of the exemplary embodiments and form part of this specification, illustrate the exemplary embodiments and, together with the description, help to explain the principles of the exemplary embodiments. In the drawings:

[0036] Figure 1 An example embodiment of the subject matter described herein is illustrated, showing a device configured to simulate a wireless network;

[0037] Figure 2 An example embodiment of the subject matter described herein is illustrated, which illustrates a simulator system;

[0038] Figure 3 An example embodiment of the subject matter described herein is illustrated, which illustrates a flowchart representation of a method for simulating a wireless network;

[0039] Figure 4 A block diagram of a network proxy according to an example embodiment is shown;

[0040] Figure 5 A block diagram of a user agent according to an example embodiment is shown;

[0041] Figure 6 A block diagram of a training algorithm according to an example embodiment is shown;

[0042] Figure 7 The diagram illustrates a state extension according to an example embodiment; and

[0043] Figure 8 A block diagram illustrating reward estimation according to an example embodiment is shown.

[0044] In the accompanying drawings, similar reference numerals are used to designate similar parts. Detailed Implementation

[0045] Reference will now be made in detail to exemplary embodiments, examples of which are illustrated in the accompanying drawings. The detailed description provided below, in conjunction with the accompanying drawings, is intended to describe the present example and is not intended to represent only the forms in which this disclosure can be constructed or utilized. This description illustrates the function of the example and the sequence of steps for constructing and operating the example. However, the same or equivalent functions and sequences can be implemented through different exemplary embodiments.

[0046] Figure 1 This is a block diagram of a device 100 configured to simulate a wireless network according to an example embodiment. Device 100 may be referred to as simulator device 100.

[0047] The simulator device 100 may include one or more processors 101 and one or more memories 102 including computer program code 103. The simulator device 100 may also include at least one communication interface 104 and other components, such as input / output modules. Figure 1 (Not shown in the image).

[0048] According to an example embodiment, the at least one memory 102 and computer program code 103 are configured to utilize the at least one processor 101 to cause the simulator device 100 to configure a DRL agent to simulate the behavior of network components. The DRL agent takes network state and user traffic as input. It generates the next network state and user performance. A training algorithm is derived for the device 100 configured for the DRL agent to handle time-related properties in the network components.

[0049] Furthermore, the simulator device 100 is configured to train algorithms that enable robust inference using real network components and a limited number of transitions collected by the user. This is derived using state augmentation via an autoencoder architecture. It is also configured using a reward estimation algorithm based on local regression, such as utilizing a Gaussian process.

[0050] Device 100 is accordingly configured as a deep reinforcement learning-based network and user simulator. Instead of a conventional model-based simulator design, Device 100 is configured to derive a model-free method to simulate cellular networks in an end-to-end manner. Device 100 enables high scalability for incrementally deployed network infrastructure, allows efficient training of distributed DRL agents, and handles extreme network dynamics.

[0051] The simulation device 100 features a distributed model-free framework comprising multiple DRL agents to simulate the behavior of different network components. This framework allows for hot-plugging and unplugging of DRL agents and can be efficiently scaled according to changes in network topology. Compared to conventional model-based methods, it achieves high-fidelity behavioral simulation by leveraging a high-dimensional deep neural network architecture.

[0052] Although the simulator device 100 may be depicted as including only one processor 101, the network node device 100 may include more processors. In the example embodiment, the memory 102 is capable of storing instructions, such as the operating system and / or various applications.

[0053] Furthermore, processor 101 may be capable of executing stored instructions. In example embodiments, processor 101 may be embodied as a multi-core processor, a single-core processor, or a combination of one or more multi-core processors and one or more single-core processors. For example, processor 101 may be embodied as one or more of a variety of processing devices, such as a coprocessor, microprocessor, controller, digital signal processor (DSP), processing circuitry with or without an accompanying DSP, or various other processing devices including integrated circuits, such as, for example, application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), microcontroller units (MCUs), hardware accelerators, application-specific computer chips, or the like. In example embodiments, processor 101 may be configured to perform hard-coded functionality. In example embodiments, processor 101 is embodied as an executor of software instructions, wherein, when the instructions are executed, the instructions may specifically configure processor 101 to perform the algorithms and / or operations described herein.

[0054] The memory 102 may be embodied as one or more volatile memory devices, one or more non-volatile memory devices, and / or a combination of one or more volatile memory devices and non-volatile memory devices. For example, the memory 102 may be embodied as a semiconductor memory (such as a mask ROM, PROM (programmable ROM), EPROM (erasable PROM), flash ROM, RAM (random access memory), etc.).

[0055] Device 100 may be embodied in, for example, a computer. Alternatively or additionally, the computer may be a cloud computing system with various distributed units.

[0056] When device 100 is configured to perform certain functionalities, one or more components of device 100—such as at least one processor 101 and / or memory 102—can be configured to perform that functionality. Furthermore, when at least one processor 101 is configured to perform a certain function, that functionality can be implemented using, for example, program code 103 included in memory 102. For example, if simulator device 100 is configured to perform operations, at least one memory 102 and computer program code 103 can be configured to enable network node device 100 to perform that operation using at least one processor 101.

[0057] Some terms used in this document may follow the naming scheme of 4G or 5G technologies in their current form. However, the terminology should not be considered limiting, and it may change over time. Therefore, the following discussion of any example embodiments can also be applied to other technologies. Deep Reinforcement Learning (DRL) may involve a subfield of machine learning that combines reinforcement learning (RL) and deep learning. RL considers the problem of computationally enabling agents to learn to make decisions through trial and error. Deep RL incorporates deep learning into the solution, allowing the agent to make decisions from unstructured input data without manually engineering the state space. Deep RL algorithms are capable of accepting very large inputs, such as every pixel rendered to the screen in a video game, and deciding what actions to take to optimize an objective, such as maximizing the game score. DRL has been used in a wide variety of applications, including but not limited to simulation, design, etc. Gaussian Process Regression (GPR) may involve nonparametric Bayesian regression methods applicable to the field of machine learning. GPR has several benefits, handling small datasets well and providing a measure of uncertainty in predictions.

[0058] Figure 2 An example embodiment of the subject matter described herein is illustrated, which illustrates a network simulator of simulator device 100.

[0059] like Figure 2The network simulator depicted consists of multiple interconnected DRL agents (200_1, 200_2, 200_3, 200_4). These agents 200 are designed using DRL technology to simulate the behavior of real wireless network components (203_1, 203_2, 203_3, 203_4). The simulator system provides training 202_1 for user agent 201 and training 202_2 for DRL agents 200. Each agent 200 takes both its internal and external states as input. Each agent 200 generates actions such as user performance and the next internal state based on the inputs and configuration. User agents (201_1, 202_2, 201_3) simulate user performance in the network simulation. The agents 200 are interconnected, meaning that when taking the next action, agent 200 will observe the states of other agents. The connections between agents 200 reflect the interdependencies of real network components, such as X2 interfaces between eNBs, and fiber optic connections, such as between eNBs and switches.

[0060] Figure 3 An example embodiment of the subject matter described herein is illustrated, showing a flowchart representation of the method. The general training and simulation operations of a network simulator are described in... Figure 3 The example is illustrated in the embodiment.

[0061] At operation 300, each agent 200 is designed to simulate the behavior of the individual network component 203. The state space and action space are defined accordingly.

[0062] At operation 301, the offline training phase is performed. The DRL agent 200 is trained independently using an offline dataset or through online interaction with the real network component 203. The user agent 201 can also be trained at this phase.

[0063] At operation 302, to enhance the robustness of DRL agent 200 when dealing with extremely diverse network dynamics, the state is expanded. This can be implemented, for example, using autoencoder techniques to traverse the state and multiple possibilities for agents 200 and 201.

[0064] At operation 303, DRL agents 200 are included on demand during the online simulation phase. Their connections are determined based on the real connections of their simulated network components 203 in the real network.

[0065] During the online simulation phase at operation 304, agent 200 is executed based on inputs, such as network traffic. DRL agent 200 takes both internal and external states as inputs. Agents 200 and 201 generate the next internal state and user performance-related information.

[0066] This method can be implemented by, for example, a simulator device 100.

[0067] Figure 4 An example of a network proxy 200 according to an exemplary embodiment is illustrated. The network proxy may also be referred to as a DRL proxy 200.

[0068] DRL agent 200 is configured to emulate, as... Figure 2 The behavior of network component 203 is illustrated. Agent 200 is configured to react to inputs (e.g., user traffic) and transition to the next state as a real network component 203 would. The behavior of network component 203 is time-dependent. Therefore, decisions made at this time affect not only the next output but also other states. Thus, the problem of behavioral simulation can be formulated as a Markov Decision Process (MDP). Considering the complex mechanisms in each network component 203 (e.g., BS and switch), agent 200 is configured to leverage DRL techniques to solve high-dimensional problems, where the state and action spaces can be hundreds or thousands of dimensions. Conventionally, the agent is designed to maximize the cumulative reward of the environment. In the example embodiment, DRL agent 200 is configured to simulate and / or mimic the environment (e.g., network component 203 or user). To this end, DRL agent 200 is configured to react like the environment, not only instantaneously but also statistically. Agent 200 includes a) a state space, b) an action space, c) a reward function, and d) functionality.

[0069] a) The state space is configured to provide input to DRL agent 200, representing comprehensive information about the network. The state space comprises two parts: internal state 400 and external state 401. Internal state 400 consists of states that can only be used and seen within that specific network component 203. For example, internal state 400 may include eNB runtime metrics, RLC buffers, and retransmission states. External state 401 includes user states and states from other agents 200. User states indicate external input from the user, such as traffic arrival and channel conditions, mobility, etc. These user states change at different analog time slots. Furthermore, each agent 200 defines states visible to other agents 200, such as interference information between X2-connected eNBs and user data transmission between the RAN and TN.

[0070] b) The action space is configured to allow DRL agent 200 to transition to the next internal state 400. For example, the action space of the DRL agent 200 is the next internal state.

[0071] c) The reward function is configured to guide the training of the neural network. The objective of the DRL agent 200 can be to maximize the cumulative reward. The goal is to simulate the behavior of network component 203. Therefore, the reward is the negative difference between the generated next internal state and the true internal state obtained from the experimental traces.

[0072] d) Functionality: The DRL agent 200 needs to adapt to different connections with other agents 200, such as when the agent is connected to multiple agents (BSs or switches) in the network. Conventional neural network architectures (convolutional, fully connected) have fixed input and output sizes, which cannot be used in this scenario. Therefore, the example embodiment is configured as a recurrent neural network (RNN), which is the architecture of the DRL agent 200. RNNs allow for flexible input and output sizes.

[0073] Figure 5 An example of a user agent 201 according to an example embodiment is illustrated.

[0074] like Figure 5 As shown, user agent 201 is configured to simulate user service generation and performance. User agent 201 is configured to observe the state from agent 200, generate user services, and achieve performance as a real user would. Since the state from agent 200 can be very high-dimensional, this is addressed by utilizing DRL techniques, as agent 200 has previously done. User agent 201 is configured for the following state space, action space, reward function, and functionality.

[0075] a) The state space is configured with observable states 403 and internal states 400 from all applications of agent 200. For example, it can observe states 403 (RLC buffer, retransmission state) from agent 200 in the RAN and states 403 (data transmission) from agent 200 in the TN. Internal states 400 are states used only internally, such as the remaining data size for transmission and the service mode of the user application. These states 400 and 403 help estimate the performance of the user agent 201, such as throughput, latency, and jitter.

[0076] b) The action space is configured for the performance of User Agent 201, such as throughput, latency, and reliability.

[0077] c) The reward function is configured to simulate the user's behavior. Therefore, the reward is the negative difference between the generated next internal state and performance and the true internal state and performance obtained from the experimental trace.

[0078] d) Functionality: When users are served by an end-to-end network service involving multiple agents 200, user agent 201 requires flexible input and output sizes. Similar to network agent 200, user agent 201 is configured with an RNN architecture.

[0079] Figure 6 A block diagram illustrating the training algorithm and operation according to an example embodiment is shown.

[0080] Agents 200 and 201 may need to be trained before they can be used for simulation. Typically, these agents are trained using offline-collected datasets that differ significantly from the domains formulated by the datasets and the real networks. Online learning methods allow agents to be trained directly using real network components. However, when operating network components under normal conditions, the state space in the agent may not be fully traversed. As a result, when some exceptional or disruptive events occur, the states input to the agent may not have been seen before. Therefore, the generated actions may be relatively random, and behavior under such conditions cannot be well simulated. To address this problem, the example embodiment is configured to use an autoencoder to augment the state space from existing transitions and to use Gaussian process regression (GPR) to determine the rewards for these augmented states accordingly, such as... Figure 6 As shown in the figure. Figure 6 The diagram illustrates the training process 600, with VAE base state expansion 602 fed to agents 200 and 201. GP local reward estimation 602 follows agents 200 and 201 and is fed to the training process 600.

[0081] The DRL agent 200 trained for existing changes may not be robust to other unseen states, since the DRL agent 200 is trained using observed changes. Figure 6 The example implementation is configured to augment the observed transitions, for example, by expanding the transition dataset. The challenge then becomes that there is no reward when states are generated randomly, since accurate rewards are likely only available from real networks / systems. Therefore, Figure 6 The example implementation is configured to generate a similar state from the observed transitions using an autoencoder architecture. In order to obtain a reward, Figure 6 The example implementation uses GPR to estimate or predict rewards. This example implementation can improve its robustness when the DRL agent 200 mimics the real network component 203.

[0082] Figure 7 A block diagram of state extension 601 according to an example embodiment is shown.

[0083] State augmentation 601 is configured by leveraging an autoencoder architecture to augment the finite states collected from the real network component 203. An autoencoder is a form of unsupervised learning, typically consisting of an encoder 701 and a decoder 702, both based on a neural network architecture. The encoder 701 encodes highly ambiguous input data 700 into a concise vector (denoted as latent vector 703). The decoder 702 is designed to decode the latent vector 703 back to the original input data 704. Because no labels are required—in other words, unsupervised learning—the loss function 705 for training the encoder 701 and decoder 702 is the difference between the input data 700 and the recovered data 704. Figure 7 As shown, state expansion 601 is configured as a variational autoencoder (VAE) to encode 701 and decode 702 the states 700 and 704 collected during the transition. This introduces regularization in the loss function 705 and improves regularization in the latent space 703 (similar latent vectors are decoded as similar input data). According to an example embodiment, state expansion 601 is configured to formulate a dataset using all state spaces collected from network component 203 and feed them into the VAE network for offline training. Variable vectors (μ, σ) are generated after the training phase, where the latent 703 is sampled from a distribution defined by the variable vectors. After training is complete (e.g., if it is generative), state expansion 601 slightly alters the encoded latent vectors from the dataset and feeds the deviated latent vectors into decoder 702 in the VAE. The output of decoder 702 will indicate a distribution similar to the previously collected input states 700. Decoder 702 decodes the deviated latent vectors. When the vector is slightly modified from the encoded latent vector, the decoder 702 generates an output that is similar to the output of the original encoded vector. This helps generate new states similar to the observed states, which can be configured as state expansion. State expansion 601 then uses these as new states 704, and said states are expanded into a buffer for further training. In this way, state expansion 601 can expand the state space used to train network agent 200 or user agent 201.

[0084] Figure 8 The diagram illustrates a block diagram of reward estimation 602 according to an example embodiment.

[0085] Using state expansion 601, the simulation device 100 can acquire a large number of states, for example, more states than the real states collected from network component 203 or the user. However, the rewards corresponding to these states under different actions are undetermined. Typically, a true reward can only be obtained when the action is brought into network component 203 in that exact state. According to the example embodiment, one objective is to achieve robust simulation of the DRL agent 200, for example, ensuring that the agent 200 will not generate unusual or unexpected actions under different states. Therefore, one may roughly estimate the reward without a precise value. Constructing a reward estimator that estimates all rewards under all states and actions requires a large number of real transitions and fails to provide accurate local estimation. Figure 7As shown in the example embodiment, reward estimation 602 is configured to use a distributional regression model 800 to construct a regression model with only local data points, such as similar state-action pairs. For example, Gaussian process regression (GPR) uses a kernel function to approximate the distribution of reward values. Reward estimation 602 is configured to feed GPR to take both state and action as input and reward value as output. Because the number of local data points used in GPR is limited, regression training of GPR is locally fast and accurate. In this way, the reward value of the augmented state is coarsely estimated. In the example embodiment, reward estimation 602 trains the distributional regression model using existing transitions 801, such as state-action as input and reward as the underlying ground truth label. These transitions are collected from real network component 203 or users; however, these transitions are limited. Once the distributional regression model is well trained (processed offline), reward estimation 602 is configured to feed the augmented states 802 into model 800, which will generate reward estimates for them. Finally, reward estimates 602 obtain sufficient transitions, either collected from a real network or generated from state augmentation 601. Simulated devices 100 can use these to train DRL agents 200 to achieve more robust performance when they perform inference.

[0086] The following describes an example embodiment of a network simulator for end-to-end network slicing. In this example, network simulator device 100 is applied to an end-to-end network slicing scenario. The entire network is considered to consist of X BSs, Y switches, and Z core network functions. The interconnections between these network components are also given. There are N network slices running on this end-to-end network.

[0087] To this end, multiple DRL agents 200 are created and trained to emulate network component 201 and network slices. For BS simulation, the internal state space is configured as follows: [CQI, RLC buffer, HARQ state, AMC, MAC user to PRB mapping, scheduler, activity queue management state, channel fading], and the external state is [slice service, user location, mobility].

[0088] It is worth noting that these states can be extended as needed for the simulation. The aim is to obtain comprehensive information about network component 203.

[0089] For slice simulation, the internal state space is configured as: [TCP / IP transport buffer, packet retransmission, application state, slice template metrics]. Then, the action space is defined based on metrics from the slice service level protocol, such as [slice throughput, throughput per user, latency, reliability, etc.].

[0090] Once the state space and action space are well defined, the simulator device 100 can use their corresponding network components 201 and network slices to train these DRL agents 200.

[0091] The device 100 may include components for performing any aspect of the methods(s) described herein. According to an example embodiment, the components include at least one processor and a memory including program code, at least one processor, and program code configured to perform any aspect of the method when executed by said at least one processor.

[0092] The functionality described herein can be implemented, at least in part, by one or more computer program product 103 components, such as software components. According to an example embodiment, device 100 includes a processor 101 configured with program code that, when executed, performs example embodiments of the described operations and functionality. Alternatively or additionally, the functionality described herein can be implemented, at least in part, by one or more hardware logic components. For example, and without limitation, illustrative types of hardware logic components that can be used include field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), complex programmable logic devices (CPLDs), and graphics processing units (GPUs).

[0093] Any range or device values ​​given herein may be extended or modified without losing the desired effect. Furthermore, any example embodiment may be combined with another example embodiment unless expressly permitted.

[0094] Although the subject matter has been described in language specific to structural features and / or actions, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. Rather, the specific features or actions described above are disclosed as examples of implementing the claims, and other equivalent features and actions are intended to be within the scope of the claims.

[0095] It will be understood that the benefits and advantages described above may relate to one example embodiment or several example embodiments. The example embodiments are not limited to example embodiments that solve any or all of the stated problems, or example embodiments that have any or all of the stated benefits and advantages. It will be further understood that references to “one” may refer to one or more of those items.

[0096] The methods described herein can be performed in any suitable order, or simultaneously where appropriate. Additionally, individual boxes can be removed from any method without departing from the spirit and scope of the subject matter described herein. Aspects of any of the example embodiments described above can be combined with aspects of any other example embodiments described to form further example embodiments without losing the desired effects.

[0097] The term “comprising” is used herein to mean including the identified method, box, or element, but such box or element does not include an exclusive list, and the method or apparatus may include additional boxes or elements.

[0098] It will be understood that the above description is given by way of example only, and various modifications can be made by those skilled in the art. The specification, examples, and data above provide a complete description of the structure and use of exemplary embodiments. Although various exemplary embodiments have been described above with a degree of specificity, or reference has been made to one or more individual exemplary embodiments, those skilled in the art can make various changes to the disclosed exemplary embodiments without departing from the spirit or scope of this specification.

Claims

1. A device for simulating a wireless network, comprising: At least one processor; and At least one memory containing computer program code; At least one memory and computer program code are configured to use the at least one processor to enable the device to: Configure deep reinforcement learning DRL agents, where each DRL agent is configured to simulate the operation of components of a wireless network, and each DRL agent is configured to represent the state of information about the wireless network and the components. The DRL agent is configured to receive and execute training data to augment the state and estimate the reward; Interconnect DRL proxies are used to simulate real connections between components in a wireless network; and DRL agents are executed based on the state as input to simulate wireless networks online.

2. The device of claim 1, wherein each DRL agent is configured to emulate an individual component in a real wireless network, wherein the component includes individual components, and the wireless network includes a real wireless network implemented in a geographic area.

3. The device of claim 1, wherein the state includes an internal state representing technical internal information of the component, and wherein each DRL agent is configured to receive the internal state as input.

4. The device of claim 1, wherein the state includes an external state representing the state of wireless network users and the state of other DRL agents, and wherein each DRL agent is configured to receive the external state as input.

5. The device of claim 1, wherein each DRL agent is further configured to output a next internal state based on the state, the next internal state representing the network configuration of the DRL agent based on the state.

6. The device of claim 1, further comprising a user agent configured to emulate the operation of a user equipment in a wireless network, and the user agent configured to generate data services and user performance within the wireless network.

7. The device of claim 6, wherein the DRL agent is configured to receive data services and user performance within the wireless network.

8. The device of claim 6, wherein the user equipment includes a mobile device.

9. The device of claim 1, wherein, for expansion, the device is further configured to use an automatic encoder to expand the state.

10. The device of claim 9, wherein the automatic encoder comprises a variational automatic encoder (VAE).

11. The device of claim 1, wherein the device is further configured to use distributional regression for reward estimation.

12. The apparatus of claim 11, wherein the apparatus is configured for Gaussian process regression (GPR) for reward estimation.

13. The device of claim 1, wherein the device is configured in an expanded state such that a large number of states are obtained for the DRL agent; and The device is configured to reward estimates of a large number of states based on state similarity via distribution regression.

14. The device of claim 1, wherein the wireless network comprises a mobile network.

15. The device of claim 1, wherein the DRL agent is configured as a base station, switch, or data processor unit emulating a wireless network.

16. A method for simulating a wireless network, comprising: Configure deep reinforcement learning DRL agents, where each DRL agent is configured to simulate the operation of components of a wireless network, and each DRL agent is configured to represent the state of information about the wireless network and the components. The DRL agent receives and executes the training data, enabling the augmentation of the state and estimation of the reward; Interconnect DRL proxies are used to simulate real connections between components in a wireless network; and DRL agents are executed based on the state as input to simulate wireless networks online.

17. The method of claim 16, further comprising training the DRL agent offline prior to state expansion.

18. The method of claim 16, wherein each DRL agent is configured to emulate an individual component in a real wireless network, wherein the component includes an individual component, and the wireless network includes a real wireless network implemented in a specific geographic area.

19. The method of claim 16, wherein the method is configured for model-free simulation.

20. A computer program product comprising program code configured to perform the method of claim 16 when the computer program product is executed on a computer.

Images