Information processing method, electronic device, and computer program product

Abstract

Embodiments of this disclosure provide an information processing method, an electronic device, and a computer program product. In this information processing method, a simulator provides a first network state representation of a simulated network and a first content request event to an agent for reinforcement learning. The first content request event indicates that a requesting node in the simulated network requests target content stored in a source node. The simulator receives first action information from the agent, indicating a first caching action determined by the agent. The first caching action includes caching the target content in at least one caching node between the requesting node and the source node. Based on performing the first caching action in the simulated network, the simulator collects first training data for training the agent. In this manner, embodiments of this disclosure provide a simple simulator and an efficient training scheme for agents making content caching decisions.

Description

TECHNICAL FIELD

[0001] Embodiments of the present disclosure generally relate to information technology and computer technology, and more particularly to an information processing method, electronic device and computer program product. BACKGROUND

[0002] Information Centric Networking (ICN) attempts to change the focus of the current Internet architecture. Previous network architectures have focused on creating a conversation between two machines. For example, this is evident in a naming system where a Uniform Resource Locator (URL) is resolved through the Domain Name System (DNS) to indicate a machine to communicate with to receive data or perform an operation. In contrast, the goal of an ICN architecture is to shift the focus from connecting to another machine to getting data. In an ICN, data or content becomes independent of location, application, storage, and transmission method, thus enabling caching and replication of data or content within the network.

[0003] On the other hand, Reinforcement Learning (RL) is now becoming increasingly popular due to its flexibility and good performance. Before applying reinforcement learning to real scenarios, a corresponding training environment in which the reinforcement learning agent can interact with is a necessary part. However, in most cases, this environment in which the agent interacts is not available. Unfortunately, this is not an exception when applying reinforcement learning to a caching system for ICN. However, the proposed ICN alleviates the overwhelming workload brought by new technologies such as 5G to some extent. The caching mechanism is one of the core ideas of ICN, which can bring many benefits to the network. Therefore, it can be desirable to use reinforcement learning to make more optimized caching strategies in ICN networks or any other network in which content can be cached. SUMMARY

[0004] Embodiments of the present disclosure propose a simulator of a reinforcement learning environment and a technical solution of the interaction between the simulator and an agent, which can be used to simulate a network caching system and can interact with the agent of reinforcement learning to train the agent to make optimized content caching decisions in the network. More specifically, embodiments of the present disclosure provide an information processing method, electronic device and computer program product.

[0005] In a first aspect of the disclosure, an information processing method is provided. The method comprises providing, from a simulator to an agent for reinforcement learning, a first network state representation of a simulated network and a first content request event, the first content request event representing a request node in the simulated network requesting a target content stored in a source node. The method further comprises receiving, from the agent, first action information, the first action information indicating a first caching action determined by the agent, the first caching action comprising caching the target content in at least one cache node between the request node and the source node. The method further comprises collecting first training data for training the agent based on performing the first caching action in the simulated network.

[0006] In a second aspect of the disclosure, an electronic device is provided. The electronic device comprises at least one processor and at least one memory storing computer program instructions. The at least one memory and the computer program instructions are configured to, with the at least one processor, cause the electronic device to provide, from a simulator to an agent for reinforcement learning, a first network state representation of a simulated network and a first content request event, the first content request event representing a request node in the simulated network requesting a target content stored in a source node. The at least one memory and the computer program instructions are further configured to, with the at least one processor, cause the electronic device to receive, from the agent, first action information, the first action information indicating a first caching action determined by the agent, the first caching action comprising caching the target content in at least one cache node between the request node and the source node. The at least one memory and the computer program instructions are further configured to, with the at least one processor, cause the electronic device to collect first training data for training the agent based on performing the first caching action in the simulated network.

[0007] In a third aspect of the disclosure, a computer program product is provided. The computer program product is tangibly stored on a non-transitory computer readable medium and comprises machine executable instructions. The machine executable instructions, when executed, cause a machine to perform the steps of the method according to the first aspect.

[0008] It is to be understood that the description of the summary section is not intended to identify key or essential features of embodiments of the disclosure, nor is it intended to limit the scope of the disclosure. Other features of the disclosure will be apparent from review of the disclosure, which is described in its entirety in the following description. BRIEF DESCRIPTION OF DRAWINGS

[0009] The above and other objects, features and advantages of embodiments of the disclosure will become more apparent from the following detailed description read in conjunction with the accompanying drawings, in which:

[0010] Figure 1A schematic diagram showing an example information processing environment in which embodiments of the present disclosure can be implemented.

[0011] Figure 2 An interaction between an agent and an environment in a reinforcement learning setup according to an embodiment of the present disclosure is shown.

[0012] Figure 3 An example scenario in which embodiments of the present disclosure can simplify a plurality of computers (or a plurality of virtual machines) required by conventional methods to only a single computing device is shown.

[0013] Figure 4 A flow diagram of an example information processing method according to an embodiment of the present disclosure is shown.

[0014] Figure 5 A schematic diagram of a first loop between a simulator and an agent according to an embodiment of the present disclosure is shown.

[0015] Figure 6 A flow diagram of an example process of performing a second loop between a simulator and an agent according to an embodiment of the present disclosure is shown.

[0016] Figure 7 A schematic diagram of a second loop between a simulator and an agent according to an embodiment of the present disclosure is shown.

[0017] Figure 8 A schematic diagram of an Nth loop between a simulator and an agent according to an embodiment of the present disclosure is shown.

[0018] Figure 9 A workflow of a conventional simulator for evaluating cache performance in an information-centric network (ICN) is shown.

[0019] Figure 10 A workflow architecture of a simulator of an embodiment of the present disclosure implemented by improving a conventional simulator is shown.

[0020] Figure 11 A workflow of a simulator of an embodiment of the present disclosure implemented by improving a conventional simulator for training an agent is shown.

[0021] Figure 12 A flow diagram of an example process of testing performance parameters of an agent with a simulator according to an embodiment of the present disclosure is shown.

[0022] Figure 13 A Q-network architecture for training and testing an agent according to an embodiment of the present disclosure is shown.

[0023] Figure 14A and Figure 14BTwo experimental results showing performance comparison of embodiments according to the present disclosure with conventional methods are shown.

[0024] Figure 15 Improvements achieved by a simulator according to embodiments of the present disclosure over a conventional information-centric network (ICN) simulator are shown.

[0025] Figure 16 A block diagram of an example device that can be used to implement embodiments of the present disclosure is shown.

[0026] Throughout the drawings, identical or similar reference numbers are used to designate identical or similar components. DETAILED DESCRIPTION

[0027] The principles and spirit of the present disclosure will be described below with reference to several example embodiments shown in the accompanying drawings. It should be understood that the description of these embodiments is only to enable those skilled in the art to better understand and implement the present disclosure, and not to limit the scope of the present disclosure in any way. In the description and claims herein, unless otherwise defined, all technical and scientific terms used herein have the meanings commonly understood by those of ordinary skill in the art to which the present disclosure belongs.

[0028] As mentioned above, it can be desirable to utilize reinforcement learning to make more optimal caching policies in ICN networks or any other network where content can be cached. As a simple introduction, information-centric networking (ICN) is a method to move the Internet infrastructure away from the host-centric paradigm. With the development of technologies such as 5G, ICN is also mentioned as an important area in the strategic roadmap of various enterprises or other entities, and many related researches are being conducted today. In ICN, it is important to find a good way to manage the caching system. In this case, reinforcement learning (RL) can be a good choice. However, to train the RL algorithm, an environment can be needed. Currently, there is no simulator available to test and train the RL agent.

[0029] In view of the aforementioned problems and other potential issues in traditional solutions, embodiments of this disclosure propose a simulator for a reinforcement learning environment and a technical solution for interaction between the simulator and an agent. This simulator can be used to simulate a network caching system and can interact with a reinforcement learning agent to train the agent to make optimized content caching decisions in the network. In embodiments of this disclosure, the simulator provides the agent for reinforcement learning with a first network state representation of the simulated network and a first content request event, the first content request event indicating that a requesting node in the simulated network requests target content stored in a source node. Next, the simulator receives first action information from the agent, the first action information indicating a first caching action determined by the agent, the first caching action including caching the target content in at least one caching node between the requesting node and the source node. Then, based on the execution of the first caching action in the simulated network, the simulator collects first training data for training the agent. In this way, embodiments of this disclosure provide a simple simulator and an efficient training scheme for agents making content caching decisions. Some exemplary embodiments of this disclosure are described below with reference to the accompanying drawings.

[0030] Figure 1 A schematic diagram of an example information processing environment 100 in which embodiments of the present disclosure may be implemented is shown. (See diagram for reference.) Figure 1 As shown, the information processing environment 100 may include a computing device 110 for implementing the simulator 120 and the intelligent agent 130. It should be noted that both the simulator 120 and the intelligent agent 130 can be implemented using software, hardware, or a combination of both. It should also be noted that, although... Figure 1 The emulator 120 and agent 130 are shown as being implemented in the same computing device 110, but this is merely exemplary and is not intended to limit the scope of this disclosure in any way. In other embodiments, the emulator 120 and agent 130 may be implemented in different computing devices. Alternatively, one or both of the emulator 120 and agent 130 may be implemented distributed across multiple computing devices.

[0031] In some embodiments, the simulator 120 can simulate an environment that requires reinforcement learning to be performed by an agent 130, whereby the agent 130 is a model, algorithm, other form, or entity used to perform reinforcement learning on the environment simulated by the simulator 120. For example, in Figure 1 In the example, simulator 120 can provide a simulated network 140, hereinafter also referred to as simulated network 140. As an example, simulated network 140 may include multiple nodes, such as nodes 140-1 to 140-6, etc. In some embodiments, simulated network 140 may have a certain network topology. For example, in Figure 1In the example shown in FIG. 1, node 140-1 is connected to node 140-2, nodes 140-2 and 140-3 are connected to node 140-5, and node 140-5 is connected to nodes 140-4 and 140-6. It is noted that, Figure 1 The particular number of nodes and the particular network topology shown in FIG. 1 are merely illustrative and are not intended to limit the scope of the present disclosure in any way. In other embodiments, the simulation network 140 can have any number of nodes and any network topology. Additionally, in some embodiments, the number of nodes and the topology of the simulation network 140 can be dynamic, i.e., can change over time.

[0032] In the simulation network 140, some or all of the nodes (e.g., nodes 140-1 through 140-6, etc.) can store content. As used herein, "content" can generally refer to any data or information that can be transmitted and cached among different nodes. It will be appreciated that the content stored in different nodes can be the same or different, and the number of content stored in different nodes can be the same or different. Thus, in some cases, a certain node can need to obtain a target content that it does not have. In such a case, the node can request a source node that stores the target content in the simulation network 140 to transmit the target content to the requesting node. In the process of transmitting the target content from the source node to the requesting node, one or more intermediate nodes can be passed through. In some cases, one or more of these intermediate nodes can cache the target content, such that when the target content is requested again by other nodes, these caches can serve as source nodes to provide the target content. For ease of description, a node that caches a target content can be referred to as a cache node.

[0033] It is noted that in the context of the present disclosure, a "requesting node" can generally refer to a node that issues a content request, a "source node" can generally refer to a node whose stored content is requested, and a "caching node" can generally refer to a node that caches content for transmission. In some embodiments, the "requesting node", "source node", and "caching node" can be the same node in terms of function or structure, and the difference can only be that the source node stores content that the requesting node and the caching node do not have, and the difference between the requesting node and the caching node can only be whether a content request is issued. In other embodiments, the "requesting node", "source node", and "caching node" can also be different nodes in terms of structure and function. For example, the "source node" can be a node that is more powerful and more complex in structure, such as a server in a computer network, a base station in a communication network, and the like. In contrast, the "requesting node" or "caching node" can be a node that is relatively weak in function and relatively simple in structure, such as a general computer in a computer network, a mobile terminal in a communication network, and the like. In some cases, the "caching node" can also be a node that is intermediate in function and structure between the "source node" and the "requesting node". In other embodiments, the "caching node" can be the same as the "source node" or "requesting node" in terms of function and structure.

[0034] Generally, the simulated network 140 provided by the simulator 120 can be any network in which a requesting node can request content from a source node, and a caching node can cache the content during transmission of the content. In some embodiments, the simulated network 140 can be an information-centric network (ICN). In an ICN, data or content becomes location, application, storage, and transport agnostic, enabling in-network caching and replication. In this way, a caching node can cache a subset of content or data and can act as a tier that can provide fast data access to requesting nodes (e.g., clients) while reducing traffic pressure on source servers. Caching nodes can be located locally on a device (e.g., a smartphone memory), at the edge of a network (e.g., a content delivery network, CDN), hosted near a database server (e.g., Redis), or a combination of the foregoing.

[0035] To some extent, ICN solves some problems by shifting the focus, but raises some new ones. For example, when delivering content, there is a question of which node to cache this content on (cache allocation). After the node to cache the content is selected, a second question arises. Namely, the storage size of these cache nodes can be limited, especially when the cache nodes are deployed on client devices. When the cached content exceeds its maximum capacity, a cache eviction algorithm is needed to decide which content to evict (cache replacement). Fortunately, reinforcement learning can help in this situation. Overall, in some embodiments of the present disclosure, the simulator 120 and the agent 130 can be designed to focus mainly on how to simulate the cache allocation process and the cache replacement process.

[0036] As mentioned above and in Figure 1 Further shown, the simulator 120 can interact with the agent 130 for reinforcement learning in order to train or test the agent 130. Due to the universality of reinforcement learning, reinforcement learning has also been studied in many other disciplines such as game theory, control theory, operations research, information theory, simulation library optimization, etc. Formally, reinforcement learning is a sequential decision-making process in which the agent 130 for reinforcement learning can learn an optimal policy through interaction with the environment (e.g., the simulated network 140 provided by the simulator 120). The interaction process between the agent and the environment in reinforcement learning is briefly introduced below with reference to Figure 2

[0037] Figure 2 The interaction between the agent 210 and the environment 220 in a reinforcement learning setting according to embodiments of the present disclosure is shown. In some embodiments, Figure 1 The agent 130 depicted in Figure 2 is an example of the agent 210, and Figure 1 The simulated network 140 provided by the simulator 120 in Figure 2 is an example of the environment 220. As Figure 2 shown, during the interaction process between the agent 210 and the environment 220, the environment 220 can provide the agent 210 with a state related to the environment 220. Then, the agent 210 can determine an action to be performed in the environment 220 based on the state provided by the environment 220. Next, the environment 220 can determine a reward (positive reward or negative reward, i.e., punishment) to be given to the action by performing the action indicated by the agent 210 to indicate to the agent 210 whether the action is good (encouraged) or bad (needs to be avoided). Generally, the goal of the agent 210 is to maximize the expected sum of future rewards, which can be achieved by training in multiple steps (state-action transitions) and choosing the action that returns the maximum discounted expected reward function. ​

[0038] Generally, reinforcement learning algorithms can be divided into two categories, namely, model-based and model-free. In model-free algorithms, the agent 210 has no systematic prior knowledge, but rather learns the impact of actions on the environment 220 by performing actions in the real environment. Thus, model-free algorithms do not require input from the developer, are very flexible, and are easy to set up. In contrast, model-based algorithms learn a model of the environment, and then the agent 210 continues learning from interactions with the resulting environment model. The environment model provides the agent 210 with more simulated steps, thus allowing the agent 210 to converge to an optimal policy more quickly. However, model-based approaches can be very challenging to design, as the environment model must accurately reflect the actual environment. Thus, in some embodiments, to facilitate the use of the system provided by embodiments of the present disclosure in practice, the simulator 120 presented by the present disclosure can be implemented for model-free reinforcement learning algorithms, and can use a weighted reward system to reflect requirements in certain scenarios. Of course, in other embodiments, the simulator 120 presented by the present disclosure can also be implemented for model-based reinforcement learning algorithms.

[0039] As described above, the basis of reinforcement learning training is an interactive environment. The simulator 120 for reinforcement learning generally needs to have the following features in order to interact with the agent 130 of reinforcement learning. The first feature that can be needed is the state representation of the environment. Generally, the state representation of the environment can be an object or an array of data that matches the sensor readings in the actual environment. For reinforcement learning, it can be important that the state has Markov properties, so that the prediction of the value can be accurate. For certain environments, this means that the resulting value is to be calculated from the observation, or the combined history of the last few observations from the sensor is represented as the state. For example, the state can be saved in the internal representation of the environment, which is a typical object-oriented approach. For another example, the state can also be passed as a parameter to other functions. In some embodiments, a simple state representation can be a fixed-size array of numbers, which can represent important features of the environment. For convenience, when used with neural networks, these numbers can be scaled between -1 and 1.

[0040] Additionally, another feature that the reinforcement learning simulator 120 can need can be an action representation. For example, a simple action representation can be an integer that can identify which of N actions starting from 0 is selected. This allows for a basic index lookup when checking value function estimates. Furthermore, yet another feature that the reinforcement learning simulator 120 can need can be a reward function. This can be part of the problem definition. Sometimes it can be desirable to have the reward function (e.g., code) as part of the environment or as part of the agent, or somewhere in between, depending on how likely it is to be changed. For example, if multiple experiments need to be run to optimize different aspects of the control, but need to be run in the same environment, then a completely independent reward computation module can be made and combined at a high level with the agent 130 (e.g., agent code) and the environment 140 (e.g., environment code).

[0041] In general, the design of the reward function can be a complex topic. However, a basic principle of the design can be that the reward value can always be a single real number (i.e., a floating point number in most programming languages). On the other hand, because one of the purposes of the methods presented in this disclosure is to build an integrated system, more factors will be taken into account in such a system. One of these factors can be the need for a flexible reward algorithm to reflect different needs of multiple scenarios. For example, sometimes there can not be much space in the cache node, so the most important requirement can be to consider the storage space, and then the requirement for efficiency. In this case, more weight can be given to the reward related to the storage space, and the weight of the reward related to the efficiency can be reduced.

[0042] Furthermore, yet another feature that the reinforcement learning simulator 120 can need can be a time step function. The time step function can take an action selection and can update the state for a time step, returning the next state of the environment and the immediate reward for the action. If the environment is real, the environment (e.g., environment code) can make actual changes (e.g., move a robot arm, etc.), can need to wait for the time step to pass, and then read sensors to get the next state of the environment and compute the reward for the action. If the environment is simulated, the environment (e.g., environment code) can call an internal model to compute the next state of the environment. In some embodiments, the time step function can call a proprietary software provided for the task.

[0043] It is additionally noted that if the actions available in the environment depend on the current state of the environment, then the functionality (e.g., code) for this case can exist in the environment simulation (e.g., simulation network 140) or in the agent 130, or can be some kind of helper function that the agent 130 can call, so it can filter the actions before one is selected. In summary, Figure 1The simulator 120 in the environment can need to be able to return the state of the environment (e.g., the simulated network 140) and a reward value for an action of the agent 130, and accept an action selection given by the agent 130.

[0044] Accordingly, in some embodiments, the simulator 120 can provide the agent 130 with a network state representation 122 of the simulated network 140, and a content request event 124. The network state representation 122 can represent the state of the simulated network 140, while the content request event 124 can represent an event in which a requesting node requests a certain target content from a source node. For example, the request event can be randomly generated. It is noted that although the network state representation 122 and the content request event 124 are depicted as being provided together to the agent 130 in the example scenario 300, this is merely exemplary and is not intended to limit the scope of the present disclosure in any way. In other embodiments, the network state representation 122 and the content request event 124 can also be provided separately to the agent 130. Figure 1

[0045] After receiving the network state representation 122 and the content request event 124, the agent 130 can determine a caching policy or decision to be executed in the simulated network 140. For example, which intermediate node(s) between the source node and the requesting node are to cache the target content, and / or which storage area(s) in the caching node(s) are to store the target content. Based on the caching decision, the agent 130 can provide action information 126 to the simulator 120 to indicate the caching action to be executed in the simulated network 140. Based on the caching action indicated by the action information 126, the simulator 120 can execute the caching action in the simulated network 140, and make relevant performance and parameter measurements in the current content delivery process, thereby collecting training data 150 for training the agent 130. As mentioned above, the agent 130 can be implemented by software, hardware, or a combination of both, where the implementation by software can have additional technical advantages, which are explained below with reference to Figure 3

[0046] Figure 3 The example scenario 300 illustrates that the multiple computers (or multiple virtual machines) 310 required by the conventional method can be simplified to only a single computing device 320 according to embodiments of the present disclosure. That is, in the example scenario 300, Figure 3 In the example scenario 300, Figure 1 The simulator 120 depicted in the example scenario 300 can be implemented in the single computing device 320. Accordingly, Figure 3 The single computing device 320 in the example scenario 300 can be considered as a Figure 1 ​​The example of the computing device 110 shown in FIG. 1. For example, in such embodiments, the emulator 120 can be implemented based on code, and thus can only require one computer with a code editor. In contrast, the conventional approach requires multiple computers (or multiple virtual machines) 310 for implementing the same functionality.

[0047] In some embodiments, the computing device 110 can include any device capable of implementing computational and / or control functions, including but not limited to, a special purpose computer, a general purpose computer, a general purpose processor, a microprocessor, a microcontroller, or a state machine. The computing device 110 can be implemented as a single computing device or a combination of computing devices, e.g., a combination of a digital signal processor (DSP) and a microprocessor, a plurality of microprocessors, one or more microprocessors in conjunction with a DSP core, or any other such configuration. It is also noted that, in the context of the present disclosure, the computing device 110 can also be referred to as an electronic device 110, and the two terms can be used interchangeably herein.

[0048] In some embodiments, the communication links between the various components in the systems involved in the present disclosure can be any form of connection or coupling between the components that enables data communication or control signal communication between the components, including but not limited to, a coaxial cable, a fiber optic cable, a twisted pair, or wireless technology such as infrared, radio, and microwave. In some embodiments, the communication links can also include, but are not limited to, network cards, hubs, modems, repeaters, bridges, switches, routers, and other devices for network connection, as well as various network connection lines, wireless links, and the like. In some embodiments, the communication links can include various types of buses. In other embodiments, the communication links can include computer networks, communication networks, or other wired or wireless networks.

[0049] It should be understood that, Figure 1 The units, elements, modules, or components related to the embodiments of the present disclosure in the information processing environment 100 are shown only schematically. In practice, the information processing environment 100 can also include other units, elements, modules, or components for other functions. In addition, Figure 1 The specific number of units, elements, modules, or components shown in FIG. 1 is only illustrative and is not intended to limit the scope of the present disclosure in any way. In other embodiments, the information processing environment 100 can include any appropriate number of computing devices and related elements. Therefore, the embodiments of the present disclosure are not limited to Figure 1 the specific scenario depicted, but are generally applicable to any technical environment for information processing.

[0050] Figure 4A flowchart of an example information processing method 400 according to an embodiment of the present disclosure is shown. In some embodiments, method 400 may be implemented by an emulator 120 in a computing device 110 in environment 100, for example by a processor or processing unit of computing device 110 for implementing emulator 120, or by various functional modules of computing device 110 for implementing emulator 120. In other embodiments, method 400 may also be implemented by an emulator in a computing device independent of environment 100, or by other units or modules in environment 100 for implementing emulator. For ease of explanation, it will be combined with Figure 5 To describe method 400.

[0051] Figure 5 A schematic diagram of a first loop 500 between the simulator 120 and the agent 130 according to an embodiment of the present disclosure is shown. It should be noted that... Figure 5 The scene shown can be considered as Figure 1 This is an example of the interaction process between the simulator 120 and the agent 130 depicted. For example, the interaction process between the simulator 120 and the agent 130 may include multiple loops, each loop referring to the simulator 130 issuing a state representation 122 and a content request event 124 of the simulation network 140 to the agent 130, then the agent 130 issuing an action representation 126 to the simulator 120, then the simulator 130 executing a cache action specified by the agent 130 in the simulation network 140, and collecting training data 150.

[0052] In some embodiments, Figure 5 The first loop 500 shown can be any one of multiple loops in the interaction process between the simulator 120 and the agent 130. In other words, the first loop 500 is not necessarily the first loop between the simulator 120 and the agent 130; that is, there may be another loop before the first loop 500. Of course, in other embodiments, the first loop 500 can also be the first loop between the simulator 120 and the agent 130. That is, there may not be another loop before the first loop 500. In this scenario, the simulator 120 and the agent 130 may be initialized and ready to begin the interaction process for training the agent 130.

[0053] refer to Figure 4 and Figure 5 ,exist Figure 4 At box 410, the simulator 120 provides the agent 130 for reinforcement learning with a first network state representation 122-1 of the simulated network 140 and a first content request event 124-1. It should be noted that, although... Figure 5The first network state representation 122-1 and the first content request event 124-1 are depicted as being sent from the simulator 120 to the agent 130 at the same time, but this is merely illustrative and is not intended to limit the scope of the present disclosure in any way. In other embodiments, the simulator 120 can also provide the first network state representation 122-1 and the first content request event 124-1 to the agent 130 separately.

[0054] In some embodiments, the first network state representation 122-1 can include a network topology of the simulated network 140 and node characteristics of the individual nodes in the simulated network 140. In this way, the simulator 120 can provide the agent 130 with necessary information about the simulated network 140 and its nodes so that the agent 130 can be trained to have the ability to make more optimized caching decisions based on these rich information. For example, the network topology in the first network state representation 122-1 will affect the transmission path of the content between the nodes, and thus will affect the caching decisions of the agent 130. In some embodiments, the topology of the simulated network 140 (e.g., the original topology) can be constructed based on the Fast Network Simulation Setup (FNSS) toolchain. In this way, the FNSS toolchain can return an adjacency matrix and a node matrix related to the individual nodes in the simulated network 140, which can almost perfectly describe the topology of the network graph.

[0055] Further, in some embodiments, the node characteristics in the first network state representation 122-1 can include node types so that the agent 130 can be trained to make more optimized caching decisions according to the types of the individual nodes. For example, if the simulated network 140 is an information-centric network, there are typically three types of nodes in the simulated network 140, i.e., source nodes, cache nodes, and request nodes (also referred to as target nodes in ICN). In the node definition of ICN, source nodes can be the nodes that store content, target nodes can be the nodes that request content, and cache nodes are special target nodes that have the ability to cache content. Additionally or alternatively, the node characteristics in the first network state representation 122-1 can include the cache status of the nodes. For example, whether the nodes cache content or not, or the identifiers of the cached content in the nodes, etc. Based on the cache status information, the agent 130 can be trained to make more optimized caching decisions according to the cache status of the individual nodes.

[0056] Additionally or alternatively, the node characteristics in the first network state representation 122-1 can include content characteristics of the content in the nodes, such that the agent 130 can be trained to make more optimal caching decisions based on the characteristics of the individual content in the individual nodes. In some embodiments, the content characteristics can include a request frequency of the content, which can indicate whether the content is frequently requested. For example, in selecting the content to be replaced, the agent 130 can prefer to replace the content with a lower request frequency. In other embodiments, the content characteristics can include a time of being cached of the content, which can indicate how recent the cached content is. For example, in selecting the content to be replaced, the agent 130 can prefer to replace the content that has been cached for a longer time.

[0057] In other embodiments, the content characteristics can include a recent popularity of the content (which can also be referred to as a popularity herein), which can be determined based on the request frequency of the content and the time of last request of the content. Since the recent popularity takes into account both the frequency of the content being requested and whether the content has been requested recently, it can better reflect the current popularity of the content. For example, in selecting the content to be replaced, if there are two contents available to be replaced, and the access frequencies of the two contents are the same or not significantly different, the agent 130 can prefer to replace the content that has been requested for a longer time. With the above content characteristics, the agent 130 can better determine whether a content can be replaced by a new content that needs to be cached, and thus be better trained to make more optimal caching decisions.

[0058] With continued reference to Figure 5 , the first content request event 124-1 can represent an event in which a requesting node in the simulated network 140 requests a target content stored in a source node. As mentioned above, in some embodiments, the first content request event 124-1 can be randomly generated. This is reasonable because in a real network, the request events that occur in the network can also be considered as a random process. Thus, based on the first content request event 124-1, the agent 130 can be trained to make more optimal caching decisions based on the characteristics of the nodes and content involved in the request event.

[0059] With continued reference to Figure 4 and Figure 5 , in Figure 4At block 420, the simulator 120 receives the first action information 126-1 from the agent 130. The first action information 126-1 can indicate the first caching action determined by the agent 130. For example, which intermediate node(s) the target content is to be cached in. For another example, which storage area(s) of the node the target content is to be cached in. As an example, after receiving the first network state representation 122-1 and the first content request event 124-1 from the simulator 120, the agent 130 can learn the topology of the simulated network 140, the request node, the source node, and the requested target content, and thus can determine the transmission path of the target content from the source node to the request node. In some embodiments, if there are multiple transmission paths that can be selected, the agent 130 can select a preferred transmission path from the multiple transmission paths to optimize the transmission performance of the target content.

[0060] After determining the transmission path of the target content, the agent 130 can further determine the intermediate node(s) that the target content will pass through between the request node and the source node. Then, based on various information in the first network state representation 122-1, such as network topology information and node characteristic information, the agent 130 can use the reinforcement learning model to make a caching decision for the target content. Generally, the caching decision made by the agent 130 can be any decision that is related to the caching of the target content. Thus, in the context of the present disclosure, the “caching decision” can also be referred to as the “caching action.” As mentioned above, in some embodiments, such a caching action can include which intermediate node(s) need to perform a caching operation on the target content. Thus, the first caching action can include caching the target content in one or more caching nodes between the request node and the source node. After determining the caching action associated with the target content, the agent 130 can provide the first action information 126-1 to the simulator 120, which represents the first caching action determined by the agent 130.

[0061] In some embodiments, the first caching action indicated by the first action information 126-1 can be selected from a first action set. The first action set can be represented using a first set of bits, the number of bits of the first set of bits being equal to the number of nodes of the simulated network 140, and each bit can indicate whether the corresponding node is to cache the target content. Each node corresponds to a bit, and the value of the bit indicates whether the node is to cache the target content. In this way, which node the target content is to be cached in can be efficiently expressed in the form of a bitmap, thereby improving the efficiency of action representation. In some embodiments, if the agent 130 is designed for the entire simulated network 140, the number of actions in the first action set can depend on the number of nodes in the simulated network 140. For example, in the case where the number of nodes is n, the number of actions in the first action set can be 2n. n In this way, information interaction between multiple agents in the case where multiple agents are set up can be excluded, thereby saving relevant resources. In some embodiments, if the agent 130 is designed for one node in the simulated network 140, the first action set can include two actions, i.e., cache and do not cache. In this way, the design of the agent 130 can be simplified.

[0062] In other embodiments, the caching action indicated by the agent 130 to the simulator 120 can further include which storage area in the caching node the caching node needs to cache the target content. That is, in the case where the storage space of the caching node is full, which existing content in the caching node the target content should replace. In this way, the agent 130 can indicate the specific storage area where the target content is to be stored or the content to be replaced through a simple action representation. Therefore, in some embodiments, the first caching action indicated by the first action information 126-1 can be selected from a second action set. The actions in the second action set can indicate caching the target content in a target storage area in the caching node. In some embodiments, if the agent 130 is designed for the entire simulated network 140, the number of actions in the second action set can depend on the number of nodes in the simulated network 140 and the number of storage areas in the nodes. In this way, information interaction between multiple agents in the case where multiple agents are set up can be excluded, thereby saving relevant resources. For example, in the case where the node has k storage areas and the simulated network 140 includes n nodes, the number of actions can be n k In some embodiments, if the agent 130 corresponds to one node in the simulated network 140, the number of actions in the second action set can depend on the number of storage areas of the node. In this way, the design of the agent 130 can be simplified. For example, in the case where the caching node has k storage areas, the number of actions can be k.

[0063] With continued reference toFigure 4 and Figure 5 ,exist Figure 4 At box 430, the simulator 120 performs a first caching action determined by the agent 130 within the simulated network 140 and collects first training data 150-1 for training the agent 130. Generally, the first training data 150-1 may include any data that enables the training of the agent 130. In some embodiments, the first training data 150-1 may include data more conducive to training the agent 130, such as a first network state representation 122-1, first action information 126-1, and / or a reward value for the first caching action, etc. In this way, the simulator 120 can collect more relevant data that is more beneficial to training the agent 130, thereby enabling better training of the agent 130. In some embodiments, the reward value may be determined based on a reward function of the simulator 120, which may be based on the hit rate of the cached content, the average latency of the requested content, and / or the bandwidth used to transmit the requested content, etc. In this way, agent 130 can be trained to make caching decisions that optimize these parameters, thereby improving content caching and transmission performance in the simulated network 140. In some embodiments, the hit rate, average latency, and bandwidth can each have adjustable weights in the reward function. Therefore, the importance of each parameter during training, and also the importance of these parameters in providing caching decisions after training, can be flexibly adjusted.

[0064] In some embodiments, the simulator 120 can execute a first buffered action indicated by the agent 130 in the simulated network 140 based on a time step determined by the simulator 120's time step function. As noted above, the time step function can take action selection and can update the state for each time step, determining the next state of the simulated network 140 and the immediate reward for the first buffered action. In this way, the simulator 120 can better control or manage the interaction process with the agent 130 in time.

[0065] In some embodiments, if the simulator 120 collects the first training data 150-1, the simulator 120 can store the first training data 150-1 instead of providing the first training data 150-1 to the agent 130 immediately. In this way, the simulator 120 can accumulate the training data 150 in batches, thereby reducing the computation amount of the simulator 120 and the agent 130, and saving the relevant resource occupation. Moreover, after the first training data 150-1 is stored, the simulator 120 can also perform pre-processing on the first training data 150-1, such as processing useless data (such as null values or outliers, etc.) in the first training data 150-1, normalizing or data scaling the first training data 150-1, encoding features, discretizing, etc. In this way, the training effect of the agent 130 based on the first training data 150-1 can be optimized. Of course, in other embodiments, if the simulator 120 collects the first training data 150-1, the simulator 120 can also provide the first training data 150-1 to the agent 130 immediately. In this way, the storage resource for storing the first training data 150-1 can be saved.

[0066] In general cases, in order to train the agent 130 sufficiently, it can be insufficient that only one cycle 500 is performed between the simulator 120 and the agent 130. Therefore, in some embodiments, after the first cycle 500 is ended, the simulator 120 and the agent 130 can continue to perform a second cycle. In other words, the simulator 130 can send another state representation 122 of the simulation network 140 and another content request event 124 to the agent 130, then the agent 130 can send another action representation 126 to the simulator 120, and then the simulator 130 can perform another cache action in the simulation network 140 as specified by the agent 130, and can collect another training data 150. In this way, the simulator 120 can train the agent 130 more sufficiently. Such embodiments are described below with reference to Figure 6 and Figure 7 .

[0067] Figure 6A flowchart illustrating an example process 600 of executing a second loop between a simulator 120 and an agent 130 according to an embodiment of the present disclosure is shown. In some embodiments, process 600 may be implemented by a simulator 120 in a computing device 110 in environment 100, for example by a processor or processing unit of computing device 110 for implementing simulator 120, or by various functional modules of computing device 110 for implementing simulator 120. In other embodiments, process 600 may also be implemented by a simulator in a computing device independent of environment 100, or by other units or modules in environment 100 for implementing simulator. For ease of explanation, it will be described in conjunction with... Figure 7 To describe process 600.

[0068] Figure 7 A schematic diagram of a second loop 700 between the simulator 120 and the agent 130 according to an embodiment of the present disclosure is shown. It should be noted that... Figure 7 The scene shown can be considered as Figure 1 This is an example portion of the interaction process between the simulator 120 and the agent 130 depicted. For example, the interaction process between the simulator 120 and the agent 130 may include multiple loops, and the second loop 700 may be... Figure 5 The first cycle, 500, is the cycle that occurs after the first cycle described in the text.

[0069] refer to Figure 6 and Figure 7 ,exist Figure 6 At box 610, the simulator 120 can update the first network state representation 122-1 based on the first cached action provided by the agent 130 in the first loop 500 to determine the second network state representation 122-2 of the simulated network 140. In other words, the simulator 120 performs [actions] in the simulated network 140. Figure 5 The first caching action is described in the diagram, performed by the first action information 126-1 provided by agent 130. For example, according to the first action information 126-1 provided by agent 130, simulator 120 caches the target content into one or more intermediate nodes. Or, for example, when targeting a specific node, simulator 120 caches the target content into the specific storage area specified in the first action information 126-1, and so on. After completing the caching action, simulator 120 can determine the second network state representation 122-2.

[0070] At block 620, the simulator 120 can provide the second network state representation 122-2 and the second content request event 124-2 to the agent 130. For example, the second network state representation 122-2 and the second content request event 124-2 can be similar to the first network state representation 122-1 and the first content request event 124-1, respectively. Therefore, the specific content and related procedures of the second network state representation 122-2 and the second content request event 124-2 are not repeated here. Of course, it needs to be explained that since the content request event 124 can be randomly generated, the second content request event 124-2 is generally different from the first content request event 124-1, but the two can also be the same.

[0071] At block 630, the simulator 120 can receive the second action information 126-2 from the agent 130, the second action information 126-2 indicating the second caching action determined by the agent 130. For example, the second action information 126-2 and the second caching action can be similar to the first action information 126-1 and the first caching action, respectively. Therefore, the specific content and related procedures of the second action information 126-2 and the second caching action are not repeated here.

[0072] At block 640, the simulator 120 can collect the second training data 150-2 for training the agent 130 based on performing the second caching action in the simulated network 140. For example, the process of the simulator 120 performing the second caching action can be similar to the process of the simulator 120 performing the first caching action, and the process of the simulator 120 collecting the second training data 150-2 can be similar to the process of the simulator 120 collecting the first training data 150-1. In addition, the content of the second training data 150-2 can be similar to the content of the first training data 150-1. Furthermore, similar to the first training data 150-1, the simulator 120 can store and preprocess the second training data 150-2. In addition, the simulator 120 can also perform the second caching action based on the time step determined by the time step function similarly. Therefore, the process of the simulator 120 performing the second caching action to collect the second training data 150-2 is not repeated here.

[0073] Figure 8 A schematic diagram of the Nth cycle 800 between the simulator 120 and the agent 130 according to an embodiment of the present disclosure is shown. It needs to be explained that, Figure 8 The scenario shown in FIG. 8 can be considered as Figure 1 One example part of the interaction process between the simulator 120 and the agent 130 depicted in FIG. 8. For example, the interaction process between the simulator 120 and the agent 130 can include multiple cycles, and the Nth cycle 800 can be considered as the last cycle in one training epoch or one training phase.

[0074] With reference to Figure 8 As described above with respect to Figure 6 and Figure 7 Similarly, the simulator 120 can update the (N-1)th network state representation based on the Nth cache action to determine the Nth network state representation 122-N of the simulated network 140. Then, the simulator 120 can provide the Nth network state representation 122-N and the Nth content request event 124-N to the agent 130. Next, the simulator 120 can receive the Nth action information 126-N from the agent 130, the Nth action information 126-N indicating the Nth cache action determined by the agent 130. Thereafter, the simulator 120 can collect the Nth training data 150-N for training the agent 130 based on performing the Nth cache action in the simulated network 140.

[0075] It is noted that the Nth network state representation 122-N, the Nth content request event 124-N, the Nth action information 126-N, the Nth training data 150-N, etc. involved in the Nth loop 800 can be similar to the first network state representation 122-1, the first content request event 124-1, the first action information 126-1, the first training data 150-1, etc. involved in the first loop 500, respectively, or can be similar to the second network state representation 122-2, the second content request event 124-2, the second action information 126-2, the second training data 150-2, etc. involved in the second loop 700, respectively. Moreover, the content of the Nth training data 150-N can be similar to the content of the first training data 150-1 and the content of the second training data 150-2. Furthermore, the simulator 120 can store and preprocess the Nth training data 150-N similarly as for the first training data 150-1 or the second training data 150-2. In addition, the simulator 120 can also perform the Nth cache action based on the time step determined by the time step function similarly. Therefore, the content of the Nth network state representation 122-N, the Nth content request event 124-N, the Nth action information 126-N, the Nth training data 150-N, etc. or related processes will not be repeated here.

[0076] In some embodiments, if the number of collected training data 150 reaches the predetermined number N, the simulator 120 can provide the collected predetermined number N of training data 150 (e.g., including training data 150-1 to 150-N) together to the agent 130. In this way, the number of loops in one training phase or training epoch can be flexibly set manually by a user. In such embodiments, the simulator 120 can perform the pre-processing of the training data 150-1 to 150-N collectively, instead of performing the pre-processing of the training data 150-1 to 150-N separately in time, thereby simplifying the pre-processing operation of the training data 150 by the simulator 120.

[0077] As mentioned above, in some embodiments, Figure 5 The first loop 500 shown in FIG. 5 can be the first loop between the simulator 120 and the agent 130. That is, there is no other loop before the first loop 500. For example, the simulator 120 and the agent 130 can complete the initialization and prepare to start the interaction process for training the agent 130. In such embodiments, the initialization of the simulator 120 can include determining the first network state representation 122-1 of the simulation network 140, the first content request event 124-1, and the predetermined number N described above. In other words, when initializing the simulator 120, the simulator 120 can determine how many loops after which the training data 150 is provided to the agent 130 collectively. By initializing the simulator 120, various simulation parameters of the simulator 120 can be configured according to the needs of training the agent 130, thereby optimizing the training of the agent 130.

[0078] The above description is made with reference to Figure 1 to Figure 8 Some embodiments of the simulator 120 of the present disclosure and its interaction process with the agent 130 are described above. In addition, the technical advantages and benefits that the simulator 120 of the present disclosure can have over the conventional solutions are briefly pointed out above for various different embodiments. In addition to the advantages pointed out above, the simulator 120 of the present disclosure can have the novelty or improvement that the simulator 120 can be a brand new code-based network cache RL simulator. For example, it can be completely based on the Python language. Traditionally, people always deploy many virtual machines or even many real computers to simulate a network (e.g., ICN), and the embodiments of the present disclosure provide a simple alternative method, which can only require one computer and a code editor such as VS code for people who want to do some experiments on a network (e.g., ICN), which is both convenient and time-saving.

[0079] A second improvement can be that the simulator 120 is a simulator that trains the RL, which can interact with the RL agent. Specifically, the reward and state information needed in the RL training are returned as inputs to the neural network in the RL agent. In addition, the simulator 120 is designed to accept action information from the RL agent. Thus, the simulator 120 can be used for any RL algorithm cache policy. A third improvement can be in the content representation of the content in the network. In most traditional simulators (e.g., the Icarus simulator), the content is only represented by numbers. That is, in the cache system, the properties of the content (which can not be unimportant in the cache system) are ignored. Embodiments of the present disclosure propose a new content representation to encode the content properties. Embodiments of the present disclosure also propose a corresponding RL training process, including a new result collection method. In addition, specific RL algorithms have been tested on the simulator 120, and the test results are very good.

[0080] It should be noted that although the simulator 120 of embodiments of the present disclosure can be completely software-based. However, in other embodiments, the simulator 120 of embodiments of the present disclosure can also be implemented based on hardware, or a combination of software and hardware. In addition, in some embodiments, in order to save resources for developing the simulator 120 of embodiments of the present disclosure, the simulator 120 can also be implemented based on a modification of a traditional simulator. For example, a traditional simulator that can be changed can be a simulator called Icarus. In other words, as an example embodiment, the simulator 120 of embodiments of the present disclosure can be implemented by improving the Icarus simulator. Such an implementation is described below with reference to Figure 9 to Figure 11

[0081] Figure 9 A workflow 900 of a conventional simulator for evaluating cache performance in an information-centric network (ICN) is shown. For example, Figure 9 The relevant workflow 900 of a conventional Icarus simulator is depicted in FIG. 9. As an ICN simulator, Icarus is a Python-based discrete event simulator for evaluating cache performance in ICN. Icarus can be unconstrained by any specific ICN architecture, and its design can enable users to implement and evaluate new cache strategies or cache and routing strategies with very few lines of code.

[0082] As Figure 9 ​As shown, the Icarus simulator’s workflow 900 can include a scenario generation 910, an experiment orchestration 920, an experiment execution 930, and a result collection 940. The scenario generation 910 phase includes all the steps needed to set up a fully configured network topology and random event generators for the simulation. For example, the scenario generation 910 can be based on the FNSS toolchain. In the experiment orchestration 920 phase, the Icarus simulator can read the parameter ranges (e.g., cache size, cache policy, content popularity distribution, etc.) that the user wants to simulate from a configuration file and use all the required parameter combinations to conduct experiments. The Icarus simulator can then orchestrate to execute the experiments in parallel in a set of flows.

[0083] The experiment execution 930 phase includes the actual execution of the experiments. An instance of the Icarus simulation engine is provided with the scenario description (i.e., network topology and event generators). The engine reads events from the generators and dispatches them to the relevant handlers. One or more data collectors measure various user-specified metrics and return the results to the engine at the end of the experiment, which then passes them to the result aggregator. With respect to the result collection 410 and analysis phase, the object collects and aggregates the results after each experiment terminates and allows the user to compute confidence intervals, plot the results, or serialize the data in various formats for later processing. However, the Icarus simulator cannot be used for reinforcement learning scenarios, and if the Icarus simulator is to be used in a reinforcement learning setting, many improvements need to be made for RL training, which will be discussed in detail below.

[0084] Traditionally, people like to use real computers or virtual machines to train and evaluate RL agents, but this is neither convenient nor efficient. Therefore, a fully code-based simulator is needed to release the effort of building a simulation environment. The Icarus simulator can meet some requirements of the RL simulator, but the Icarus simulator is still far from being able to be applied in RL training, and many improvements and work need to be made. The first improvement needed is to use code simulation of the cache process. Although a baseline of a code-based simulator is found by luck, which can solve this problem to some extent, but this is still an initial problem that people encounter at the beginning.

[0085] Second, the required improvement is the emulator adaptation. As mentioned before, the current Icarus emulator needs to be modified a lot to become an RL emulator. For example, the content is currently represented by pure numbers in the Icarus emulator, but the properties of the content are also important in the cache policy training and testing. Then the required improvement is the RL emulator design. How to design a scalable and flexible RL emulator is another problem, which should be designed and easily modified reward and state system to use in different RL settings, also need to accept different forms of action and execute the action according to the input. Further, the required improvement is the RL training framework of the RL cache emulator. Due to the characteristics of the Icarus emulator, a specific RL training framework should be designed to perform the entire training process, and the data flow also needs to be carefully designed and processed to perform the RL training.

[0086] Figure 10 The workflow 1000 architecture of the emulator 1010 implementing the embodiments of the present disclosure by improving the conventional emulator is shown. It should be noted that, Figure 10 The emulator 1010 depicted in the present disclosure can be considered as an example or embodiment of the emulator 120 in Figure 1 In order to provide the emulator 1010 for the RL cache algorithm, for example, a pure code-based emulator. In the present disclosure, the workflow of the emulator 1010 is first designed, some parts of which (such as the simulation process) can be adapted from the Icarus emulator. Then, in the present disclosure, the output (state and reward) and input (action) of the system are designed, and finally the detailed training framework is proposed to train the agent on the RL emulator.

[0087] In the workflow 1000, the emulator 1010 can perform topology and content initialization 1012, content request and delivery 1014, and content caching 1016, respectively. The agent 1030 (which can be an example of the agent 130 of Figure 1 The agent 1030 can instruct the emulator 1010 how to perform the content caching 1016. After completing each cycle, the emulator 1010 can store the training data in the data pool 1040. After completing N cycles, the training data stored in the data pool 1040 can be provided to the agent 1030 for training the agent 1030, for example, updating the reinforcement learning model of the agent 1030.

[0088] During the simulation, although the Icarus simulator does a good job in simulating the ICN content delivery and caching process, some details are ignored. Among these details, the content representation is the most important. The content attributes are also important when deciding the caching strategy. But in the Icarus simulator, the content is represented in a pure number (content ID), which loses a lot of information during the training. Therefore, the embodiments of the present disclosure also store the attributes of these contents, for example, stored in a Python dictionary. Specifically, the attributes added to the content dictionary can include request frequency, cache time, and popularity, etc. The request frequency can be defined as counting the number of requests for each content in a period of time (e.g., 1000 requests), and then storing the pair of (time, request source). The cache time is the time of caching each content, which can also be recorded in the dictionary. The popularity is calculated according to the request frequency and the last request time, and the final value is used to represent the popularity of a node or content.

[0089] Regarding the input and return value of the simulator 1010, after completing the simulation part, the simulator 1010 needs to be adapted to become an RL simulator. Regarding the state representation, since the topology of ICN does indeed affect the caching decision, the state can contain two parts: the topology structure and the node features. The original topology can be constructed based on the FNSS, which can return the adjacency matrix and the node matrix, which can perfectly describe the topology of the graph. The node features can include node type, cache state, and content attributes, etc. The node type can refer to the three types of nodes in ICN. The source node is the node that stores the content, the target node is the node that requests the content, and the cache node is a special target node with the ability to cache content. The cache state can refer to the content ID cached in each node returned by the simulator. The simulator 1010 will return the current delivery content attributes and cache node state.

[0090] In addition, regarding the reward function, it can be a weighted sum of parameters such as byte hit rate, average delay, bandwidth, etc. Users can easily modify the weights of these parameters in the code and can be designed according to the needs. Table 1 below shows an example of the reward function, which assumes that the user cares most about the delay time, which can then be set to a weight of 15, which can be the largest weight among the three maximum weights.

[0091] Table 1: Example of reward function

[0092] Type Weight Bytes Hit 3 Average Latency 15 Bandwidth 5

[0093] Additionally, regarding the action input to the simulator 1010, the simulator 1010 can accept a binary vector whose dimension is the number of nodes. Each element in this vector can represent the caching selection for each node. During the content delivery phase, if nodes are chosen to be used as agents to train the model, there are only two operations (caching or not), and if the agent is the entire network with n nodes, there are two... n The possible actions determine which node to cache. For example, if there are 2 nodes in this network, the possible action is 10, which means choosing node 1 to cache this content, and not using node 0 this time.

[0094] During the content replacement phase, if a node with K MB of storage space is chosen as the agent to train the model, then a 2 k The possible operations for each item. For example, if K is 2, then there are 10 possible operations, meaning the first MB is used to cache this content, while the last MB is not used. If the agent is the entire network with n nodes, then each node has K MB of space, and there are n... k There are several possible operations. For example, if K is 2 and n is 2, the possible operations should be 10 (caching status of node 1) 01 (caching status of node 2), and the meaning of the caching status of each node is the same as in the previous example.

[0095] Figure 11 The workflow 1100 of a simulator 1010, an embodiment of this disclosure implemented by modifying a conventional simulator, for training an agent 1030 is shown. For example... Figure 11 As shown, since the conventional Icarus simulator was not originally designed for reinforcement learning training, the related experimental workflow should also be modified for reinforcement learning training. The entire training process 1100 can be divided into four parts (modules): initialization part 1110, experiment execution part 1120, transmission part 1130, and storage in storage pool 1140. The initialization part 1110 may further include scene generation 1112 and experiment orchestration 1114. It will be understood that... Figure 11 The storage pool 1140 described in the text can be Figure 10An example of the data pool 1040 in the framework 1000 is shown in FIG. 11. Specifically, in the workflow 1100, the reinforcement learning simulator 1010 is first initialized. It is worth noting that the scenario generation 1112 and experiment orchestration 1114 in the workflow 1100 are different from the conventional Icarus simulator, which can only initialize the configuration of topology and content requests, etc. In this case, the cache policy is not configured, and the cache decision can be made in the experiment execution part. Then, the required training data is collected from the simulator 1010. After collection, these training data can be stored in the storage pool 1140, and the pre-processing process can also be performed in this module. After pre-processing, the training data will be sent and fed into the training module. Finally, the trained reinforcement learning model 1030 can update the value function, and then use the model to generate more simulation results, and start a new cycle. This training process 1100 will not terminate until the reward threshold is reached. In a related test experiment, this framework has been used to test a reinforcement learning algorithm, and satisfactory results have been obtained.

[0096] In some embodiments, the utilization of the simulator 120 of embodiments of the present disclosure can not only be used to train the agent 130, but also can be used to test the performance parameters of the agent 130 after the agent 130 completes the training. More generally, the simulator 120 can be used to test any agent for making cache decisions in a network. By testing the agent 130 (or any other agent) using the simulator 120, the performance of the agent 130 (or any other agent) can be flexibly and conveniently measured, thereby providing a basis for the training effect of the agent 130 and the performance judgment of other agents. An example process of testing the agent 130 based on the simulator 120 will be described below with reference to FIG. 12. Figure 12

[0097] Figure 12 A flowchart showing an example process 1200 of testing the performance parameters of the agent 130 using the simulator 120 according to embodiments of the present disclosure is shown. In some embodiments, the process 1200 can be implemented by the simulator 120 in the computing device 110 in the environment 100, for example, can be implemented by a processor or processing unit of the computing device 110 for implementing the simulator 120, or by various functional modules of the computing device 110 for implementing the simulator 120. In other embodiments, the process 1200 can also be implemented by a simulator in a computing device independent of the environment 100, or can be implemented by other units or modules for implementing the simulator in the environment 100.

[0098] ​At box 1210, the simulator 120 can provide the agent 130 with a test network state representation and a test content request event for the test simulation network 140. For example, the test network state representation and the test content request event can be similar to the network state representation 122 and content request event 124 described above, respectively, except that they are used for testing and training, respectively. Therefore, a detailed description of the content and related processes of the test network state representation and the test content request event will be omitted. At box 1220, the simulator 120 can receive test action information from the agent 130, which indicates test cache actions determined by the agent 130. For example, the test action information can be similar to the action information 126 described above, except that they are used for testing and training, respectively. Therefore, a detailed description of the content and related processes of the test action information will be omitted.

[0099] At box 1230, simulator 120 can determine the performance parameters of agent 130 based on test caching actions performed in test simulation network 140. For example, test caching actions can be similar to the caching actions described above, the difference being that they are used for testing and training respectively. Therefore, a detailed description of the content and related processes of test caching actions will be omitted. In some embodiments, the performance parameters of agent 130 may include the Q-value for each test action given to agent 130. The Q-value can be determined based on the relevant caching and transmission performance parameters of the content in simulation network 140, such performance parameters may include cache hit rate and link load, etc. Furthermore, it should be noted that since example process 1200 is used to test the performance of agent 130, simulator 120 may not provide agent 130 with a reward value generated based on the reward function in example process 1200.

[0100] Figure 13 A Q-network architecture 1300 for training and testing an agent 130 according to an embodiment of this disclosure is shown. Figure 13 As shown, in the Q-network architecture 1300, the network topology 1310 and node features 1320 can be input into a graph convolutional neural network (GCN) 1330, thereby deriving a concatenation 1340 of the outputs of all nodes. This concatenation 1340 can then be input into a fully connected neural network (FCN) 1350, thereby deriving the Q-value 1360 for each action. In test experiments, reinforcement learning agents can be trained using Q-learning. Q-learning can be based on... Figure 13 The Q-network architecture is shown. (As shown...) Figure 13As shown, for the topology information of the network (e.g., ICN), a GCN (graph convolutional neural network) 1330 can be used as a feature extractor, and then a fully connected neural network 1350 is used to obtain the final Q value 1360. In this test experiment, the average cache hit rate and link load are compared with the Least Recently Used (LRU) and Leave Copy Down (LCD) methods, and the experimental results are shown in Figure 14A and Figure 14B .

[0101] Figure 14A and Figure 14B respectively show two experimental results 1400 and 1405 for performance comparison of embodiments according to the present disclosure and conventional methods. In Figure 14A , a comparison of the method (RLCas) of embodiments of the present disclosure with the conventional LRU and LCD methods in terms of cache hit rate is shown, with the horizontal axis being the number of epochs of training and the vertical axis being the tested cache hit rate. In Figure 14B , a comparison of the method (RLCas) of embodiments of the present disclosure with the conventional LRU and LCD methods in terms of link load is shown, with the horizontal axis being the number of epochs of training and the vertical axis being the tested link load. From Figure 14A and Figure 14B it can be seen that the test results of embodiments of the present disclosure are superior to the conventional LRU+LCD method, and are also competitive in the state-of-the-art methods in the field.

[0102] Figure 15 shows a number of improvements 1500 implemented by a simulator according to embodiments of the present disclosure with respect to a conventional information-centric network (ICN) simulator (e.g., Icarus). As mentioned above, in some implementations, the simulator of embodiments of the present disclosure can be implemented by improving a conventional ICN simulator, for example, so as to derive a simulator 1010 shown in Figure 10 . More specifically, in order to train and test a reinforcement learning strategy for cache decision, the simulator 1010 can be considered as an improvement for a conventional Python-based ICN simulator in terms of reinforcement learning algorithm training. In this regard, Figure 15 explicitly shows a number of differences or improvements 1500 between the simulator 1010 and the conventional ICN simulator.

[0103] As Figure 15As shown, the simulator 1010 disclosed herein possesses all the functionalities of a reinforcement learning training environment, including input, output, environment setup, and data processing workflows, features not included in the original Icarus simulator. Furthermore, the simulator 1010 of this disclosure adds additional features such as content representation to make training results more reliable. In addition, a major contribution includes providing an easy-to-use reinforcement learning simulator 1010 for ICN caching systems, capable of performing ICN simulations and reinforcement learning training; almost all reinforcement learning algorithms aimed at finding better caching systems can be trained and tested. Specifically, the simulator 1010 of this disclosure has three main improvements in adapting the original Icarus simulator. First, the RL design, i.e., designing the input and output of the reinforcement learning simulator 1010 to have the same format as other reinforcement learning simulators. Then, reinforcement learning researchers can easily use the simulator 1010 to test their algorithms, much like using OpenAI's Gym API. Another improvement lies in more detailed representation. For example, the original Icarus simulator ignores some details, while the embodiments of this disclosure add more details to simulator 1010 (which is very useful during RL training) to more comprehensively represent the ICN state. Another improvement lies in the new training framework, which adjusts the workflow of the Icarus simulator for general reinforcement learning training.

[0104] Figure 16 A block diagram of an example device 1600 that can be used to implement embodiments of the present disclosure is shown. In some embodiments, device 1600 may be an electronic device that can be used to implement... Figure 1 The computing device 110 in the middle. For example... Figure 16 As shown, device 1600 includes a central processing unit (CPU) 1601, which can perform various appropriate actions and processes according to computer program instructions stored in read-only storage device (ROM) 1602 or loaded from storage unit 1608 into random access storage device (RAM) 1603. Various programs and data required for the operation of device 1600 can also be stored in RAM 1603. CPU 1601, ROM 1602, and RAM 1603 are interconnected via bus 1604. Input / output (I / O) interface 1605 is also connected to bus 1604.

[0105] A number of the components in device 1600 are connected to the I / O interface 1605, including an input unit 1606, such as a keyboard, mouse, etc.; an output unit 1607, such as various types of displays, speakers, etc.; a storage unit 1608, such as a disk, a CD, etc.; and a communication unit 1609, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 1609 allows device 1600 to exchange information / data with other devices over a computer network, such as the Internet, and / or various telecommunication networks.

[0106] The various processes and processes described above, such as the example methods or example processes, can be performed by processing unit 1601. For example, in some embodiments, the various example methods or example processes can be implemented as a computer software program tangibly embodied in a machine-readable medium, such as storage unit 1608. In some embodiments, some or all of the computer program can be loaded and / or installed onto device 1600 via ROM 1602 and / or communication unit 1609. When the computer program is loaded onto RAM 1603 and executed by CPU 1601, one or more of the steps of the example methods or example processes described above can be performed.

[0107] As used herein, the term “includes” and its variants are to be read as open-ended terms that mean “includes, but is not limited to.” The term “based on” is to be construed as “based at least in part on.” The term “one embodiment” or “the embodiment” are to be read as “at least one embodiment.” The terms “a” and “an” are to be read as “at least one” and “one or more.” Other explicit and implicit definitions can also be included herein.

[0108] As used herein, the term “determining” encompasses a wide variety of actions. For example, “determining” can include calculating, computing, processing, deriving, investigating, looking up (such as, for example, looking up in a table, a database or another data structure), ascertaining and the like. Also, “determining” can include receiving (such as, for example, receiving information), accessing (such as, for example, accessing data in a memory) and the like. Also, “determining” can include resolving, selecting, choosing, establishing and the like.

[0109] It should be noted that embodiments of the present disclosure can be implemented by hardware, software, or a combination of software and hardware. The hardware portion can be implemented by using special logic; the software portion can be stored in a memory and executed by an appropriate instruction execution system, such as a microprocessor or a specially designed hardware. Those skilled in the art can understand that the above-mentioned device and method can be implemented using computer executable instructions and / or included in processor control code, such as providing such code on a programmable memory or a data carrier such as an optical or electronic signal carrier.

[0110] Further, although operations of the methods of the present disclosure are described in a particular order in the figures, this is not required or implied in any particular order to perform the operations, or that all operations be performed, to achieve desirable results. Rather, the steps depicted in the flowcharts can change order. Additionally or alternatively, certain steps can be omitted, combined into a single step, and / or separated into multiple steps. It should also be noted that features and functions of two or more devices according to the present disclosure can be embodied in a single device. Conversely, features and functions of one device described above can be further partitioned into multiple devices.

[0111] While the present disclosure has been described with reference to several particular embodiments, it is to be understood that the disclosure is not limited to the specific embodiments disclosed. Rather, the disclosure is intended to cover any modifications and equivalent arrangements included within the spirit and scope of the appended claims.

Claims

1. A method for processing information, comprising: providing, from a simulator, a first network state representation of a simulated network and a first content request event to an agent for reinforcement learning, the first content request event representing a request node in the simulated network requesting a target content stored in a source node; receiving, from the agent, first action information indicating a first caching action determined by the agent, the first caching action comprising caching the target content in at least one caching node between the request node and the source node; and collecting first training data for training the agent based on performing the first caching action in the simulated network; wherein the first caching action is selected from a first action set, the first action set is represented using a first bit set, a number of bits of the first bit set is equal to a number of nodes of the simulated network, and each bit indicates whether a corresponding node is to cache the target content.

2. The method of claim 1, wherein the first training data comprises the first network state representation, the first action information, and a reward value for the first caching action.

3. The method of claim 2, wherein the reward value is determined based on a reward function of the simulator, the reward function being based on at least one of: a hit rate of cached content, an average latency of requested content, and a bandwidth occupied by transmitting requested content.

4. The method of claim 3, wherein the hit rate, the average latency, and the bandwidth each have an adjustable weight in the reward function.

5. The method of claim 1, further comprising: updating the first network state representation based on the first caching action to determine a second network state representation of the simulated network; providing the second network state representation and a second content request event to the agent; receiving, from the agent, second action information indicating a second caching action determined by the agent; and collecting second training data for training the agent based on performing the second caching action in the simulated network.

6. The method of claim 5, further comprising: in accordance with a determination that a number of collected training data reaches a predetermined number, providing the predetermined number of collected training data to the agent.

7. The method of claim 6, further comprising: initializing the simulator to determine the first network state representation, the first content request event, and the predetermined number.

8. The method of claim 1, further comprising: performing the first caching action in the simulated network based on a time step determined by a time step function of the simulator.

9. The method of claim 1, further comprising: in accordance with a determination that the first training data is collected, storing the first training data; and performing pre-processing on the first training data.

10. The method of claim 1, further comprising: providing, to the agent, a test network state representation of a test simulated network and a test content request event; receive test action information from the agent, the test action information indicating a test caching action determined by the agent; and determine a performance parameter of the agent based on performing the test caching action in the test simulated network. 11.The method of claim 1, wherein the first network state representation comprises a network topology and node characteristics. 12.The method of claim 11, wherein the node characteristics comprise at least one of a node type, whether the node caches content, a content identifier of content cached in the node, and content characteristics of content cached in the node. 13.The method of claim 12, wherein the content characteristics comprise at least one of a request frequency of the content, a time for which the content is cached, and a recent hotness of the content, the recent hotness being determined based on the request frequency and a time at which the content was last requested. 14.The method of claim 1, wherein if the agent is for the simulated network, a number of actions in the first action set depends on a number of nodes in the simulated network. 15.The method of claim 1, wherein if the agent is for one node in the simulated network, the first action set comprises two actions. 16.The method of claim 1, wherein the simulator is implemented in a single computing device. 17.An information processing method, comprising: providing, from a simulator to an agent for reinforcement learning, a first network state representation of a simulated network and a first content request event, the first content request event representing a request node in the simulated network requesting target content stored in a source node; receiving first action information from the agent, the first action information indicating a first caching action determined by the agent, the first caching action comprising caching the target content in at least one caching node between the request node and the source node; and collecting first training data for training the agent based on performing the first caching action in the simulated network; wherein the first caching action is selected from an action set, an action in the action set indicating caching the target content in a target storage area in a node. 18.The method of claim 17, wherein if the agent is for the simulated network, a number of actions in the action set depends on a number of nodes in the simulated network and a number of storage areas in a node. 19.The method of claim 17, wherein if the agent corresponds to one node in the simulated network, a number of actions in the action set depends on a number of storage areas of the node. 20.An electronic device, comprising: at least one processor; and at least one memory storing computer program instructions, the at least one memory and the computer program instructions being configured to, with the at least one processor, cause the electronic device to: providing, from a simulator, a first network state representation of a simulated network and a first content request event to an agent for reinforcement learning, the first content request event representing a request node in the simulated network requesting target content stored in a source node; receiving, from the agent, first action information indicating a first caching action determined by the agent, the first caching action comprising caching the target content in at least one caching node between the request node and the source node; and collecting first training data for training the agent based on performing the first caching action in the simulated network; wherein the first caching action is selected from a first action set, the first action set is represented using a first set of bits, a number of bits of the first set of bits is equal to a number of nodes of the simulated network, and each bit indicates whether a corresponding node is to cache the target content.

21. The electronic device of claim 20, wherein the first training data comprises the first network state representation, the first action information, and a reward value for the first caching action.

22. The electronic device of claim 21, wherein the reward value is determined based on a reward function of the simulator, the reward function being based on at least one of: a hit rate of cached content, an average latency of requested content, and a bandwidth occupied by transmitting requested content.

23. The electronic device of claim 22, wherein the hit rate, the average latency, and the bandwidth each have an adjustable weight in the reward function.

24. The electronic device of claim 20, wherein the at least one memory and the computer program instructions are further configured to, with the at least one processor, cause the electronic device to: update the first network state representation based on the first caching action to determine a second network state representation of the simulated network; provide the second network state representation and a second content request event to the agent; receive, from the agent, second action information indicating a second caching action determined by the agent; and collect second training data for training the agent based on performing the second caching action in the simulated network.

25. The electronic device of claim 24, wherein the at least one memory and the computer program instructions are further configured to, with the at least one processor, cause the electronic device to: provide the predetermined number of collected training data to the agent in accordance with a determination that a number of collected training data reaches a predetermined number.

26. The electronic device of claim 25, wherein the at least one memory and the computer program instructions are further configured to, with the at least one processor, cause the electronic device to: initialize the simulator to determine the first network state representation, the first content request event, and the predetermined number.

27. The electronic device of claim 20, wherein the at least one memory and the computer program instructions are further configured to, with the at least one processor, cause the electronic device to: perform the first caching action in the simulated network based on a time step determined by a time step function of the simulator.

28. The electronic device of claim 20, wherein the at least one memory and the computer program instructions are further configured to, with the at least one processor, cause the electronic device to: store the first training data in accordance with a determination that the first training data is collected; and perform pre-processing on the first training data.

29. The electronic device of claim 20, wherein the at least one memory and the computer program instructions are further configured to, with the at least one processor, cause the electronic device to: provide the agent with a test network state representation and a test content request event of a test simulated network; receive test action information from the agent, the test action information indicating a test caching action determined by the agent; and determine a performance parameter of the agent based on performing the test caching action in the test simulated network.

30. The electronic device of claim 20, wherein the first network state representation includes a network topology and node characteristics.

31. The electronic device of claim 30, wherein the node characteristics include at least one of a node type, whether a node caches content, a content identifier of content in a node, and content characteristics of content in a node.

32. The electronic device of claim 31, wherein the content characteristics include at least one of a request frequency of content, a time content is cached, and a recent hotness of content determined based on the request frequency and a time content was last requested.

33. The electronic device of claim 20, wherein a number of actions in the first action set depends on a number of nodes in the simulated network if the agent is for the simulated network.

34. The electronic device of claim 20, wherein the first action set includes two actions if the agent is for one node in the simulated network.

35. The electronic device of claim 20, wherein the simulator is implemented in a single computing device.

36. An electronic device, comprising: at least one processor; and at least one memory storing computer program instructions, the at least one memory and the computer program instructions configured to, with the at least one processor, cause the electronic device to: provide, from a simulator, an agent for reinforcement learning with a first network state representation of a simulated network and a first content request event, the first content request event representing a requesting node in the simulated network requesting target content stored in a source node; receive first action information from the agent, the first action information indicating a first caching action determined by the agent, the first caching action comprising caching the target content in a target storage area in at least one cache node between the request node and the source node; and collect first training data for training the agent based on performing the first caching action in the simulated network; wherein the first caching action is selected from a set of actions, an action in the set of actions indicating caching the target content in a target storage area in a node.

37. The electronic device of claim 36, wherein if the agent is for the simulated network, a number of actions in the set of actions depends on a number of nodes in the simulated network and a number of storage areas in a node.

38. The electronic device of claim 36, wherein if the agent corresponds to one node in the simulated network, a number of actions in the set of actions depends on a number of storage areas of the node.

39. A computer program product tangibly stored in a non-transitory computer readable medium and comprising machine executable instructions that, when executed, cause a machine to perform the steps of the method of any one of claims 1-16 or claims 17-19.

Images