Model training method, data processing method, electronic device and program product

Abstract

Embodiments of the present disclosure relate to a model training method, a data processing method, an electronic device, and a computer program product. The method includes: obtaining storage information associated with a simulated network environment; and using simulated data to train a reinforcement learning model based on simulated data read requests for nodes in a plurality of nodes included in the simulated network environment, each of which has a cache memory. Using the technical solution of the present disclosure, a data cache scheme that meets predetermined criteria can be determined in a dynamic environment by using a reinforcement learning model, thereby solving the cache memory allocation and replacement problems at the same time, which can not only improve the accuracy and efficiency of determining the data cache scheme with less cost overhead, but also improve the user experience of users using the cache system.

Description

Technical Field

[0001] Embodiments of the present disclosure generally relate to computer technology, and specifically to a model training method, a data processing method, an electronic device, and a computer program product, which can be used in the field of cache memory management. Background Art

[0002] Information-centric networking (ICN) is an approach to moving Internet infrastructure away from the host-centric paradigm. At the same time, information-centric networking also brings new challenges regarding cache management, content addressing, routing, etc. There are two basic characteristics in information-centric networking: (1) content is accessed by name; and (2) caches are universal in the network.

[0003] Traditionally, there are two main issues to consider in the problem of managing cache systems in information-centric networks: the first is cache replacement, and the second is cache allocation. From the perspective of a single node, how to choose a cache replacement method for each network node, i.e., the policy should manage decisions about which objects should be cached, the duration they should be cached, and if the cache is full, decisions about evicting those from the cache. Regarding cache allocation. From the perspective of the entire network, how to best distribute data among all network nodes, i.e., the policy determines which node should be used to cache the desired content. In addition to the above issues, the topology of an information-centric network may sometimes change, which makes it difficult to adaptively solve this problem using traditional rule-based methods. Rule-based methods are static and cannot adapt to changes in workloads, resulting in caches operating in a suboptimal state, thereby affecting the user experience of users of cache management products. Summary of the invention

[0004] Embodiments of the present disclosure provide a model training method, an electronic device, and a computer program product.

[0005] In a first aspect of the present disclosure, a model training method is provided. The method includes: obtaining storage information associated with a simulated network environment, the simulated network environment includes multiple nodes each having a cache memory, the storage information indicates the usage of the cache memory of each node; and using simulated data, training a reinforcement learning model based on simulated data read requests for nodes in the multiple nodes, the reinforcement learning model includes an environment corresponding to the simulated network environment, the reinforcement learning model includes an agent corresponding to the multiple nodes, a state corresponding to the storage information, an action corresponding to a candidate data cache scheme, and a reward corresponding to a performance evaluation of the candidate data cache scheme for the simulated data read request, the reinforcement learning model is trained so that the performance evaluation in the reinforcement learning model meets a predetermined performance evaluation requirement, so as to be used to determine a data cache scheme for input data.

[0006] In a second aspect of the present disclosure, a data processing method is provided, which includes: receiving input data; and determining a data cache scheme for the input data using a reinforcement learning model obtained according to the first aspect of the present disclosure.

[0007] In a third aspect of the present disclosure, an electronic device is provided. The electronic device includes: at least one processing unit; and at least one memory, the at least one memory is coupled to the at least one processing unit and stores instructions for execution by the at least one processing unit, the instructions, when executed by the at least one processing unit, cause the device to perform an action, the action including: obtaining storage information associated with a simulated network environment, the simulated network environment including multiple nodes each having a cache memory, the storage information indicating the usage of the cache memory of each node; and using simulated data, training a reinforcement learning model based on simulated data read requests for nodes in the multiple nodes, the reinforcement learning model including an environment corresponding to the simulated network environment, the reinforcement learning model including an agent corresponding to the multiple nodes, a state corresponding to the storage information, an action corresponding to a candidate data cache scheme, and a reward corresponding to a performance evaluation of the candidate data cache scheme for the simulated data read request, the reinforcement learning model is trained so that the performance evaluation in the reinforcement learning model meets a predetermined performance evaluation requirement, so as to determine a data cache scheme for input data.

[0008] In a fourth aspect of the present disclosure, an electronic device is provided. The electronic device includes: at least one processing unit; and at least one memory, the at least one memory is coupled to the at least one processing unit and stores instructions for execution by the at least one processing unit, and when the instructions are executed by the at least one processing unit, the device performs an action, the action including: receiving input data; and using the reinforcement learning model obtained according to the third aspect of the present disclosure, determining a data cache scheme for the input data.

[0009] In a fifth aspect of the present disclosure, a computer program product is provided. The computer program product is tangibly stored on a non-transitory computer readable medium and includes machine executable instructions, which, when executed, cause a machine to perform any step of the method described in the first aspect of the present disclosure.

[0010] In a sixth aspect of the present disclosure, a computer program product is provided, which is tangibly stored on a non-transitory computer-readable medium and includes machine-executable instructions, which, when executed, cause a machine to perform any step of the method described in the first aspect of the present disclosure.

[0011] The Summary is provided to introduce a selection of concepts in a simplified form that are further described in the Detailed Description below. The Summary is not intended to identify key features or essential features of the embodiments of the present disclosure, nor is it intended to limit the scope of the embodiments of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] The above and other objects, features and advantages of the present disclosure will become more apparent through a more detailed description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, wherein like reference numerals generally represent like components throughout the exemplary embodiments of the present disclosure.

[0013] Figure 1 A schematic diagram showing an example environment 100 in which devices and / or methods according to embodiments of the present disclosure may be implemented;

[0014] Figure 2 A schematic diagram of a simulated network environment 200 according to an embodiment of the present disclosure is shown;

[0015] Figure 3 A schematic diagram of a reinforcement learning model 300 according to an embodiment of the present disclosure is shown;

[0016] Figure 4 A flow chart of a model training method 400 according to an embodiment of the present disclosure is shown;

[0017] Figure 5A flow chart of a data processing method 500 according to an embodiment of the present disclosure is shown;

[0018] Figure 6 A schematic diagram showing an average cache hit rate comparison 600 according to an embodiment of the present disclosure is shown;

[0019] Figure 7 A schematic diagram showing a link load comparison 700 according to an embodiment of the present disclosure; and

[0020] Figure 8 A schematic block diagram of an example device 800 that may be used to implement embodiments of the present disclosure is shown.

[0021] In the various drawings, the same or corresponding reference numerals represent the same or corresponding parts. DETAILED DESCRIPTION

[0022] The preferred embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although the preferred embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided to make the present disclosure more thorough and complete, and to fully convey the scope of the present disclosure to those skilled in the art.

[0023] As used herein, the term "including" and its variations represent open inclusion, for example, "including but not limited to". Unless otherwise stated, the term "or" means "and / or". The term "based on" means "based at least in part on". The terms "an example embodiment" and "an embodiment" mean "at least one embodiment". The term "another embodiment" means "at least one additional embodiment". The terms "first", "second", etc. may refer to different or the same objects. Other explicit and implicit definitions may also be included below.

[0024] Information-centric networks have alleviated the overwhelming workload to some extent and brought about new technologies such as 5G. The caching mechanism is one of the core ideas of information-centric networks, which does bring many benefits to the network. However, at the same time, it also brings some new problems. Although some algorithms have been developed to solve these problems, they cannot solve them in an adaptive and integrated way.

[0025] Specifically, information-centric networking aims to change the focus of the current Internet architecture. The previous architecture focused on creating a conversation between two machines. This is evident in the naming system, where a URL is resolved through DNS to indicate the machine to communicate with in order to receive data or perform an action. The goal of information-centric network architecture is to shift the focus from connecting to another machine to fetching data. In information-centric networking, data becomes independent of location, application, storage, and transmission methods, thus supporting caching and replication in the network.

[0026] The cache stores a subset of the data and acts as a layer that provides fast data access to clients while reducing traffic pressure on the origin server. The cache can be local to the device, such as a smartphone memory, at the edge of a network, such as a content delivery network CDN, hosted near a database server, such as Redis, or a mix of the foregoing.

[0027] To some extent, information-centric networks solve some problems by shifting the focus, but they also raise some new problems. For example, when delivering content, which node should be selected to cache this content, which involves cache memory allocation. After selecting the node to store the content, a second problem arises. The second problem involves the limited size of these cache nodes, especially the cache nodes deployed on client devices. When the cache memory exceeds its maximum capacity, a cache memory eviction algorithm should be used to decide which elements to evict, which involves cache replacement.

[0028] However, in traditional cache management products, rule-based methods are difficult to adaptively solve the aforementioned problems. Since rule-based methods are static and cannot adapt to changes in workloads, caches operate in a suboptimal state and are difficult to adapt to changes in network topology and network parameters, and cannot be easily used in new networks, the user experience of users of cache management products is affected.

[0029] In order to at least partially solve one or more of the above-mentioned problems and other potential problems, an embodiment of the present disclosure proposes a heuristic modeling method to use a reinforcement learning method to solve the problems occurring in the caching process, which method is effective with respect to integration and adaptation. In general, in the embodiments of the present disclosure, the main focus is on how to use a reinforcement learning method to determine a data caching scheme that meets predetermined criteria in a dynamic environment, which can simultaneously solve the cache memory allocation and replacement problems. The reinforcement learning solution to the cache strategy design problem proposed in the embodiments of the present disclosure can be referred to as a reinforcement learning caching system (RLCaS).

[0030] Figure 1 A schematic block diagram of a model training environment 100 in which the model training method in some embodiments of the present disclosure can be implemented is shown. According to an embodiment of the present disclosure, the model training environment 100 can be a cloud environment.

[0031] like Figure 1 As shown in , the model training environment 100 includes a computing device 110. In the model training environment 100, training-related data 120 including a simulated network environment and simulated data and a simulated data read request for training a reinforcement learning model 130 are provided to the computing device 110 as inputs of the computing device 110. According to an embodiment of the present disclosure, the training-related data 120 may also include storage information associated with the simulated network environment, topology information of multiple nodes included in the simulated network environment, and data transmission information in the simulated network environment.

[0032] The computing device 110 may interact with the reinforcement learning model 130. For example, the computing device 110 may provide at least a portion of the training-related data 120 to the reinforcement learning model 130, receive from the reinforcement learning model 130 a data caching scheme determined by the reinforcement learning model 130 based on the training-related data 120 and a simulation data read request, and issue an instruction to the reinforcement learning model 130 to stop training by determining whether the data caching scheme determined by the reinforcement learning model 130 can meet predetermined requirements.

[0033] According to an embodiment of the present disclosure, the simulation data and simulation data reading requests used to train the reinforcement learning model 130 may not be included in the training-related data 120, but may be provided to the reinforcement learning model 130 by the computing device 110 or generated by a generator included in the reinforcement learning model 130 itself according to predetermined requirements or randomly.

[0034] Figure 2FIG. 2 shows a schematic diagram of a simulated network environment 200 according to an embodiment of the present disclosure. Figure 2 As shown in , the simulated network environment 200 includes a content server 210, a plurality of nodes 220-1, 220-2, 220-3 and 220-4 which may be collectively referred to as nodes 220, and clients 230-1 and 230-2 which request data and may be collectively referred to as clients 230. It should be understood that the simulated network environment 200 may also include more content servers 210, more nodes and more clients. The node 220 includes a cache memory, and when the client 230 requests data from a node in the nodes 220 in the simulated network environment 200, if the data is not cached in the node, the data needs to be read from the content server 210 to the node for being read by the client 230. According to an embodiment of the present disclosure, the node 220 may include a router.

[0035] In order to enable the client 230 to respond to subsequent read requests for this data more quickly, the simulated network environment 200 can cache this data in at least one of the nodes 220. However, it is a difficult problem to determine in which nodes to cache this data. In traditional solutions, solutions such as leave copy everywhere (LEC), move copy done (MCD), and leave copy down (LCD) can be adopted.

[0036] In the ubiquitous replica solution, the same data is cached in each node, so that the client can get the fastest response when issuing a data read request. However, this solution requires a large cache memory space, and when the data is changed, the data in the cache memory in each node needs to be refreshed, resulting in a large refresh traffic.

[0037] In the mobile copy completion scheme, data is cached only in the cache memory of a node from which the data has been read, thereby making it possible to greatly save cache memory space. However, this scheme will obviously result in a very low data read hit rate.

[0038] In the scheme of making the copy down, when the client 230 reads data hits on a certain node, the data is cached only in the path from the content server 210 to this node and in the direct neighbor nodes of this node. This scheme is a compromise scheme relative to the first two schemes, which neither requires too much cache memory space nor leads to too low data read hit rate. However, since in this scheme, the method of determining the node for caching content is too single and rigid, it is difficult to provide a better data caching scheme.

[0039] From the perspective of speeding up data reading alone, it is expected that when the client 230 requests to read data from a node in the simulated network environment 200, there is a high probability that the requested data has been cached in the cache memory of this node, and at the same time, this data is not cached in the cache memories of too many nodes in the simulated network environment 200.

[0040] As mentioned above, when using a cache system, the problem of cache replacement also needs to be considered. In traditional solutions, this problem can be handled based on rules such as first in first out (FIFO), least recently used (LRU), and least frequently used (LFU).

[0041] However, these methods are not flexible and are not suitable for dynamic environments. At the same time, these methods are obviously rule-based, which also means that they can only achieve suboptimal states and cannot be adjusted for different workloads.

[0042] Figure 3 A schematic diagram of a reinforcement learning model 300 according to an embodiment of the present disclosure is shown. Reinforcement learning (RL) is a subtype of machine learning that learns how to operate dynamically in any setting. It has been successfully applied in a variety of problems, such as controlling complex robots, playing computer games, etc. As mentioned earlier, due to the dynamic environment, reinforcement learning becomes a good choice in such conditions. Using reinforcement learning, the system learns an ideal caching strategy that is customized for the individual system and its observed traffic patterns without any prior assumptions about the data.

[0043] However, there is no integrated solution in traditional techniques that can solve both problems at the same time. Moreover, although some models are beginning to add reinforcement learning to their models, there is no pure reinforcement learning solution for this problem, and current hybrid methods are not flexible enough, especially when the topology of the network changes.

[0044] Regarding reinforcement learning, due to its universality, it has been studied in many disciplines, such as game theory, control theory, operations research, information theory, simulation-based optimization, etc. In general, reinforcement learning is a sequential decision-making process in which an agent learns the best strategy through interaction with the environment.

[0045] like Figure 3 As shown in FIG. 3 , the reinforcement learning model 300 includes an agent 310 and an environment 320. An arrow 330 pointing from the environment 320 to the agent 310 indicates that a state is provided from the environment 320 to the agent 310, an arrow 340 pointing from the agent 310 to the environment 320 indicates that an action by the agent 310 is provided from the agent 310 to the environment 320, and an arrow 350 pointing from the environment 320 to the agent 310 indicates a reward provided from the environment 320 to the agent 310.

[0046] The goal of the agent 310 is to maximize the expected sum of future rewards, which is accomplished by training multiple steps of, for example, state-action transitions and selecting actions that return the maximum discounted expected reward function. The purpose of the method proposed in the embodiment of the present disclosure is to establish an integrated system. According to an embodiment of the present disclosure, more factors will be considered in such a system, and the different needs of multiple scenarios can be reflected based on a flexible reward algorithm. For example, sometimes there may not be much space in a cache node, so the most important requirement is to consider space, followed by requirements for efficiency. In this case, more weight will be given to rewards related to space, and therefore less weight will be given to rewards related to efficiency.

[0047] According to an embodiment of the present disclosure, reinforcement learning algorithms can be divided into two categories, namely model-based and model-free. In a model-free algorithm, the agent has no prior knowledge of the system, but performs operations in the actual environment to understand its impact on the environment. Therefore, the model-free algorithm does not require any input from the developer, is very flexible, and is easy to set up. In contrast, the model-based algorithm learns the model of the environment, and then the agent learns from the interaction with the resulting model. The model provides more simulation steps for the agent, so that the agent can converge toward the optimal strategy faster. However, the model-based approach is very challenging to design because the model must accurately reflect the real environment. Therefore, in the embodiments described in the subsequent description of the present disclosure, a model-free reinforcement learning algorithm is selected for the simplicity of utilizing this system in practice, and a weighted reward system is used to reflect the requirements in certain situations.

[0048] Back to Figure 1 ,exist Figure 1In the model training environment 100 shown, input of training related data 120 to the computing device 110 and interaction between the computing device 110 and the reinforcement learning model 130 can be performed through a network. In addition, in the embodiments of the present disclosure, the computing device 110 is used as an example for explanation, and the computing device 110 may include any physical or virtual central processing unit, dedicated processing unit, and proprietary accelerator, etc. However, the protection scope of the present disclosure is not limited thereto, but may be applicable to various computing elements, units, modules, or systems, etc. that can provide input and output processing capabilities.

[0049] It should be understood that the example environment 100 is merely exemplary and not restrictive, and it is expandable or scalable. For example, the example environment 100 may include more computing devices 110, and more training-related data 120 may be provided to the computing device 110 as input, and the computing device 110 may also interact with more reinforcement learning models 130, so that more users can use more computing devices 110 at the same time, and even use more training-related data 120 to train the reinforcement learning model 130 simultaneously or non-simultaneously.

[0050] The following Figure 1 The computing device 110, training related data 120 and reinforcement learning model 130 included in the example are used to illustrate Figure 4 and Figure 5 The model training method 200 and the data processing method 500 shown in .

[0051] Figure 4 4 shows a flow chart of a model training method 400 according to an embodiment of the present disclosure. The method 400 may be Figure 1 The model training method 400 may be implemented by the computing device 110 shown in the figure, or may be implemented by other suitable devices. It should be understood that the model training method 400 may also include additional steps not shown and / or may omit the steps shown, and the scope of the embodiments of the present disclosure is not limited in this respect.

[0052] In block 402, the computing device 110 obtains storage information associated with the simulated network environment. According to an embodiment of the present disclosure, the simulated network environment involved in block 402 may be Figure 2 The simulated network environment 200 shown in FIG. Figure 2 As shown in , the simulated network environment 200 includes a plurality of nodes 220, and each node 220 has a cache memory that can be used to cache simulated data that a client attempts to read from the simulated network environment 200. The storage information associated with the simulated network environment 200 can indicate the usage of the cache memory of each node 220, including information such as the size of the cache memory in each node 220 and the portion that has been used.

[0053] According to some embodiments of the present disclosure, the computing device 110 may also obtain topological information of nodes in the simulated network environment 200. According to other embodiments of the present disclosure, the computing device 110 may also obtain topological information of nodes in the simulated network environment 200 and data transmission information in the simulated network environment 200, for example, information such as delay and bandwidth of data transmission between nodes.

[0054] In block 404, the computing device 110 uses simulated data for simulating data read in the simulated network environment 200 to train a reinforcement learning model based on simulated data read requests for nodes in the plurality of nodes included in the simulated network environment 200. According to an embodiment of the present disclosure, the reinforcement learning model involved in block 404 may be Figure 3 The reinforcement learning model 300 shown in FIG. In the reinforcement learning model 300, an environment 320 corresponding to the simulated network environment 200, an agent 310 corresponding to the plurality of nodes 220 in the simulated network environment 200, a state 330 corresponding to storage information associated with the simulated network environment 200, an action 340 corresponding to a candidate data caching scheme in the simulated network environment 200, and a reward 350 corresponding to a performance evaluation of the candidate data caching scheme for a simulated data read request are included.

[0055] According to an embodiment of the present disclosure, a plurality of agents 310 corresponding to the plurality of nodes 220 in the simulated network environment 200 may also be included in the reinforcement learning model 300. For example, each node corresponds to an agent. In these embodiments, information related to the respective states and operations, etc., may be shared between agents through additional information interaction between agents, and it may be helpful to realize a more fine-grained, node-based study of the reinforcement learning model 300.

[0056] According to an embodiment of the present disclosure, in box 404, the reinforcement learning model 300 is trained so that the performance evaluation in the reinforcement learning model 300 meets the predetermined performance evaluation requirements. At this time, the computing device 110 can instruct the reinforcement learning model 300 to stop training, or the reinforcement learning model 300 can automatically stop training. After training, the reinforcement learning model 300 can determine a data cache scheme for the input data, and the determined data cache scheme has a higher performance evaluation. The input data may include, for example, data stored in the content server 210 of the simulated network environment 200 and to be read by the client 230. Since this data can be input into the reinforcement learning model 300 so that the reinforcement learning model 300 is trained, it can be referred to as input data in an embodiment of the present disclosure.

[0057] According to an embodiment of the present disclosure, simulation data and simulation data read requests for training the reinforcement learning model 130 may be received from the environment 320 in the reinforcement learning model 130, and may be provided to the reinforcement learning model 130 by the computing device 110 or generated by a generator included in the reinforcement learning model 130 itself according to predetermined requirements or randomly.

[0058] According to some embodiments of the present disclosure, when the computing device 110 obtains the topology information of the nodes in the simulated network environment 200 in box 402, the reinforcement learning model 130 includes a state 330 corresponding to the storage information and the topology information of the nodes. According to some other embodiments of the present disclosure, when the computing device 110 obtains the topology information of the nodes in the simulated network environment 200 and the data transmission information in the simulated network environment 200 in box 402, the reinforcement learning model 130 includes a state 330 corresponding to the storage information, the topology information of the nodes, and the data transmission information.

[0059] As mentioned above, the state 330 in the reinforcement learning model 300 may correspond to the storage information associated with the simulated network environment 200. Therefore, in the embodiment of the present disclosure, the storage information may also be referred to as a state space. For the state space, if a single node is used as an agent, the state space should be relatively simple. For example, the size of the possible state space may be 2*2*2 k , where the first 2 indicates whether the node is to be used for caching data, the second 2 indicates the current stage, such as the allocation stage or the replacement stage, and k indicates a discrete cache space, such as dividing the cache memory space into k blocks, and using 1 to indicate that the kth block has been used for caching data, and using 0 to indicate that the kth block is not used. Similarly, if the entire system is regarded as an intelligent agent, the size of the state space should be 2*k n , where 2 indicates the current stage, such as the allocation stage or the replacement stage, n indicates that there are n nodes in the simulated network environment 200, and k indicates that the cache memory space of each node is divided into k parts.

[0060] For example, if the agent is a node and it has 4MB of cache memory space for caching data, then a possible state 330 may be 101110, where the first 1 indicates that it is in the replacement phase, the first 0 indicates that the node will not be used to cache data, and the following 1110 indicates that the first 3MB of the cache memory space is occupied, while the last 1MB has not yet been used.

[0061] For another example, if all nodes 220 in the simulated network environment 200 are regarded as an intelligent agent, and there are 2 nodes in the simulated network environment 200, each node has 2MB of cache memory space available for caching, then the possible state 330 may be 0010001, where the first 0 indicates that it is in the allocation stage, and the subsequent 01 indicates that node 1 will not be used to cache data, and node 2 will be used to cache data, and the subsequent 00 indicates that the cache memory space of node 1 is empty, and the last 01 indicates that the second MB of node 2 has been occupied.

[0062] It is worth noting that when the agent is in the replacement phase, all bits related to the representation of whether a node should be used to cache data should be 0, because no node will be used to cache data in this phase.

[0063] According to an embodiment of the present disclosure, the data caching scheme corresponding to action 340 in the reinforcement learning model 300 includes at least one of the following: a cache memory in which simulation data or input data is to be cached; and cached data to be cleared from the cache memory, wherein the remaining available space of the cache memory in which simulation data or input data is to be cached is insufficient.

[0064] As mentioned above, the action 340 in the reinforcement learning model 300 may correspond to the candidate data caching scheme in the simulated network environment 200. Therefore, in the embodiment of the present disclosure, the candidate data caching scheme may also be referred to as an action space.

[0065] In the allocation phase, if we choose to use each as an agent to train the model, the action space only includes executing 2 operations, whether to cache or not. If the agent is a simulated network environment 200 with n nodes, the action space includes 2 n There are 10 possible actions to determine which nodes are used for caching. For example, if there are 2 nodes in the simulated network environment 200, the possible action space is 10, which indicates that node 1 is selected to cache space, and node 0 is not used this time.

[0066] In the replacement phase, if a node with k MB of memory is selected as the agent to train the model, the action space consists of 2 k For example, if k is 2, the possible actions can be expressed as 10, which means that the first MB is used to cache data, and the second MB is not used. If the agent is the entire simulated network environment 200 with n nodes, and each node has k MB space, then the action space includes n kFor example, if k is 2 and n is 2, the possible actions can be represented as 1001, where the first 10 represents the cache status of node 1, and the second 01 represents the cache status of node 2.

[0067] According to an embodiment of the present disclosure, if further content is introduced into the simulated network environment 200, the content of the aforementioned state space and action space may be changed accordingly. For example, the content of the state space and action space may be changed based on client requirements, computing resource limitations, or other content.

[0068] According to an embodiment of the present disclosure, training the reinforcement learning model 300 includes obtaining an initial data cache scheme and training the reinforcement learning model 300 based on the initial data cache scheme. For example, before training the reinforcement learning model 300, a previously determined initial data cache scheme suitable for the simulated network environment 200 or a similar simulated network environment may be obtained, and the reinforcement learning model 300 may be trained based on the initial data cache scheme, so that the time required for training the reinforcement learning model 300 can be reduced, and a more preferred data cache scheme can be obtained by training the reinforcement learning model 300.

[0069] According to an embodiment of the present disclosure, training the reinforcement learning model 300 includes obtaining restrictions on the data cache scheme and training the reinforcement learning model 300 based on the restrictions. For example, before training the reinforcement learning model 300, user-defined restrictions on the data cache scheme, such as user preferences, may be obtained first. These restrictions may include, for example, which nodes the user prefers to cache data in and which types of data may be cleared preferentially during the replacement phase. By introducing these restrictions when training the reinforcement learning model 300, the trained data cache scheme may be able to reflect the user's personalized requirements.

[0070] According to an embodiment of the present disclosure, training the reinforcement learning model 300 includes training the reinforcement learning model 300 so that the performance evaluation in the trained reinforcement learning model 300 is better than the performance evaluation of the predetermined candidate data cache scheme for the simulated data read request. For example, the previously determined initial data cache scheme suitable for the simulated network environment 200 or a similar simulated network environment can be first obtained as a candidate data cache scheme, and the performance evaluation of this candidate data cache scheme for the simulated data read request is determined, and by determining that the performance evaluation in the trained reinforcement learning model 300 is better than the performance evaluation of the predetermined candidate data cache scheme for the simulated data read request, it is determined that the performance evaluation in the reinforcement learning model 300 meets the predetermined performance evaluation requirement.

[0071] According to an embodiment of the present disclosure, the performance evaluation may be indicated by a performance evaluation score, the performance evaluation score including a weighted sum of sub-performance evaluation scores of a plurality of sub-performance evaluations, and training the reinforcement learning model 300 includes training the reinforcement learning model 300 so that the performance evaluation score in the trained reinforcement learning model 300 is higher than a predetermined performance evaluation score. For example, the sub-performance evaluation may include an average cache hit rate, an average link delay, an average bandwidth usage, etc. In addition, the scores of these sub-performance evaluations may be weighted and summed, for example, based on specific parameters of the simulated network environment 200 and user preferences, and a sub-performance evaluation of greater concern may be weighted higher, so that the trained data cache scheme can better reflect the preference for the sub-performance evaluation of the tendency.

[0072] According to an embodiment of the present disclosure, the performance evaluation may be indicated by a performance evaluation score, the performance evaluation score including a weighted sum of sub-performance evaluation scores of a plurality of sub-performance evaluations, and training the reinforcement learning model 300 includes using the trained reinforcement learning model 300 to determine a data cache scheme based on a candidate data cache scheme corresponding to the highest performance evaluation score among a plurality of performance evaluation scores in the trained reinforcement learning model 300. For example, the trained reinforcement learning model 300 may provide a certain number of candidate data cache schemes during the training process, for example, in a certain time period, at which time the candidate data cache scheme corresponding to the highest performance evaluation score among a plurality of performance evaluation scores corresponding to these candidate data cache schemes may be determined as the data cache scheme to be used, so that the optimal data cache scheme may be selected from the plurality of data cache schemes. In addition, the time period factor may not be considered, but only after a certain number of candidate data cache schemes are provided by the reinforcement learning model 300, the optimal data cache scheme may be selected from the candidate data cache schemes.

[0073] According to an embodiment of the present disclosure, in one example, Q learning can be used when training the reinforcement learning model 300. For example, the topological information of the simulated network environment 200 and node features such as storage information can be input into a graph convolutional neural network, and the graph convolutional neural network extracts the output of each node as a feature extractor, and these outputs are then cascaded and input into a fully connected neural network, and then the fully connected neural network can output a Q value for each action, which can be associated with the aforementioned performance evaluation score or be the aforementioned performance evaluation score.

[0074] Figure 5 1 shows a flow chart of a data processing method 500 according to an embodiment of the present disclosure. The method 500 may be Figure 1The model training method 500 may be implemented by the computing device 110 shown in the figure, or may be implemented by other suitable devices. It should be understood that the model training method 500 may also include additional steps not shown and / or may omit the steps shown, and the scope of the embodiments of the present disclosure is not limited in this respect.

[0075] In block 502 , the computing device 110 receives input data. According to an embodiment of the present disclosure, the computing device 110 may not directly receive the input data, but may receive an indication of the input data stored in the content server 210 of the simulated network environment 200 to be read by the client 230 .

[0076] In block 504, the computing device 110 determines a data caching scheme for the input data using the reinforcement learning model 300 obtained according to the model training method 400. For example, the computing device 110 may determine, for the input data, in which cache memories of which nodes in the simulated network environment 200 the input data should be cached, and when the input data is cached in these cache memories, which data needs to be cleared from these cache memories, or which parts or ratios of these cache memories need to be cleared.

[0077] References Figures 1 to 5 The following describes an example environment 100 in which the device and / or method according to an embodiment of the present disclosure can be implemented, a simulated network environment 200 according to an embodiment of the present disclosure, a reinforcement learning model 300 according to an embodiment of the present disclosure, a model training method 400 according to an embodiment of the present disclosure, and a data processing method 500 according to an embodiment of the present disclosure. It should be understood that the above description is for the purpose of better illustrating the contents recorded in the embodiments of the present disclosure, and is not intended to be limiting in any way.

[0078] It should be understood that the numbers and sizes of various elements used in the embodiments of the present disclosure and in the various drawings are only examples, and are not intended to limit the protection scope of the embodiments of the present disclosure. The above numbers and sizes can be arbitrarily set as needed without affecting the normal implementation of the embodiments of the present disclosure.

[0079] Through the above reference Figures 1 to 5 According to the description of the embodiment of the present invention, a reinforcement learning cache system (RLCaS) is proposed, thereby solving the problems of the cache system in the information-centric network in a comprehensive manner. Specifically, the technical solution according to the embodiment of the present invention has many advantages over the traditional solution.

[0080] For example, using the technical solution of the present disclosure, since reinforcement learning is fully adopted, the two problems mentioned above can be seamlessly combined compared with manual combination, and an integrated solution as an end-to-end solution can be provided. At the same time, previous studies based on rules and machine learning mainly focus on one of the two problems of cache allocation and replacement that arise in the cache strategy design process, while the technical solution of the present disclosure can consider these two situations to try to find the relatively optimal strategy.

[0081] For another example, using the technical solution of the present disclosure, flexible rewards in reinforcement learning models can be implemented. In traditional solutions, there is no universal system solution that can provide all the benefits, such as efficiency, simplicity, scalability, security, privacy, adaptability, etc., which means that people always need to balance different trade-offs in engineering design. According to the technical solution of the present disclosure, when designing the rewards of an intelligent agent, rewards can be adaptively added or deleted based on the goal, and the weight of the reward can be changed to adjust the importance of the goal.

[0082] For another example, the technical solution of the present disclosure can be used to realize a flexible action space. According to the embodiments of the present disclosure, the action can be an existing paradigm or completely random, so as to meet more requirements.

[0083] For another example, the technical solution of the present disclosure may not require an environment model. According to the embodiments of the present disclosure, it is not necessary to accurately model the environment of the information-centric network in advance, and therefore, no assumptions need to be made.

[0084] For another example, using the technical solution of the present disclosure, it is easy to adapt to new environments. The reinforcement learning model is constantly trained during the entire simulation process, and the direction of training is always to enhance the defined rewards. Therefore, the agent can easily cope with topological differences and can even avoid other changes that occur during the training process.

[0085] For another example, using the technical solution of the present disclosure, no pre-set data set is required for training. Since training data may not always be suitable, there is no need to generate training data in advance. In addition, the defined intelligent agent can learn strategies in a completely new environment.

[0086] The following describes the comparison of technical effects between the data processing method according to the embodiment of the present disclosure and the traditional solution in combination with two comparative examples.

[0087] Figure 6 FIG. 6 is a schematic diagram showing an average cache hit rate comparison 600 according to an embodiment of the present disclosure. Figure 6, the solid line represents the average cache hit rate when the least recently used (LRU) and the keep copies down (LCD) schemes are used according to the traditional scheme, and the dotted line represents the average cache hit rate when the data processing method according to the embodiment of the present disclosure is used. Figure 6 The horizontal axis in represents the number of training times, and its unit can be one round (epoch) of training, and the vertical axis represents the statistical average cache memory hit rate, wherein 0.2 represents a hit rate of 20%, and so on. It can be seen that after a certain number of training times, the data processing method according to the embodiment of the present disclosure can achieve a significantly higher average cache memory hit rate than the traditional scheme using the least recently used (LRU) and the copy down (LCD) scheme.

[0088] Figure 7 FIG. 7 is a schematic diagram showing an average link load comparison 700 according to an embodiment of the present disclosure. Figure 7 In FIG. 1 , the solid line represents the average link load when the least recently used (LRU) and the drop copy down (LCD) schemes are used according to the traditional scheme, and the dotted line represents the average link load when the data processing method according to the embodiment of the present disclosure is used. Figure 7 The horizontal axis in represents the number of training times, and its unit may be one round (epoch) of training, and the vertical axis represents the statistical average link load, and its unit is megabits per second (MBPS). It can be seen that after a certain number of training times, the data processing method according to the embodiment of the present disclosure can achieve a significantly lower average link load than the traditional scheme using the least recently used (LRU) and the copy down (LCD) scheme.

[0089] Figure 8 FIG. 8 is a schematic block diagram of an example device 800 that can be used to implement an embodiment of the present disclosure. According to an embodiment of the present disclosure, Figure 1 The computing device 110 in the embodiment of the present invention can be implemented by the device 800. As shown in the figure, the device 800 includes a central processing unit (CPU) 801, which can perform various appropriate actions and processes according to computer program instructions stored in a read-only memory (ROM) 802 or computer program instructions loaded from a storage unit 808 into a random access memory (RAM) 803. In the RAM 803, various programs and data required for the operation of the device 800 can also be stored. The CPU 801, the ROM 802, and the RAM 803 are connected to each other via a bus 804. An input / output (I / O) interface 805 is also connected to the bus 804.

[0090] A number of components in the device 800 are connected to the I / O interface 805, including: an input unit 806, such as a keyboard, a mouse, etc.; an output unit 807, such as various types of displays, speakers, etc.; a storage unit 808, such as a disk, an optical disk, etc.; and a communication unit 809, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 809 allows the device 800 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0091] The various processes and processing described above, such as method 400 and method 500, may be performed by the processing unit 801. For example, in some embodiments, method 400 and method 500 may be implemented as a computer software program, which is tangibly contained in a machine-readable medium such as storage unit 808. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 800 via ROM 802 and / or communication unit 809. When the computer program is loaded into RAM 803 and executed by CPU 801, one or more actions of method 400 and method 500 described above may be performed.

[0092] Embodiments of the present disclosure may be directed to methods, devices, systems and / or computer program products. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for executing various aspects of the embodiments of the present disclosure.

[0093] Computer readable storage medium can be a tangible device that can keep and store the instructions used by the instruction execution device. Computer readable storage medium can be, for example, but not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination thereof. More specific examples of computer readable storage medium as a non-exhaustive list include: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disk read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanical encoding device, for example, a punch card or a convex structure in a groove on which instructions are stored, and any suitable combination thereof. The computer readable storage medium used here is not interpreted as an instantaneous signal itself, such as a radio wave or other freely propagating electromagnetic waves, for example, an electromagnetic wave propagated by a waveguide or other transmission medium of a light pulse of an optical fiber cable, or an electrical signal transmitted by a wire.

[0094] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to each computing / processing device, or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network can include copper transmission cables, optical fiber transmissions, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions for storage in the computer-readable storage medium in each computing / processing device.

[0095] The computer program instructions for performing the operation of the embodiments of the present disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-related instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages, such as Smalltalk, C++, etc., and conventional procedural programming languages, such as "C" language or similar programming languages. Computer-readable program instructions may be executed completely on a user's computer, partially on a user's computer, as an independent software package, partially on a user's computer, partially on a remote computer, or completely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer via an Internet connection, for example, using an Internet service provider. In some embodiments, an electronic circuit, such as a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), may be personalized by utilizing the state information of a computer-readable program instruction, and the electronic circuit may execute a computer-readable program instruction, thereby realizing various aspects of the embodiments of the present disclosure.

[0096] Various aspects of the embodiments of the present disclosure are described herein with reference to the flowcharts and / or block diagrams of the methods, devices / systems, and computer program products according to the embodiments of the present disclosure. It should be understood that each box in the flowchart and / or block diagram and the combination of each box in the flowchart and / or block diagram can be implemented by computer-readable program instructions.

[0097] These computer-readable program instructions can be provided to a processing unit of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine, so that when these instructions are executed by the processing unit of the computer or other programmable data processing device, a device that implements the functions / actions specified in one or more boxes in the flowchart and / or block diagram is generated. These computer-readable program instructions can also be stored in a computer-readable storage medium, and these instructions cause the computer, programmable data processing device, and / or other equipment to work in a specific manner, so that the computer-readable medium storing the instructions includes a manufactured product, which includes instructions for implementing various aspects of the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0098] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operating steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to implement the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0099] The flow chart and block diagram in the accompanying drawings show the possible architecture, function and operation of the system, method and computer program product according to multiple embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a part of a module, program segment or instruction, and a part of the module, program segment or instruction includes one or more executable instructions for realizing the specified logical function. In some alternative implementations, the function marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two continuous square boxes can actually be executed substantially in parallel, and they can sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs the specified function or action, or can be implemented with a combination of special hardware and computer instructions.

[0100] The embodiments of the present disclosure have been described above, and the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The selection of terms used herein is intended to best explain the principles of the embodiments, practical applications, or technical improvements to the technology in the market, or to enable other persons of ordinary skill in the art to understand the embodiments disclosed herein.

Claims

1. A model training method, comprising: acquiring storage information associated with a simulated network environment, wherein the simulated network environment includes a plurality of nodes each having a cache memory, and the storage information indicates usage of the cache memory of each node; as well as Using simulated data, training a reinforcement learning model based on simulated data read requests for nodes among the plurality of nodes, wherein the reinforcement learning model includes an environment corresponding to the simulated network environment, the reinforcement learning model includes agents corresponding to the plurality of nodes, states corresponding to the storage information, actions corresponding to candidate data cache schemes, and rewards corresponding to performance evaluations of the candidate data cache schemes for the simulated data read requests, and the reinforcement learning model is trained so that the performance evaluations in the reinforcement learning model meet predetermined performance evaluation requirements and so that the reinforcement learning model can be used to determine a data cache scheme for input data; Wherein acquiring the storage information includes acquiring the storage information and topology information of the plurality of nodes; wherein the reinforcement learning model includes the state corresponding to the storage information and the topology information; wherein the performance evaluation is indicated by a performance evaluation score, the performance evaluation score comprising a weighted sum of sub-performance evaluation scores of a plurality of sub-performance evaluations; and Wherein training the reinforcement learning model includes using the trained reinforcement learning model to determine a data cache scheme based on candidate data cache schemes corresponding to higher performance evaluation scores in the trained reinforcement learning model relative to other performance evaluation scores. 2 . The method of claim 1 , wherein the simulated data read request is received from the environment in the reinforcement learning model.

3. The method of claim 1 , wherein the data caching scheme indicates at least one of the following: A cache memory in which said simulation data is to be cached; and The cached data is to be cleared from the cache memory, wherein the remaining available space of the cache memory in which the simulation data is to be cached is insufficient.

4. The method of claim 1, wherein training the reinforcement learning model comprises: Get an initial data cache plan; as well as The reinforcement learning model is trained based on the initial data caching scheme.

5. The method of claim 1, wherein training the reinforcement learning model comprises: Get restrictions on data caching schemes; as well as The reinforcement learning model is trained based on the constraints.

6. The method of claim 1, wherein training the reinforcement learning model comprises: The reinforcement learning model is trained so that the performance evaluation in the trained reinforcement learning model is better than the performance evaluation of a predetermined candidate data cache solution for the simulated data read request.

7. The method of claim 1, wherein the performance evaluation is indicated by a performance evaluation score, the performance evaluation score comprising a weighted sum of sub-performance evaluation scores of a plurality of sub-performance evaluations, and training the reinforcement learning model comprises: The reinforcement learning model is trained so that the performance evaluation score in the trained reinforcement learning model is higher than a predetermined performance evaluation score.

8. The method according to claim 1, further comprising: receiving the input data; as well as The data caching scheme is determined for the input data using the reinforcement learning model.

9. An electronic device, comprising: at least one processing unit; as well as at least one memory coupled to the at least one processing unit and storing instructions for execution by the at least one processing unit, wherein the instructions, when executed by the at least one processing unit, cause the electronic device to perform actions, the actions comprising: acquiring storage information associated with a simulated network environment, wherein the simulated network environment includes a plurality of nodes each having a cache memory, and the storage information indicates usage of the cache memory of each node; and Using simulated data, training a reinforcement learning model based on simulated data read requests for nodes among the plurality of nodes, wherein the reinforcement learning model includes an environment corresponding to the simulated network environment, the reinforcement learning model includes agents corresponding to the plurality of nodes, states corresponding to the storage information, actions corresponding to candidate data cache schemes, and rewards corresponding to performance evaluations of the candidate data cache schemes for the simulated data read requests, and the reinforcement learning model is trained so that the performance evaluations in the reinforcement learning model meet predetermined performance evaluation requirements and so that the reinforcement learning model can be used to determine a data cache scheme for input data; Wherein acquiring the storage information includes acquiring the storage information and topology information of the plurality of nodes; wherein the reinforcement learning model includes the state corresponding to the storage information and the topology information; wherein the performance evaluation is indicated by a performance evaluation score, the performance evaluation score comprising a weighted sum of sub-performance evaluation scores of a plurality of sub-performance evaluations; and Wherein training the reinforcement learning model includes using the trained reinforcement learning model to determine a data cache scheme based on candidate data cache schemes corresponding to higher performance evaluation scores in the trained reinforcement learning model relative to other performance evaluation scores. 10 . The electronic device of claim 9 , wherein the simulated data read request is received from the environment in the reinforcement learning model.

11. The electronic device of claim 9, wherein the data caching scheme indicates at least one of the following: A cache memory in which said simulation data is to be cached; and The cached data is to be cleared from the cache memory, wherein the remaining available space of the cache memory in which the simulation data is to be cached is insufficient.

12. The electronic device of claim 9, wherein training the reinforcement learning model comprises: Get an initial data cache plan; as well as The reinforcement learning model is trained based on the initial data caching scheme.

13. The electronic device of claim 9, wherein training the reinforcement learning model comprises: Get restrictions on data caching schemes; as well as The reinforcement learning model is trained based on the constraints.

14. The electronic device of claim 9, wherein training the reinforcement learning model comprises: The reinforcement learning model is trained so that the performance evaluation in the trained reinforcement learning model is better than the performance evaluation of a predetermined candidate data cache solution for the simulated data read request.

15. The electronic device of claim 9, wherein the performance evaluation is indicated by a performance evaluation score, the performance evaluation score comprising a weighted sum of sub-performance evaluation scores of a plurality of sub-performance evaluations, and training the reinforcement learning model comprises: The reinforcement learning model is trained so that the performance evaluation score in the trained reinforcement learning model is higher than a predetermined performance evaluation score.

16. A computer program product tangibly stored on a non-transitory computer readable medium and comprising machine executable instructions, wherein the machine executable instructions, when executed, cause a machine to perform the steps of a method, the method comprising: acquiring storage information associated with a simulated network environment, wherein the simulated network environment includes a plurality of nodes each having a cache memory, and the storage information indicates usage of the cache memory of each node; as well as Using simulated data, training a reinforcement learning model based on simulated data read requests for nodes among the plurality of nodes, wherein the reinforcement learning model includes an environment corresponding to the simulated network environment, the reinforcement learning model includes agents corresponding to the plurality of nodes, states corresponding to the storage information, actions corresponding to candidate data cache schemes, and rewards corresponding to performance evaluations of the candidate data cache schemes for the simulated data read requests, and the reinforcement learning model is trained so that the performance evaluations in the reinforcement learning model meet predetermined performance evaluation requirements and so that the reinforcement learning model can be used to determine a data cache scheme for input data; Wherein acquiring the storage information includes acquiring the storage information and topology information of the plurality of nodes; wherein the reinforcement learning model includes the state corresponding to the storage information and the topology information; wherein the performance evaluation is indicated by a performance evaluation score, the performance evaluation score comprising a weighted sum of sub-performance evaluation scores of a plurality of sub-performance evaluations; and Wherein training the reinforcement learning model includes using the trained reinforcement learning model to determine a data cache scheme based on candidate data cache schemes corresponding to higher performance evaluation scores in the trained reinforcement learning model relative to other performance evaluation scores.

17. The computer program product of claim 16, wherein the data caching scheme indicates at least one of: A cache memory in which said simulation data is to be cached; and The cached data is to be cleared from the cache memory, wherein the remaining available space of the cache memory in which the simulation data is to be cached is insufficient.

18. The computer program product of claim 16, wherein training the reinforcement learning model comprises: Get an initial data cache plan; as well as The reinforcement learning model is trained based on the initial data caching scheme.

19. The computer program product of claim 16, wherein training the reinforcement learning model comprises: Get restrictions on data caching schemes; as well as The reinforcement learning model is trained based on the constraints.

20. The computer program product of claim 16, wherein training the reinforcement learning model comprises: The reinforcement learning model is trained so that the performance evaluation in the trained reinforcement learning model is better than the performance evaluation of a predetermined candidate data cache solution for the simulated data read request.

Images