Heterogeneous cellular network content caching method and system

Abstract

The invention discloses a heterogeneous cellular network content caching method and system in the technical field of mobile network content caching and intelligent optimization, and aims to solve the optimization problem of content caching and transmission in a mobile network. The method comprises the following steps: acquiring a content request probability of a current access user, a content cache state of a base station and a content cache state of a local cache based on a built user request model; acquiring a current system state according to the content request probability of the current access user, a content cache state of a base station and a content cache state of a local cache; and according to the current system state, generating a cache decision based on an online decision model and executing a cache action. According to the method, the dynamic change of the content cache optimization target can be quickly adapted, and meanwhile, good performance is kept.

Description

Technical Field

[0001] The present invention relates to the technical field of mobile network content caching and intelligent optimization, and in particular to a heterogeneous cellular network content caching method and system. Background Art

[0002] With the popularity of smartphones and mobile devices, the demand for mobile data has exploded. The increasing demand for high-speed data services, high-definition video streaming, online games and other data-intensive applications has led to an increasing network load, and new technologies need to be developed to effectively alleviate the network load. The increase in network load is largely due to the repeated transmission of large-size content. How to adopt some reasonable caching strategies to reduce the transmission of repeated content is a problem worth exploring.

[0003] According to statistics, the hot content that users frequently request within a period of time is limited. Therefore, caching popular video content in base stations or mobile terminals can effectively alleviate the network load problem caused by repeated content transmission and reduce content transmission delay. Considering that in real life, users' demand for hot content, the popularity of content, and service performance requirements in different time periods are dynamically changing, this requires the network's content caching strategy to be able to perceive service performance and be predictive.

[0004] Existing methods mainly focus on caching strategies with a single optimization goal, but ignore the comprehensive impact of multiple cache performance factors, such as latency, cache hit rate, redundancy rate, and other service indicators. A good caching strategy needs to be able to perceive the performance requirements of various network services and update them in a timely manner. Summary of the invention

[0005] The purpose of the present invention is to overcome the deficiencies in the prior art and provide a heterogeneous cellular network content caching method and system, which can achieve rapid learning capabilities for different optimization target tasks through learning only with small samples, quickly adapt to dynamic changes in cache performance indicators, and achieve rapid convergence of cache decision problems.

[0006] In order to solve the above technical problems, the present invention is implemented by adopting the following technical solutions:

[0007] In a first aspect, the present invention provides a heterogeneous cellular network content caching method, comprising:

[0008] Based on the established user request model, the content request probability of the current access user, the content cache status of the base station and the content cache status of the local cache are obtained;

[0009] Acquire the current system state according to the content request probability of the current access user, the content cache state of the base station and the content cache state of the local cache;

[0010] According to the current system state, a cache decision is generated based on an online decision model and a cache action is executed.

[0011] Optionally, based on the established user request model, the content request probability of the current access user, the content cache status of the base station and the content cache status of the local cache are obtained, including:

[0012] According to formula (1), calculate the user Requesting content over a cellular network Probability:

[0013] (1)

[0014] in, Indicates user Requesting content over a cellular network The probability of Represents the user, Indicates the content, Indicates the base station serial number; Indicates local cache Content The cache status; Indicates base station Users in the covered cells Request content The probability of Obey Zipf distribution; Indicates user Local cache Cache received content probability;

[0015] According to formula (2), the user Get content locally Probability:

[0016] (2)

[0017] in, Indicates user Get content locally probability;

[0018] Collect local buffer Content The cache status ,like , local cache Cached content ;like , local cache Content not cached ;

[0019] Collection base station Content The cache status is ,like , base station Cached content ;like , base station Content not cached .

[0020] Optionally, the construction of the online decision model includes:

[0021] Construct a state space based on the user's content request probability, the base station's content cache status, and the local cache's content cache status The state space The expression is as follows:

[0022] (3)

[0023] in, Indicates the system status. Indicates time The system status, Indicates any moment; Indicates time The probability of user content request, Represents the user, Indicates the content, Indicates the base station number. Represents a collection of user requested content. Indicates base station All users covered, Indicates the total number of base stations, Indicates base station Users in the covered cells Request content The probability of Indicates local cache and base station At the moment The cache status, Indicates local cache Content The cache status, Indicates base station Content The cache status;

[0024] Constructing action space based on the content cache status of the base station ; The action space The expression is as follows:

[0025] (4)

[0026] in, Indicates cache action. Indicates time Cache actions;

[0027] Construct a reward function based on the established content cache joint optimization problem ; The reward function The expression is as follows:

[0028] (5)

[0029] in, Indicates the system status Execute cache action The reward value obtained; represents the performance correlation coefficient; is the cache hit rate, which represents the ratio of the probability that the base station directly responds to the content request to the total probability of the content request; is the cache redundancy rate, which indicates the ratio of the same content cached in the base station and the local cache; Delay efficiency is the ratio of the content delivery delay reduced by the caching strategy to the content delivery delay without caching.

[0030] Optionally, the content cache joint optimization problem is expressed as follows:

[0031] (6)

[0032] in, represents the content caching joint optimization problem, Display content The amount of data, Indicates base station storage capacity, , Indicates constraints, constraints Indicates base station The total size of cached content cannot exceed the storage capacity .

[0033] Optionally, the online decision model adopts an online decision algorithm based on meta-reinforcement learning to solve the content cache joint optimization problem in content cache decision, including:

[0034] Use the intelligent agent Actor-Critic structure to build the basic structure of the online decision-making algorithm, and initialize the Actor network, Critic network, and Actor network parameters , Critic network parameters and Experience Replay Buffer ;

[0035] According to the joint optimization problem, the performance correlation coefficient is initialized , and construct the coefficient set ,in, Indicates the number of tasks;

[0036] According to the coefficient set , construct the target learning task set ,in, Indicates tasks, Indicates The coefficient set, Indicates the task number. ;

[0037] For the target learning task set Sampling is performed and the sample task set is obtained as ;

[0038] For the sample task set Any task in , based on the current system status , perform cache actions , get rewards and the next system state ;

[0039] Constructing task data and stored in the experience replay buffer , forming a task Support set ;

[0040] According to the sample task set Any task in ,right Constructed support set Sampling and calculation tasks Loss ;

[0041] According to the task Loss Update the parameters of the Actor network and the Critic network respectively to obtain the updated Actor network parameters And the updated Critic network parameters ;

[0042] Calculate tasks based on the updated Actor network and Critic network Updated loss ;

[0043] According to the task Updated loss Calculate the global meta-loss ;

[0044] According to the global meta-loss Gradient descent is performed on the meta-loss function, and the global Actor network parameters and Critic network parameters are updated to solve the joint optimization problem.

[0045] Optionally, the task Loss Calculated by the following formula:

[0046] (7)

[0047] in, represents the expected function; is a superposition factor, which indicates the impact of the current system return value on the future value; Indicates that using the Critic network, Execute cache actions in system status Rewards received; Indicates that using the Critic network, Execute cache actions in system status Rewards received;

[0048] The updated Actor network parameters Calculated by the following formula:

[0049] (8)

[0050] in, Represents the learning rate of the Actor network;

[0051] The updated Critic network parameters Calculated by the following formula:

[0052] (9)

[0053] in, Represents the learning rate of the Critic network;

[0054] The tasks are calculated based on the updated Actor network and Critic network Updated loss This is achieved through the following formula:

[0055] (10)

[0056] in, Indicates that using the updated Critic network, Execute cache actions in system state Rewards received; Indicates that using the updated Critic network, Execute cache actions in system state Rewards received;

[0057] The global meta-loss Calculated by the following formula:

[0058] (11);

[0059] The global Actor network parameters are calculated using the following formula:

[0060] (12)

[0061] in, represents the meta-learning rate;

[0062] The global critic network parameters are calculated by the following formula:

[0063] (13).

[0064] Optionally, each time the current system state is acquired according to the content request probability of the current access user, the content cache state of the base station and the content cache state of the local cache, the following steps are performed:

[0065] Get the current time Performance correlation coefficient ;

[0066] when When building the task (Task is the target task), and record it in the target learning task set , execute the online decision algorithm on the target learning task set Carry out sampling to the last step;

[0067] according to System status at the moment , generates cache decisions and executes cache actions based on the latest Actor network ;

[0068] Obtain system rewards based on the latest Critic network , get System status at the moment ;

[0069] Constructing task data , and store it in the experience replay buffer middle;

[0070] From the Experience Replay Buffer Sample the task data sample , and calculate the loss function ;

[0071] According to the loss function Update the parameters of the Actor network and the Critic network;

[0072] Among them, the loss function The expression is as follows:

[0073] (14)

[0074] in, represents the expected function, is a superposition factor, which indicates the impact of the current system return value on the future value. Indicates that using the Critic network, Execute cache actions in system status The rewards you get, Indicates that using the Critic network, Execute cache actions in system status Rewards received;

[0075] According to the loss function The parameter update of the Actor network is achieved through the following formula:

[0076] (15)

[0077] in, Represents the learning rate of the Actor network;

[0078] According to the loss function The parameter update of the Critic network is achieved through the following formula:

[0079] (16)

[0080] in, Represents the learning rate of the Critic network.

[0081] Optionally, the cache hit rate The expression is as follows:

[0082] (17)

[0083] in, Indicates user Requesting content over a cellular network The probability of Indicates user The total probability of making a content request;

[0084] The cache redundancy rate The expression is as follows:

[0085] (18);

[0086] The delay efficiency The expression is as follows:

[0087] (19)

[0088] in, Indicates user Get content locally The probability of Indicates base station To User The wireless transmission delay between Indicates base station Backhaul link latency to the core network.

[0089] Optionally, the base station To User Wireless transmission delay between The expression is as follows:

[0090] (20)

[0091] in, Indicates base station With local cache Downlink transmission rate; Display content The amount of data;

[0092] The base station Backhaul link latency to the core network The expression is as follows:

[0093] (twenty one)

[0094] in, represents the long-term average return transmission rate, represents the delay error compensation coefficient;

[0095] The base station With local cache Downlink transmission rate The expression is as follows:

[0096] (twenty two)

[0097] in, Indicates base station For users The allocated bandwidth, Indicates base station and local cache The signal-to-noise ratio between .

[0098] In a second aspect, the present invention provides a computer system, comprising:

[0099] Memory, for storing computer instructions;

[0100] A processor is used to execute the computer instructions to implement the steps of the heterogeneous cellular network content caching method described in any one of the first aspects.

[0101] Compared with the prior art, the present invention has the following beneficial effects:

[0102] 1. The heterogeneous cellular network content caching method provided by the present invention establishes a joint optimization problem for the three service performance indicators of the network caching strategy, namely, the transmission delay, cache hit rate, and cache redundancy rate, and improves the service performance indicators by optimizing the content caching decision. By using an online decision algorithm that integrates MAML (Model Agnostic Meta Learning) meta-learning and the DDPG algorithm, it is possible to achieve rapid learning capabilities for different optimization target tasks through learning with only a small sample; it can quickly adapt to the dynamic changes of the cache performance indicators, so that the cache decision problem can be quickly converged, which is convenient for engineering application, and has practical significance and good application prospects.

[0103] 2. The computer system provided by the present invention can execute the steps of the heterogeneous cellular network content caching method provided by the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0104] Figure 1 A flow chart of a heterogeneous cellular network content caching method provided according to an embodiment of the present invention;

[0105] Figure 2 A comparison diagram of the convergence behaviors of the DDPG, DQN and MAML-DDPG algorithms provided according to an embodiment of the present invention;

[0106] Figure 3This is a graph showing the relationship between the normalized performance of the system and the buffer capacity of the base station according to an embodiment of the present invention. DETAILED DESCRIPTION

[0107] The technical solution of the present invention is described in detail below through the accompanying drawings and specific embodiments. It should be understood that the embodiments of the present application and the specific features in the embodiments are detailed descriptions of the technical solution of the present application, rather than limitations on the technical solution of the present application. In the absence of conflict, the embodiments of the present application and the technical features in the embodiments can be combined with each other.

[0108] It should be noted that the term "and / or" in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship.

[0109] Embodiment 1:

[0110] The embodiment of the present invention discloses a heterogeneous cellular network content caching method, referring to Figure 1 As shown, the specific steps include:

[0111] S1, based on the established user request model, obtain the content request probability of the current access user, the content cache status of the base station and the content cache status of the local cache;

[0112] S2, acquiring the current system state according to the content request probability of the current access user, the content cache state of the base station and the content cache state of the local cache;

[0113] S3: Generate a cache decision based on the online decision model according to the current system state and execute a cache action.

[0114] Specifically,

[0115] This embodiment sets the total probability of each user making a content request is fixed, and the probability of requesting each content is the same. At the same time, each local cache has a probability Cache received content , and the next time you request the content, you can get it directly from the local cache.

[0116] In step S1, the construction of the user request model includes:

[0117] Define Base Station Users in the covered cells Request content The probability of ,and Obey Zipf distribution;

[0118] Defining Users With fixed probability A content request is generated, and the user's local cache By probability Cache received content ;

[0119] Defining a local cache Content The cache status is , where the local cache Cached content ,but ,otherwise ;

[0120] Define Base Station Content The cache status is , where the base station Cached content ,but ,otherwise ;

[0121] According to the above definition, we get the user Requesting content over a cellular network The probability calculation formula is as follows:

[0122] (1)

[0123] in, Indicates user Requesting content over a cellular network The probability of Represents the user, Indicates the content, Indicates the base station serial number;

[0124] According to formula (1), we can get the user Get content locally The probability calculation formula is as follows:

[0125] (2)

[0126] in, Indicates user Get content locally probability.

[0127] In step S3, the construction of the online decision model includes:

[0128] Construct a state space based on the user's content request probability, the base station's content cache status, and the local cache's content cache status The state space The expression is as follows:

[0129] (3)

[0130] in, Indicates the system status. Indicates time The system status, Indicates any moment; Indicates time The probability of user content request, Represents the user, Indicates the content, Indicates the base station number. Represents a collection of user requested content. Indicates base station All users covered, Indicates the total number of base stations, Indicates base station Users in the covered cells Request content The probability of Indicates local cache and base station At the moment The cache status, Indicates local cache Content The cache status, Indicates base station Content The cache status;

[0131] Constructing action space based on the content cache status of the base station Each base station needs to perform content caching and updating actions to meet the service quality. The action space should dynamically contain the possible caching decisions of the base station. Therefore, the action space The expression is as follows:

[0132] (4)

[0133] in, Indicates cache action. Indicates time Cache actions;

[0134] Construct a reward function based on the established content cache joint optimization problem , so that the reward function guides the base station BS to make decisions that can improve user experience; the reward function The expression is as follows:

[0135] (5)

[0136] in, Indicates the system status Execute cache action The reward value obtained; represents the performance correlation coefficient; is the cache hit rate, which represents the ratio of the probability that the base station directly responds to the content request to the total probability of the content request; is the cache redundancy rate, which indicates the ratio of the same content cached in the base station and the local cache; Delay efficiency is the ratio of the content delivery delay reduced by the caching strategy to the content delivery delay without caching.

[0137] In this embodiment, the content cache joint optimization problem is established using a dynamic weight parameter model, and the content cache joint optimization problem is expressed as follows:

[0138] (6)

[0139] in, represents the content caching joint optimization problem, Display content The amount of data, Indicates base station storage capacity, , Indicates constraints, constraints Indicates base station The total size of cached content cannot exceed the storage capacity .

[0140] The objective function of P1 depends on the performance correlation coefficient settings, these coefficients usually change within a day; the change in parameter values ​​reflects the current optimization focus of the system; the above optimization problem P1 can be transformed into a Markov decision problem based on state-action value. Due to the diverse optimization objectives caused by the dynamic parameter changes in this problem, the traditional reinforcement learning RL-based method has low training efficiency and lacks a clear training direction; therefore, the embodiment designs an online decision-making method based on meta-reinforcement learning to quickly implement the optimization problem P1.

[0141] In this embodiment, the content cache joint optimization problem is established according to a multi-dimensional performance model; the performance model models three indicators: cache hit rate, cache redundancy rate and transmission delay, and the three indicators can be used as performance indicators for subsequent cache strategy optimization.

[0142] When the user cannot retrieve the target content on the local terminal device, the content request will be forwarded to a nearby base station. Whether the base station can respond to the request depends on whether the content is cached in the base station. Therefore, this embodiment defines the cache hit rate is the ratio of the probability that the base station directly responds to the content request to the total probability of the content request; the cache hit rate The expression is as follows:

[0143] (17)

[0144] in, Indicates user Requesting content over a cellular network The probability of Indicates user The total probability of making a content request.

[0145] This embodiment defines the cache redundancy rate is the ratio of the same content cached in the base station and the local cache; If the content of the request Local Neutralization are cached in , they are considered redundant; therefore, given The cache status , get the cache redundancy rate The expression is as follows:

[0146] (18).

[0147] Define delay efficiency based on the established communication model and content transmission delay model is the ratio of the content delivery delay reduced by the caching strategy to the content delivery delay without caching. The reduced transmission delay depends on the cache status of the base station and the local cache. Therefore, the delay efficiency The expression is as follows:

[0148] (19)

[0149] The denominator in the formula represents the total expected content delivery delay, and the numerator represents the delay saved by the current caching strategy; Indicates user Get content locally The probability of Indicates base station To User The wireless transmission delay between Indicates base station Backhaul link latency to the core network.

[0150] The latency experienced by users accessing content on a mobile network includes the latency of base stations. To User Wireless transmission delay between and base station Backhaul link latency to the core network In this embodiment, these two parts of delay are counted for subsequent construction of a multi-dimensional performance model; the content transmission delay model includes base station To User Wireless transmission delay between and base station Backhaul link latency to the core network ;

[0151] The base station To User Wireless transmission delay between Depends on the cellular transmission rate, and its expression is as follows:

[0152] (20)

[0153] in, Display content The amount of data, Indicates base station With local cache The downlink transmission rate.

[0154] The base station Backhaul link latency to the core network Depending on the average transmission rate of the link, it can be set to follow a general model, which is expressed as follows:

[0155] (twenty one)

[0156] in, represents the long-term average return transmission rate, Indicates the delay error compensation coefficient.

[0157] The communication model includes a base station With local cache Downlink transmission rate ; The downlink transmission rate The expression is as follows:

[0158] (twenty two)

[0159] in, Indicates base station For users The allocated bandwidth, express and The signal-to-noise ratio between the two is set to change dynamically according to rules to simulate the changes in wireless channel conditions throughout the day.

[0160] In view of the above optimization problem P1, this embodiment designs an online decision-making algorithm based on meta-reinforcement learning to implement multi-task model training, including:

[0161] Build the target network: Use the intelligent agent Actor-Critic structure to build the basic structure of the online decision-making algorithm, and initialize the Actor network, Critic network, and Actor network parameters , Critic network parameters and Experience Replay Buffer ;

[0162] Initialize the target learning task: Initialize the performance correlation coefficient according to the joint optimization problem , and construct the coefficient set ,in, represents the number of tasks; according to the coefficient set , construct the target learning task set ,in, Indicates tasks, Indicates coefficient sets, Indicates the task number. ;

[0163] Task sampling: learning task set for the target Sampling is performed and the sample task set is obtained as ; For the sample task set Any task in , based on the current system status , perform cache actions , get rewards , and get the next system status ;Construct task data and stored in the experience replay buffer , forming a task Support set ;

[0164] Actor network and Critic network parameter update: based on the sample task set Any task in ,right Constructed support set Sampling and calculation tasks Loss According to the task Loss Update the parameters of the Actor network and the Critic network respectively to obtain the updated Actor network parameters And the updated Critic network parameters ;

[0165] Global meta-loss calculation: Calculate the task based on the updated Actor network and the updated Critic network Updated loss ; Updated according to the task Loss Calculate the global meta-loss ;

[0166] Meta parameter update: according to the global meta loss Perform gradient descent on the meta-loss function and update the global Actor network parameters and Critic network parameters.

[0167] Specifically, the task Loss Calculated by the following formula:

[0168] (7)

[0169] in, represents the expected function; is a superposition factor, which indicates the impact of the current system return value on the future value; express The cached actions selected by the Actor network in the system state; Indicates that using the Critic network, Execute cache actions in system state Rewards received; Indicates that using the Critic network, Execute cache actions in system state The reward you get.

[0170] The updated Actor network parameters Calculated by the following formula:

[0171] (8)

[0172] in, Represents the learning rate of the Actor network.

[0173] The updated Critic network parameters Calculated by the following formula:

[0174] (9)

[0175] in, Represents the learning rate of the Critic network.

[0176] The tasks are calculated based on the updated Actor network and Critic network Updated loss This is achieved through the following formula:

[0177] (10)

[0178] in, express The cached actions selected by the Actor network after the update in the system state; Indicates that using the updated Critic network, Execute cache actions in system state Rewards received; Indicates that using the updated Critic network, Execute cache actions in system state The reward you get.

[0179] The global meta-loss Calculated by the following formula:

[0180] (11).

[0181] The global Actor network parameters are calculated using the following formula:

[0182] (12)

[0183] in, represents the meta-learning rate.

[0184] The global critic network parameters are calculated by the following formula:

[0185] (13)

[0186] in, represents the meta-learning rate.

[0187] In this embodiment,

[0188] Each time the system status is updated, the following steps are performed:

[0189] Get the current time Performance correlation coefficient ;

[0190] when When building the task (Task is the target task), and record it in the target learning task set , execute the online decision algorithm on the target learning task set Carry out sampling to the last step;

[0191] according to System status at the moment , generates cache decisions and executes cache actions based on the latest Actor network ;

[0192] Obtain system rewards based on the latest Critic network , get System status at the moment ;

[0193] Constructing task data , and store it in the experience replay buffer middle;

[0194] From the Experience Replay Buffer Sample the task data sample , and calculate the loss function ;

[0195] According to the loss function Update the parameters of the Actor network and the Critic network.

[0196] The loss function The expression is as follows:

[0197] (14)

[0198] in, represents the expected function, is a superposition factor, which indicates the impact of the current system return value on the future value. express The cached actions selected by the Actor network in the system state, Indicates that using the Critic network, Execute cache actions in system state The rewards you get, Indicates that using the Critic network, Execute cache actions in system state The reward you get.

[0199] According to the loss function The parameter update of the Actor network is achieved through the following formula:

[0200] (15)

[0201] in, Represents the learning rate of the Actor network;

[0202] According to the loss function The parameter update of the Critic network is achieved through the following formula:

[0203] (16)

[0204] in, Represents the learning rate of the Critic network.

[0205] In order to verify the effectiveness of the heterogeneous cellular network content caching method proposed in this embodiment, this embodiment conducts a simulation experiment to evaluate the performance of the proposed solution. The experiment is as follows:

[0206] Four different simulation models are set up to study the impact of different optimization strategies on system performance: a) standard DDPG algorithm, b) standard DDPG algorithm (Deep Deterministic Policy Gradient), and the target parameter is switched when the number of iteration cycles is 100, c) MAML-DDPG algorithm (MAML: Model Agnostic MetaLearning, MAML-DDPG is the online decision-making algorithm based on meta-reinforcement learning proposed in this embodiment), and the target parameter is switched when the number of iteration cycles is 100, d) MAML-DDPG algorithm (i.e. the online decision-making algorithm based on meta-reinforcement learning proposed in this embodiment), and the target parameter is switched when the number of iteration cycles is 150.

[0207] The initial target parameters of all simulation models are set to 0.33; the simulation environment consists of 10 base stations BS and 100 local caches UD, and the video content set consists of 500 files, each of which is 20MB in size; the cache capacity of the base station BS and the local cache UD are configured to be 1000MB and 250-300MB respectively.

[0208] refer to Figure 2The convergence performance of the four models is demonstrated; the results show that the convergence value of the DDPG model drops sharply with the change of the task (at 100 iterations) and finally stabilizes at a low level; in contrast, although the MAML-DDPG model experiences a significant drop when the optimization target task changes (at 100 and 150 iterations), it quickly recovers and converges to a better strategy due to its meta-learning ability; the experimental results demonstrate the task perception and rapid adaptability of the online decision-making algorithm proposed in this embodiment.

[0209] refer to Figure 3 The figure shows the variation of the system normalized performance with the increase of the base station cache capacity under different parameters of the online decision algorithm proposed in this embodiment; as shown in the figure, the parameters are set to (cache hit rate is the main optimization goal) and (delay efficiency is the main optimization goal), the system performance of the method of the present invention and the DDPG method gradually increases with the cache capacity. (cache redundancy rate is the main goal), the cache redundancy rates of the online decision algorithm and DDPG algorithm proposed in this embodiment decrease with the total price of cache capacity; at the same time, the online decision algorithm proposed in this embodiment and the traditional DDPG both showed performance advantages during the test, verifying the performance of the online decision algorithm proposed in this embodiment in terms of decision accuracy.

[0210] Through the above simulation comparison, it can be seen that the heterogeneous cellular network content caching method proposed in this embodiment has obvious advantages.

[0211] In summary, the heterogeneous cellular network content caching method proposed in this embodiment is based on meta-reinforcement learning, and establishes a joint optimization problem for the three service performance indicators of the network caching strategy, namely, the transmission delay, cache hit rate, and cache redundancy rate, in order to improve the service performance indicators by optimizing the content caching decision. The method adopts a reinforcement learning algorithm that integrates MAML meta-learning and DDPG algorithm, and can achieve rapid learning ability for different optimization target tasks only through learning of small samples, and quickly adapt to the dynamic changes of cache performance indicators, namely, transmission delay, cache hit rate, and cache redundancy rate, so as to achieve rapid convergence of cache decision problems.

[0212] Embodiment 2:

[0213] This embodiment provides a computer system, including:

[0214] Memory, for storing computer instructions;

[0215] A processor is used to execute the computer instructions to implement the steps of the heterogeneous cellular network content caching method described in any one of the first aspects.

[0216] It will be appreciated by those skilled in the art that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0217] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0218] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0219] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0220] The embodiments of the present invention are described above in conjunction with the accompanying drawings, but the present invention is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the enlightenment of the present invention, ordinary technicians in this field can also make many forms without departing from the scope of protection of the purpose of the present invention and the claims, which all fall within the protection of the present invention.

Claims

1. A heterogeneous cellular network content caching method, characterized in that: include: Based on the established user request model, the content request probability of the current access user, the content cache status of the base station and the content cache status of the local cache are obtained; Acquire the current system state according to the content request probability of the current access user, the content cache state of the base station and the content cache state of the local cache; According to the current system state, a cache decision is generated based on an online decision model and a cache action is executed.

2. The heterogeneous cellular network content caching method according to claim 1, characterized in that: Based on the established user request model, the content request probability of the current access user, the content cache status of the base station and the content cache status of the local cache are obtained, including: According to formula (1), calculate the user Requesting content over a cellular network Probability: （1） in, Indicates user Requesting content over a cellular network The probability of Represents the user, Indicates the content, Indicates the base station serial number; Indicates local cache Content The cache status; Indicates base station Users in the covered cells Request content The probability of Obey Zipf distribution; Indicates user Local cache Cache received content probability; According to formula (2), the user Get content locally Probability: （2） in, Indicates user Get content locally probability; Collect local buffer Content The cache status ,like , local cache Cached content ;like , local cache Content not cached ; Collection base station Content The cache status is ,like , base station Cached content ;like , base station Content not cached .

3. The heterogeneous cellular network content caching method according to claim 1, characterized in that: The construction of the online decision model includes: Construct a state space based on the user's content request probability, the base station's content cache status, and the local cache's content cache status The state space The expression is as follows: （3） in, Indicates the system status. Indicates time The system status, Indicates any moment; Indicates time The probability of user content request, Represents the user, Indicates the content, Indicates the base station number. Represents a collection of user requested content. Indicates base station All users covered, Indicates the total number of base stations, Indicates base station Users in the covered cells Request content probability; Indicates local cache and base station At the moment The cache status, Indicates local cache Content The cache status, Indicates base station Content The cache status; Constructing action space based on the content cache status of the base station ; The action space The expression is as follows: （4） in, Indicates cache action. Indicates time Cache actions; Construct a reward function based on the established content cache joint optimization problem ; The reward function The expression is as follows: （5） in, Indicates the system status Execute cache action The reward value obtained; represents the performance correlation coefficient; is the cache hit rate, which represents the ratio of the probability that the base station directly responds to the content request to the total probability of the content request; is the cache redundancy ratio, which indicates the ratio of the same content cached in the base station and the local cache; Delay efficiency is the ratio of the content delivery delay reduced by the caching strategy to the content delivery delay without caching.

4. The heterogeneous cellular network content caching method according to claim 3, characterized in that: The content cache joint optimization problem is expressed as follows: （6） in, represents the content caching joint optimization problem, Display content The amount of data, Indicates base station storage capacity, , Indicates constraints, constraints Indicates base station The total size of cached content cannot exceed the storage capacity .

5. The heterogeneous cellular network content caching method according to claim 4, characterized in that: The online decision model adopts an online decision algorithm based on meta-reinforcement learning to solve the content cache joint optimization problem in content cache decision, including: Use the intelligent agent Actor-Critic structure to build the basic structure of the online decision-making algorithm, and initialize the Actor network, Critic network, and Actor network parameters , Critic network parameters and Experience Replay Buffer ; According to the joint optimization problem, the performance correlation coefficient is initialized , and construct the coefficient set ,in, Indicates the number of tasks; According to the coefficient set , construct the target learning task set ,in, Indicates tasks, Indicates The coefficient set, Indicates the task number. ; For the target learning task set Sampling is performed to obtain the sample task set: ; For the sample task set Any task in , based on the current system status , perform cache actions , get rewards and the next system state ; Constructing task data and stored in the experience replay buffer , forming a task Support set ; According to the sample task set Any task in ,right Constructed support set Sampling and calculation tasks Loss ; According to the task Loss Update the parameters of the Actor network and the Critic network respectively to obtain the updated Actor network parameters And the updated Critic network parameters ; Calculate tasks based on the updated Actor network and Critic network Updated loss ; According to the task Updated loss Calculate the global meta-loss ; According to the global meta-loss Gradient descent is performed on the meta-loss function, and the global Actor network parameters and Critic network parameters are updated to solve the joint optimization problem.

6. The heterogeneous cellular network content caching method according to claim 5, characterized in that: The task Loss Calculated by the following formula: （7） in, represents the expected function; is a superposition factor, which indicates the impact of the current system return value on the future value; Indicates that using the Critic network, Execute cache actions in system status Rewards received; Indicates that using the Critic network, Execute cache actions in system status Rewards received; The updated Actor network parameters Calculated by the following formula: （8） in, Represents the learning rate of the Actor network; The updated Critic network parameters Calculated by the following formula: （9） in, Represents the learning rate of the Critic network; The tasks are calculated based on the updated Actor network and Critic network Updated loss This is achieved through the following formula: （10） in, Indicates that using the updated Critic network, Execute cache actions in system status Rewards received; Indicates that using the updated Critic network, Execute cache actions in system status Rewards received; The global meta-loss Calculated by the following formula: （11）； The global Actor network parameters are calculated using the following formula: （12） in, represents the meta-learning rate; The global critic network parameters are calculated by the following formula: （13）。 7. The heterogeneous cellular network content caching method according to claim 3, characterized in that: The cache hit rate The expression is as follows: （17） in, Indicates user Requesting content over a cellular network The probability of Indicates user The total probability of making a content request; The cache redundancy rate The expression is as follows: （18）； The delay efficiency The expression is as follows: （19） in, Indicates user Get content locally The probability of Indicates base station To User The wireless transmission delay between Indicates base station Backhaul link latency to the core network.

8. The heterogeneous cellular network content caching method according to claim 7, characterized in that: The base station To User Wireless transmission delay between The expression is as follows: （20） in, Indicates base station With local cache Downlink transmission rate; Display content The amount of data; The base station Backhaul link latency to the core network The expression is as follows: （21） in, represents the long-term average return transmission rate, represents the delay error compensation coefficient; The base station With local cache Downlink transmission rate The expression is as follows: （22） in, Indicates base station For users The allocated bandwidth, Indicates base station and local cache The signal-to-noise ratio between .

9. A computer system, characterized in that: include: Memory, for storing computer instructions; A processor, configured to execute the computer instructions to implement the steps of the heterogeneous cellular network content caching method according to any one of claims 1 to 8.

Images