Edge hot object cache access method

Abstract

The invention provides an edge hot object cache admission method, and relates to the technical field of information, the edge hot object cache admission method constructs a time sequence model, the time sequence model comprises feature mapping, multi-object time coding, LSTM and an attention network, the time sequence model is subjected to supervised training to obtain a trained time sequence model, according to the method, the position coding of the request sequence is realized through a multi-object time coding method, so that the algorithm can learn the two-stage sequence characteristics of the access sequence, and the learning ability of the model is remarkably improved. According to the method, a buffer area is set, a target request corresponding to admission is inserted into a hot object cache, a shadow request object is created and inserted into the hot object cache, then corresponding rewards are calculated, semi-online reinforcement learning is carried out on a trained time sequence model based on the state, admission decision and rewards, and through a model agent decision-making process, the time sequence model is obtained. The model can stably accumulate rewards to the maximum for a long time, and indexes such as the hit rate of hot object caching are remarkably improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of information technology, in particular to an edge hot object cache admission method. BACKGROUND

[0002] With the rise of computing power network concept, various computing, storage and network resources in the network are uniformly arranged and dynamically scheduled, forming a distributed computing platform that is cross-domain, heterogeneous and on-demand elastic. Under this framework, the content distribution network (CDN) cache system is not only a link of the distributed storage node, but also undertakes the dual tasks of reducing the load of the network core bandwidth and improving the response performance of edge computing. The typical CDN scheme deploys "hot object cache + disk cache" on various nodes of the computing power network, the former stores the most frequently accessed data, and the latter is used for persistent storage. When a user request arrives, the hot object cache on the edge node closest to the user responds first, and when it is not hit, it can use the local disk cache or other nodes in the computing power network to coordinate scheduling or access the backend storage. Through intelligent scheduling and admission decision, the CDN cache system optimizes the data transmission delay in the computing power network, reduces the load of the core network and computing center, and improves the overall resource utilization efficiency. However, the hot object cache has limited capacity and cannot cache all request data indefinitely. Therefore, when the hot object cache is missing, i.e. when the user initiates a request that is not hit by the hot object cache, it must be selectively decided whether to cache the current object in the hot object cache. This process is usually referred to as hot object cache admission decision.

[0003] The Chinese patent "CN202410685449.X Cache admission method and system based on machine learning in multiple variable cache space scenarios" provides a hot object cache admission method: the system first stores the user's last K (preferably 19) access records and current access information into a feature buffer window when a cache miss occurs each time; all records in the window are sequentially subjected to one-hot encoding dimension reduction, embedding processing and tanh normalization to obtain a uniform length feature vector; then the vector is input into the offline trained LSTM model to predict the "reuse distance" classification label; finally, the intermediate parameter A is calculated in combination with the current available cache capacity threshold, when A ≥ 0 (the predicted reuse distance is within the loadable range), the data is allowed to be written into the hot object cache, otherwise it is rejected, so that high-accuracy admission decisions can still be made when the cache capacity changes dynamically.

[0004] Existing hot object cache admission methods mainly include two categories: hotness information prediction and admission threshold judgment.

[0005] The technical solution described in Chinese patent "CN202410685449.X Cache Admission Method and System Based on Machine Learning in Variable Cache Space Scenarios" relies on a high-dimensional feature set, including various user access markers, object information markers, and system load markers, to ensure prediction accuracy during model training and online decision-making. If applied to a general dataset containing only basic attributes (such as id, size, and time), the lack of these high-dimensional features leads to a significant drop in model performance. Furthermore, the available feature dimensions and distributions vary across different platforms or business scenarios, making this method highly targeted to specific datasets and lacking cross-scenario and cross-platform versatility, thus limiting its applicability.

[0006] Threshold-based admission methods require determining optimized thresholds for object metrics (such as object size, object frequency, and access time). An object is only admitted to the cache if its metrics meet the corresponding threshold. Representative methods include AdaptSize, HillClimbing, Percentile, and Darwin, which improve the intelligence of caching strategies to some extent. However, these strategies still rely on manually defined feature metrics, cannot learn complex dependencies in request patterns, and cannot support information-based access decisions. Summary of the Invention

[0007] To address the shortcomings of existing technologies, the present invention aims to propose an edge hot object caching admission method, comprising:

[0008] Step 1: Obtain the training request and the corresponding historical request. Based on the first information of the training request and the second information of the historical request, train the time series model to obtain the trained time series model.

[0009] The first information includes the unique identifier of the request object corresponding to the training request, the size of the request object corresponding to the training request, and the initiation time t of the training request. The second information includes the unique identifier of the request object corresponding to the historical request, the size of the request object corresponding to the historical request, and the initiation time of the historical request. The initiation time of all historical requests is represented as {tm,…,t-1}, where m represents the number of historical requests.

[0010] The temporal model includes feature mapping, multi-object temporal coding, LSTM, and attention network;

[0011] Step 2: Construct the state space, and pass the states in the state space through the trained temporal model, fully connected layer and Sigmoid transformation in sequence to obtain the admission decision, and then determine whether the target request of the user in the state is allowed to be admitted to the hot object cache of the content delivery network.

[0012] The state in the state space includes information Str requested by a user, request object information Sth of a historical request corresponding to the user request, and a time sequence Ste of initiation of the historical request;

[0013] Step 3: setting a reward, updating the reward based on the state in the state space, and continuously optimizing and updating parameters of the trained time sequence model, parameters of the full connection layer, and parameters of the Sigmoid transformation based on the state, the admission decision, and the reward;

[0014] Step 4: calculating the admission decision of the next state based on the updated time sequence model, the full connection layer, and the Sigmoid transformation, and then determining whether a target request in the state is allowed to access the hot object cache of the content distribution network, and then returning to step 3.

[0015] Optionally, step 1 specifically includes:

[0016] Step 1.1: obtaining first information of a training request, a plurality of historical requests corresponding to the training request, and second information of each historical request;

[0017] Step 1.2: offline labeling the second information by using the Belady algorithm to obtain a reuse distance label, the reuse distance label including a request distance label or a stack distance label, the request distance label representing a number of request objects between two requests responding to a same request object, and the stack distance label representing a size distance of the request objects between the two requests responding to the same request object, all the second information corresponding to the training request and the reuse distance label forming a training sample, and a plurality of training samples forming a training sample set;

[0018] Step 1.3: inputting all the second information corresponding to the training request in the training sample into the time sequence model to obtain context information;

[0019] Step 1.4: obtaining a predicted reuse distance by using the context information through the full connection layer and the Softmax transformation, and the predicted reuse distance is specifically represented by the following formula:

[0020] D = Softmax(dense(Gt));

[0021] Wherein, D represents the predicted reuse distance, dense represents the full connection layer, and Gt represents the context information;

[0022] Step 1.5: updating parameters in the time sequence model, parameters of the full connection layer, and parameters of the Softmax transformation based on the predicted reuse distance and the reuse distance label, and iteratively updating the parameters of the time sequence model based on the training sample set to obtain a trained time sequence model.

[0023] Optionally, step 1.3 specifically comprises:

[0024] Step 1.3.1: For each second information, the unique identifier of the request object in the second information and the size of the request object are feature mapped to obtain an identifier embedding vector and a size embedding vector;

[0025] Step 1.3.2: The identifier embedding vectors and the size embedding vectors of all second information are input into the LSTM to obtain a hidden state vector H and a hidden state H t at time t;

[0026] Specifically, the identifier embedding vector and the size embedding vector corresponding to time t-m are input into the LSTM to obtain the cell state C t-m at time t-m and the hidden state H t-m at time t-m, which are specifically represented by the following formula:

[0027] (C t-m ,H t-m )=LSTM(E(id t-m );E(size t-m ));

[0028] Wherein, E(id t-m ) represents the identifier embedding vector corresponding to time t-m, and E(size t-m ) represents the size embedding vector corresponding to time t-m;

[0029] The cell state C t-m at time t-m, the identifier embedding vector and the size embedding vector corresponding to time t-m+1 are input into the LSTM to obtain the cell state C t-m+1 at time t-m+1 and the hidden state H t-m+1 at time t-m+1, which are specifically represented by the following formula:

[0030] (C t-m+1 ,H t-m+1 )=LSTM([E(id t-m+1 );E(size t-m+1 )],C t-m );

[0031] Wherein, E(id t-m+1 ) represents the identifier embedding vector corresponding to time t-m+1, and E(size t-m+1 ) represents the size embedding vector corresponding to time t-m+1;

[0032] Similarly, the cell state from the previous time step, the corresponding identifier embedding vector and size embedding vector for the current time step are input into the LSTM until time t, at which point the hidden state H at time t is obtained from the LSTM output. t Simultaneously, the hidden states from time tm to time t-1 are obtained. These hidden states from time tm to time t-1 form the hidden state vector H, which is specifically represented by the following formula:

[0033] H =(H t-m ,…,H t-1 );

[0034] Step 1.3.3: For each time in {tm,…,t-1} Multi-object time encoding is performed to obtain the time code for each time point, which is represented by the following formula:

[0035] ;

[0036] ;

[0037] ;

[0038] Where n represents time The corresponding historical request's sequential position within all historical requests, where p represents request density and d... TE Encoding dimension for time, This indicates the scale at position 0. This indicates the scale of position i. Indicates position The scaling factor is L, which represents a predefined time scale constant.

[0039] Step 1.3.4: Based on the temporal encoding and hidden state vector H at each time step in {tm,…,t-1}, calculate the key vector K and value vector V of the attention network, specifically through the following formula:

[0040] (K,V)=H+[MO-TE(tm,m,0),..,MO-TE(t-1,m,m)];

[0041] Where MO-TE(tm,m,0) represents the time code of time tm, and MO-TE(t-1,m,m) represents the time code of time t-1;

[0042] Step 1.3.5: H of the hidden state at time t t As the query vector Q, the query vector Q, the key vector K, and the value vector V are input into the attention network to obtain the context information Gt, which is specifically represented by the following formula:

[0043] Gt= attention(K,Q,V);

[0044] wherein attention represents an attention network.

[0045] Optionally, the parameters in the time series model, the parameters of the full connection layer and the parameters of the Softmax transformation are updated in step 1.5, specifically including:

[0046] The parameters of the feature mapping, the parameters of the LSTM, the parameters of the multi-object time encoding and the parameters of the attention network are updated.

[0047] Optionally, step 2 specifically includes:

[0048] Step 2.1: constructing a state space, the state space containing a plurality of states, the state containing information Str of a user request, request object information Sth of a historical request corresponding to the user request and a time sequence of initiation of the historical request ;

[0049] wherein Str=(id,size, ), id represents a unique identifier of a request object corresponding to the user request, size represents a size of the request object corresponding to the user request, represents an initiation time of the user request;

[0050] wherein , represents a unique identifier of a request object of a historical request at time , represents a unique identifier of a request object of a historical request at time , represents a size of a request object of a historical request at time , represents a size of a request object of a historical request at time ;

[0051] Step 2.2: inputting the states in the state space into the trained time series model to obtain context information of the states;

[0052] Step 2.3: obtaining an admission decision At by subjecting the context information of the states to a full connection layer and a Sigmoid transformation, specifically through the following formula:

[0053] ;

[0054] wherein represents the context information of the states, and dense represents the full connection layer;

[0055] Step 2.4: in the case of the admission decision At = 1, the target request object representing the user request allows the hot object cache of the content distribution network to admit the content distribution network, and in the case of the admission decision At = 0, the target request object representing the user request refuses the hot object cache of the content distribution network to admit the content distribution network.

[0056] Optionally, step 3 specifically includes:

[0057] Step 3.1: in the case of the admission decision At = 1, the target request object of the user request is inserted into the hot object cache of the content distribution network, and in the case of the admission decision At = 0, the shadow request object of the user request is created, the shadow request object includes the unique identifier and size of the request object corresponding to the user request, and the shadow request object is inserted into the hot object cache of the content distribution network;

[0058] Step 3.2: set a triple for the target request object or the shadow request object, the triple is represented as (S, A, R), wherein S represents the input state, A represents the admission decision, and R represents the reward, and the triple is stored to the buffer;

[0059] Step 3.3: obtain the next state in the state space, update the reward R of the target request object or the shadow request object based on the request object of the user request in the state, in the case that the target request object or the shadow request object meets the preset eviction condition, evict the target request object or the shadow request object from the hot object cache, and move the triple of the target request object or the triple of the shadow request object from the buffer to the replay pool;

[0060] The eviction condition includes the longest request object storage time in the hot object cache or the lowest hit frequency of the request object in the hot object cache;

[0061] Step 3.4: when the number of triples in the replay pool reaches a preset threshold, according to the soft actor-critic SAC algorithm, based on the triples in the replay pool, the parameters in the trained time sequence model, the parameters of the full connection layer and the parameters of the Sigmoid transformation are updated and optimized.

[0062] Optionally, step 3.3 specifically includes:

[0063] Obtain the next state in the state space, when the request object of the user request in the state is the target request object, that is, when the target request object is hit, the value of R is increased by 1, in the case that the target request object is not hit until the preset eviction condition is met, the value of R is-1, and in the case that the target request object meets the preset eviction condition, the target request object is evicted from the hot object cache, and the triple of the target request object is moved from the buffer to the replay pool;

[0064] When the requested object of the state user request is a shadow request object, that is, when the shadow request object is accessed, the value of R is reduced by 1, in the case that the shadow request object is not accessed until the preset eviction condition is met, the value of R is set to 1, in the case that the shadow request object meets the preset eviction condition, the shadow request object is evicted from the hot object cache, and the triple of the shadow request object is moved from the buffer to the playback pool.

[0065] The beneficial effects generated by the above technical solutions are:

[0066] The timing model is constructed, the timing model includes feature mapping, multi-object time encoding, LSTM and attention network, the timing model is supervised training, and the training completed timing model is obtained, wherein the position encoding of the request sequence is realized through the multi-object time encoding method, the algorithm can learn the two-level sequence features of the access sequence, and the learning ability of the model is significantly improved. The buffer is set, and for the access condition, the target request corresponding to the access is inserted into the hot object cache, for the rejection condition, the shadow request object is created and inserted into the hot object cache, and then the corresponding rewards are calculated respectively, and based on the state, the access decision and the reward, the training completed timing model is semi-online reinforcement learning, through the intelligent agent decision process, the model can maximize the cumulative reward for a long time, and the hit rate of the hot object cache and other indicators are significantly improved. BRIEF DESCRIPTION OF DRAWINGS

[0067] Figure 1 It is a structure schematic diagram of an edge hot object cache access method in an embodiment of the application.

[0068] Figure 2 It is a flowchart of an edge hot object cache access method in an embodiment of the application.

[0069] Figure 3 It is a schematic diagram of a conventional position encoding method in an embodiment of the application.

[0070] Figure 4 It is a schematic diagram of multi-object time encoding in an embodiment of the application.

[0071] Figure 5 It is a schematic diagram of intelligent agent decision in an embodiment of the application. DETAILED DESCRIPTION

[0072] The specific embodiments of the application will be further described in detail below in combination with the drawings and examples. The following examples are used to illustrate the application, but not to limit the scope of the application.

[0073] In order to describe the application more clearly, the content distribution network application scenario is introduced first. The content distribution network is a distributed server network system deployed all over the world, which is used to accelerate the content transmission and delivery service on the Internet. When a user initiates a content request, the content distribution network will select the nearest edge server to provide content service according to the geographical location of the user, instead of loading from the source server website, thereby reducing bandwidth waste and time delay, and also increasing user experience and reducing service provider cost.

[0074] The content distribution network storage structure is generally divided into two layers of hot object cache and disk cache. The hot object cache has the advantages of fast read-write speed and no loss, but generally has smaller capacity. The disk cache has the advantage of large capacity, but has slow read-write speed, and its service life is limited by the number of read-write times. However, the content distribution network edge server has limited capacity and cannot completely store all source station contents, so it is necessary to filter some contents with high popularity in a short time for storage according to the edge user behavior. When a user initiates a request, the content distribution network will first check whether the requested object is cached in the hot object cache. If the requested object is cached in the hot object cache, it will be directly returned to the requesting user. If the requested object is not cached in the hot object cache, it will be checked in the disk cache. If the requested object is cached in the disk cache, it will be returned to the user and it will be decided whether to promote the object to the hot object cache for next request. If the requested object is not cached in the disk cache, the source service will be requested for the object and it will be decided whether to cache the object to the hot object cache. In the case of hot object cache miss, the process of deciding whether to cache the object to the hot object cache is called hot object cache admission decision.

[0075] In view of the problems existing in the prior art, the application provides an edge hot object cache admission method. In order to optimize the cache admission strategy of the content distribution network, the application proposes a double-head model architecture. The model combines multi-object time coding to extract complex time sequence features, and uses reinforcement learning with buffer mechanism to optimize the cache admission strategy. The overall architecture of the model is shown in Figure 1. The application includes a time sequence model, a fully connected layer and a Softmax transformation, a fully connected layer and a Sigmoid transformation. The application first performs supervised training on the model composed of the time sequence model, the fully connected layer and the Softmax transformation, obtains the trained time sequence model, and then performs semi-online reinforcement learning based on the trained time sequence model, the fully connected layer and the Sigmoid transformation. Specifically, the model combines the time sequence features of the training request and the historical request corresponding to the training request, and the time sequence features of the training request and the historical request corresponding to the training request are input into the time sequence model to obtain the time sequence feature vector of the training request and the historical request corresponding to the training request. Figure 2 The method can include the following steps:

[0076] Step 1: Obtain a training request and a historical request corresponding to the training request, train the time sequence model based on the first information of the training request and the second information of the historical request, and obtain a trained time sequence model;

[0077] The first information includes a unique identifier of the request object corresponding to the training request, a size of the request object corresponding to the training request, and an initiation time t of the training request, and the second information includes a unique identifier of the request object corresponding to the historical request, a size of the request object corresponding to the historical request, and an initiation time of the historical request, and the initiation times of all the historical requests are represented as {t-m,...,t-1}, wherein m represents the number of the historical requests.

[0078] It can be understood that the historical request corresponding to the training request is a plurality of requests before the time t of the training request.

[0079] The timing model includes feature mapping, multi-object time encoding, LSTM, and attention network.

[0080] The multi-object time encoding is designed to solve the problem that the existing position encoding cannot effectively express the irregular multi-object access sequence in the content distribution network environment. Next, the problems in the prior art and the multi-object time encoding provided by the present application are described in detail.

[0081] There are usually dependent features between request sequences (i.e., sequences composed of multiple requests), that is, the access pattern of some objects is affected by historical requests or the preferences of user groups. The Attention mechanism has strong historical information extraction and learning ability when processing sequence data, but the encoding mechanism of Attention mainly targets single-dimensional sequences. The content distribution network request access sequence usually exhibits irregular multi-object access, that is, multiple objects are accessed in sequence at the same time, and the number of access objects varies at different time points. This access pattern forms two layers of sequence relationships in the time dimension: one is the sequence within the time, that is, the order of multiple access objects within a time; the other is the sequence between times, that is, the access order between different time points. Existing sequence data for irregular sampling usually converts it into regular sequence data in the form of supplementing missing data. However, traditional position encoding methods can only focus on a single time sequence and cannot effectively express irregular multi-object access sequences in the content distribution network environment, resulting in insufficient modeling of the relative order and access density of requests. For example, as shown in FIG. 1, there are 25 access objects (i.e., R Figure 3 Figure 3 x ​​, x = 1, …, 8), while the number of access objects within each second is not uniform but has a front-back sequence characteristic, presenting obvious timing fluctuations. If the position is encoded according to the access order, the encoded sequence will ignore the time information between t1 to t6, so that the access patterns at different time points cannot be distinguished. If the position is encoded according to the access time, multiple access objects at the same time will be regarded as arriving at the same time, so that the sequence information within the time is lost, affecting the learning of the access dependency relationship.

[0082] To solve these problems, the present application proposes a multi-object time encoding (MO-TE, Multi-Object Time Encoding), as shown in Figure 4 The present application expresses the double-layer sequence relationship as a single-dimensional irregular sequence based on density, and then extends the time encoding from processing a single-dimensional regular sequence to processing an irregular density sequence, which can better capture the time characteristics in the content distribution network request. MO-TE designs a density-aware time encoding method, which meets:

[0083] Property 1 (absolute distance): If multiple requests at the same time have a front-back relationship, the encoding values should be different.

[0084] Property 2 (relative distance): If the time interval between high-density access requests is short, the encoding vector should have stronger correlation.

[0085] Based on this, step 1 of the present application can include the following steps:

[0086] Step 1.1: Obtain the first information of the training request, and a plurality of historical requests corresponding to the training request, and obtain the second information of each historical request;

[0087] Step 1.2: offline label the second information by Belady algorithm to obtain a reuse distance label, the reuse distance label includes a request distance label or a stack distance label, the request distance label represents the number of request objects between two requests responding to the same request object, and the stack distance label represents the size distance of the request object between two requests responding to the same request object, all second information corresponding to the training request and the reuse distance label form a training sample, and a plurality of training samples form a training sample set;

[0088] Step 1.3: input all second information corresponding to the training request in the training sample into a timing model to obtain context information;

[0089] Step 1.3.1: for each second information, feature map the unique identifier of the request object and the size of the request object in the second information to obtain an identifier embedding vector and a size embedding vector;

[0090] Step 1.3.2: Embed the identifier embedding vector and size embedding vector of all second information into LSTM to capture long-term dependencies of access patterns, to obtain the hidden state vector H and the hidden state H at time t t ;

[0091] Specifically, the identifier embedding vector and size embedding vector corresponding to time t-m are input into LSTM to obtain the cell state C at time t-m t-m and the hidden state H at time t-m t-m , which are specifically represented by the following formula:

[0092] (C t-m ,H t-m )=LSTM(E(id t-m );E(size t-m ));

[0093] Wherein, E(id t-m ) represents the identifier embedding vector corresponding to time t-m, and E(size t-m ) represents the size embedding vector corresponding to time t-m;

[0094] The cell state C at time t-m t-m , the identifier embedding vector and size embedding vector corresponding to time t-m+1 are input into LSTM to obtain the cell state C at time t-m+1 t-m+1 and the hidden state H at time t-m+1 t-m+1 , which are specifically represented by the following formula:

[0095] (C t-m+1 ,H t-m+1 )=LSTM([E(id t-m+1 );E(size t-m+1 )],C t-m );

[0096] Wherein, E(id t-m+1 ) represents the identifier embedding vector corresponding to time t-m+1, and E(size t-m+1 ) represents the size embedding vector corresponding to time t-m+1;

[0097] Similarly, the cell state of the previous time, the corresponding identifier embedding vector and size embedding vector of the current time are input into LSTM until time t to obtain the hidden state H at time t output by LSTM t , and the hidden state from time t-m to time t-1 is obtained at the same time, the hidden state from time t-m to time t-1 forms a hidden state vector H, which is specifically represented by the following formula:

[0098] H =(Ht-m ,…,H t-1 );

[0099] Combining Figure 1 For example, embedding the identifier vector E(id t-1 ) corresponding to time t-1, embedding the size vector E(size t-1 ) corresponding to time t-1, and the cell state C t-2 of time t-2 into the LSTM to obtain the hidden state H t-1 of time t-1. t Embedding the identifier vector E(id t ) corresponding to time t, embedding the size vector E(size t-1 ) corresponding to time t, and the cell state C t of time t-1 into the LSTM to obtain the hidden state H TE of time t.

[0100] Step 1.3.3: Perform multi-object time encoding on each time in {t-m,…,t-1} to enhance the ability to model the time information between multiple historical requests, and obtain the time encoding of each time, which is specifically represented by the following formula:

[0101] ;

[0102] ;

[0103] ;

[0104] where n represents the order position of the historical request corresponding to time in all historical requests, p represents the request density, d TE is the time encoding dimension, and are frequency control factors, represents the scale of position 0, represents the scale of position i, represents the scale of position , and L represents a predefined time scale constant.

[0105] Step 1.3.4: Based on the time encoding and hidden state vector H of each time in {t-m,…,t-1}, calculate the key vector K and value vector V of the attention network, which is specifically realized by the following formula:

[0106] (K,V)=H+[MO-TE(t-m,m,0),..,MO-TE(t-1,m,m)];

[0107] Wherein, MO-TE(t-m, m, 0) represents time encoding of time t-m, MO-TE(t-1, m, m) represents time encoding of time t-1;

[0108] Step 1.3.5: the hidden state H of time t is obtained by the following formula: t As a query vector Q, the query vector Q, the key vector K and the value vector V are input into the attention network to obtain the context information Gt, which is specifically represented by the following formula:

[0109] Gt= attention(K,Q,V);

[0110] Wherein, attention represents the attention network.

[0111] Step 1.4: the context information is transformed through the full connection layer (Linear) and the Softmax, to obtain the predicted reuse distance, which is specifically represented by the following formula:

[0112] D=Softmax(dense(Gt));

[0113] Wherein, D represents the predicted reuse distance, dense represents the full connection layer, and Gt represents the context information.

[0114] Step 1.5: based on the predicted reuse distance and the reuse distance label, the parameters in the time sequence model, the parameters of the full connection layer and the parameters of the Softmax transformation are updated, specifically, the parameters of the feature mapping, the parameters of the LSTM, the parameters of the multi-object time encoding and the parameters of the attention network are updated, the parameters of the time sequence model are updated multiple times based on the training sample set, and the trained time sequence model is obtained.

[0115] Therefore, the supervised training of the time sequence model is completed, and then the semi-online reinforcement learning based on the trained time sequence model is performed, the semi-online reinforcement learning in the application is improved based on reinforcement learning, and reinforcement learning has strong decision-making ability and is usually used to process sequential decision-making problems, that is, the process of maximizing long-term cumulative rewards through multiple rounds of interaction of decision-making-sensing-rewarding of an intelligent agent in an environment. However, in the cache admission decision scenario of hot object cache, the problem cannot be completely modeled as a typical sequential decision-making problem, because the admission decision of the cache request has a reward delay problem, that is, the current decision cannot immediately obtain its effectiveness feedback. Combined with the characteristics of the cache admission decision problem, the semi-online reinforcement learning is improved based on the reinforcement learning, and the semi-online reinforcement learning is used to solve the cache admission decision problem. Figure 5 , Figure 5 S t , S t-1 , …,S t-nrepresents the input state, Ast represents the admission decision, Rt represents the reward, and the agent needs to decide whether to admit the request R1 into the hot object cache. However, due to the buffer of cache space and the timeliness of re-access, the agent cannot immediately obtain the reward of the decision. Only when the future re-access occurs, the effectiveness of the decision will be revealed, for example:

[0116] If admitted: Only when the object is accessed again in the future can it be determined whether the decision improves the cache hit rate, otherwise if it is not accessed again before being evicted from the hot object cache, the admission decision is invalid and should be punished.

[0117] If evicted: Only when it is not accessed again within the effective access of the hot object cache capacity can the rationality of the current eviction decision be measured, and the effective access in this process depends on the admission decision of all subsequent objects.

[0118] Therefore, the hot object cache admission decision scenario deviates from the standard MDP (Markov Decision Process) in reinforcement learning, and the actual reward of the admission or eviction decision made by the agent at the current time will only be fed back at an uncertain future time. The core of this problem lies in the fact that the capacity buffer of the cache and the dynamic decision of the agent affect the life cycle of the object, making the delay of the reward signal more complex.

[0119] In view of the reward delay problem faced by reinforcement learning in the hot object cache admission environment, the present application designs a buffer mechanism to optimize the training process of reinforcement learning. The buffer mainly maintains decision data that has not obtained the corresponding reward after the agent makes a decision and decision data that has not been evicted logically in the hot object cache, which is realized by the following steps:

[0120] Step 2: Construct a state space, and sequentially pass the states in the state space through the trained time sequence model, the full connection layer and the Sigmoid transformation to obtain the admission decision, and then determine whether the target request of the user request is allowed to admit the content distribution network hot object cache;

[0121] Step 2.1: Construct a state space, which includes a plurality of states, wherein the state includes information Str of a user request, request object information Sth of a historical request corresponding to the user request, and a time sequence of initiation of the historical request

[0122] Str=(id,size, ), id represents a unique identifier of a request object corresponding to the user request, size represents a size of the request object corresponding to the user request, represents the initiation time of the user request;

[0123] ​wherein, , represents a unique identifier of a request object of a historical request at a time , represents a unique identifier of a request object of a historical request at a time , represents a size of a request object of a historical request at a time , represents a size of a request object of a historical request at a time ;

[0124] Step 2.2: inputting a state in the state space into the trained time sequence model to obtain context information of the state;

[0125] Step 2.3: transforming the context information of the state through a full connection layer (Linear) and a Sigmoid transformation to obtain an admission decision At, which is specifically represented by the following formula:

[0126] ;

[0127] wherein, represents the context information of the state, and dense represents the full connection layer;

[0128] Step 2.4: in the case of the admission decision At=1, representing that the target request object of the user request is allowed to access the hot object cache of the content distribution network, and in the case of the admission decision At=0, representing that the target request object of the user request is rejected to access the hot object cache of the content distribution network.

[0129] wherein, the state in the state space includes information Str of a user request, request object information Sth of a historical request corresponding to the user request, and a time sequence Ste of initiation of the historical request;

[0130] Step 3: setting a reward, updating the reward based on the state in the state space, and continuously optimizing and updating parameters of the trained time sequence model, parameters of the full connection layer, and parameters of the Sigmoid transformation based on the state, the admission decision, and the reward;

[0131] Step 3.1: in the case of the admission decision At=1, inserting the target request object of the user request into the hot object cache of the content distribution network, and in the case of the admission decision At=0, creating a shadow request object of the user request, the shadow request object including a unique identifier and a size of a request object corresponding to the user request, and inserting the shadow request object into the hot object cache of the content distribution network;

[0132] The shadow request object does not occupy the actual storage space of the hot object cache, but completely follows the cache management mechanism of the hot object cache, including being evicted according to the eviction policy of the hot object cache, the shadow request object simulates the cache behavior of the object after being admitted, and the triple of the request is also stored in the buffer. In this way, all request objects in the hot object cache (including rejected shadow request objects) and triples in the buffer form a one-to-one mapping relationship.

[0133] Step 3.2: Set a triple for the target request object or the shadow request object, the triple is represented as (S, A, R), wherein S represents the input state, A represents the admission decision, and R represents the reward, and the triple is stored in the buffer;

[0134] Step 3.3: Obtain the next state in the state space, update the reward R of the target request object or the shadow request object based on the request object of the user request in the state, and in the case that the target request object or the shadow request object meets the preset eviction condition, evict the target request object or the shadow request object from the hot object cache, and move the triple of the target request object or the triple of the shadow request object from the buffer to the replay pool;

[0135] The eviction condition includes the longest storage time of the request object in the hot object cache or the lowest hit frequency of the request object in the hot object cache.

[0136] Specifically, in the state space, when the request object of the user request in the state is the target request object, that is, when the target request object is hit, the value of R is increased by 1, in the case that the target request object is not hit until the preset eviction condition is met, the value of R is set to -1, and in the case that the target request object meets the preset eviction condition, the target request object is evicted from the hot object cache, and the triple of the target request object is moved from the buffer to the replay pool.

[0137] When the request object of the user request in the state is the shadow request object, that is, when the shadow request object is accessed, the value of R is reduced by 1, in the case that the shadow request object is not accessed until the preset eviction condition is met, the value of R is set to 1, and in the case that the shadow request object meets the preset eviction condition, the shadow request object is evicted from the hot object cache, and the triple of the shadow request object is moved from the buffer to the replay pool.

[0138] Step 3.4: When the number of triples in the replay pool reaches a preset threshold, according to the soft actor-critic SAC algorithm, based on the triples in the replay pool, the parameters in the trained time sequence model, the parameters of the full connection layer and the parameters of the Sigmoid transformation are updated and optimized.

[0139] Through the buffer mechanism, the reward delay problem in reinforcement learning is effectively alleviated, so that the agent can more stably learn the cache admission policy, and the hit rate of hot object cache and the overall system performance are improved.

[0140] Step 4: Based on the updated timing model, the full connection layer and the Sigmoid transformation, the admission decision of the next state is calculated, and then it is judged whether the target request of the user request in the state is allowed to access the hot object cache of the content distribution network, and then step 3 is returned to execute.

[0141] The application can more accurately model the complex dependency relationship between request objects by representing the multi-object time sequence as a density-based irregular sequence through the multi-object time coding mechanism; the application solves the delay reward problem in reinforcement learning through the buffer, so that the model can make correct decisions based on historical information; the application improves the decision-making ability and long-term stability of the model through the design of the double-head model and the two-stage training.

[0142] The application realizes the position coding of the request sequence through the multi-object time coding method, so that the algorithm can learn the two-level sequence features of the access sequence, and the learning ability of the model is significantly improved; the agent decision-making process is modeled through the buffer, so that the model can maximize the cumulative reward in the long term, and the hit rate of the hot object cache and other indicators are significantly improved.

[0143] The above description is only the preferred embodiments of the present disclosure and the explanation of the applied technical principles. It should be understood by those skilled in the art that the scope of the application involved in the embodiments of the present disclosure is not limited to the technical solutions formed by the specific combinations of the above technical features, and should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above inventive concept. For example, the above features are replaced with the technical features disclosed in the embodiments of the present disclosure (but not limited to) having similar functions to form technical solutions.

Claims

1. An edge hot object cache admission method, characterized by, The method comprises the following steps: Step 1: obtaining a training request and a historical request corresponding to the training request, training a time sequence model based on first information of the training request and second information of the historical request, and obtaining a trained time sequence model; Wherein, the first information comprises a unique identifier of a request object corresponding to the training request, a size of the request object corresponding to the training request and an initiation time t of the training request, and the second information comprises a unique identifier of a request object corresponding to the historical request, a size of the request object corresponding to the historical request and an initiation time of the historical request, the initiation times of all historical requests are represented as {t-m,…,t-1}, wherein m represents the number of historical requests; Wherein, the time sequence model comprises feature mapping, multi-object time coding, LSTM and attention network; Step 2: constructing a state space, sequentially passing states in the state space through the trained time sequence model, a full connection layer and a Sigmoid transformation to obtain an access decision, and then judging whether a target request of a user request in the state is allowed to access a hot object cache of a content distribution network; Wherein, the states in the state space comprise information Str of a user request, request object information Sth of a historical request corresponding to the user request and an initiation time sequence Ste of the historical request; Step 3: setting a reward, updating the reward based on the states in the state space, and then continuously optimizing and updating parameters of the trained time sequence model, parameters of the full connection layer and parameters of the Sigmoid transformation based on the states, the access decision and the reward; Step 4: based on the updated time sequence model, the full connection layer and the Sigmoid transformation, calculating an access decision of a next state, and then judging whether a target request of a user request in the state is allowed to access a hot object cache of a content distribution network, and then returning to execute step 3.

2. The edge thermal object cache admission method of claim 1, wherein, Step 1 specifically comprises: Step 1.1: obtaining first information of a training request, and a plurality of historical requests corresponding to the training request, and obtaining second information of each historical request; Step 1.2: offline labeling the second information by Belady algorithm to obtain a reuse distance label, the reuse distance label comprising a request distance label or a stack distance label, the request distance label representing the number of request objects between two requests responding to the same request object, the stack distance label representing the size distance of request objects between two requests responding to the same request object, all second information corresponding to the training request and the reuse distance label forming a training sample, and a plurality of training samples forming a training sample set; Step 1.3: inputting all second information corresponding to the training request in the training sample into the time sequence model to obtain context information; Step 1.4: transforming the context information through the full connection layer and the Softmax transformation to obtain a predicted reuse distance, which is specifically represented by the following formula: D=Softmax(dense(Gt)); Wherein, D represents the predicted reuse distance, dense represents the full connection layer, and Gt represents the context information; Step 1.5: updating the parameters in the time series model, the parameters of the full connection layer and the parameters of the Softmax transformation based on the predicted reuse distance and the reuse distance label, and iteratively updating the parameters of the time series model based on the training sample set multiple times to obtain the trained time series model.

3. The edge thermal object cache admission method of claim 2, wherein, Step 1.3 specifically includes: Step 1.3.1: for each second information, the unique identifier of the request object and the size of the request object in the second information are feature mapped to obtain an identifier embedding vector and a size embedding vector; Step 1.3.2: Embed the identifiers of all second information into the vector and size embedding vectors into the LSTM to get the hidden state vector H and the hidden state H at time t t ; Specifically, the identifier embedding vector corresponding to time t-m and the size embedding vector are input into the LSTM to obtain the cell state C of time t-m t-m and the hidden state H of time t-m t-m , and are specifically represented by the following formula: (C t-m ,H t-m )=LSTM(E(id t-m );E(size t-m )); wherein E(id t-m ) represents an identifier embedding vector corresponding to time t-m, and E(size t-m ) represents a size embedding vector corresponding to time t-m. The cell state C of time t-m is input into the LSTM t-m , the identifier embedding vector and the size embedding vector corresponding to the time t-m+1 are input into the LSTM, to obtain the cell state C of time t-m+1 t-m+1 and the hidden state H of time t-m+1 t-m+1 , which are specifically represented by the following formula: (C t-m+1 ,H t-m+1 )=LSTM([E(id t-m+1 );E(size t-m+1 )],C t-m ); wherein E(id t-m+1 ) represents an identifier embedding vector corresponding to time t-m+1, and E(size t-m+1 ) represents a size embedding vector corresponding to time t-m+1. Similarly, the cell state at the previous time, the corresponding identifier embedding vector and the size embedding vector at the current time are input into the LSTM to obtain the hidden state H at time t of the LSTM output until time t t At the same time, the hidden states from time t-m to time t-1 are obtained, and the hidden states from time t-m to time t-1 form a hidden state vector H, which is specifically represented by the following formula: H = (H t-m ,…,H t-1 ) Step 1.3.3: Time encoding for each time in {t-m,...,t-1} performing multi-object time encoding to obtain time encoding for each time, specifically by the following formula: ； ； ； where n represents time the order position of the corresponding historical request in all historical requests, p represents the request density, d TE is the time encoding dimension, represents the scale of position 0, represents the scale of position i, represents the scale of position L represents a predefined time scale constant; Step 1.3.4: based on the time encoding and the hidden state vector H of each time in {t-m,…,t-1}, the key vector K and the value vector V of the attention network are calculated, which is specifically realized by the following formula: (K,V)=H+[MO-TE(t-m,m,0),..,MO-TE(t-1,m,m)]; Where MO-TE(t-m,m,0) represents the time encoding of time t-m, and MO-TE(t-1,m,m) represents the time encoding of time t-1; Step 1.3.5: The hidden state H at time t is obtained by t As the query vector Q, the query vector Q, the key vector K and the value vector V are input into the attention network to obtain the context information Gt, which is specifically represented by the following formula: Gt= attention(K,Q,V); Where attention represents the attention network.

4. The edge thermal object cache admission method of claim 2, wherein, The updating of the parameters in the time series model, the parameters of the full connection layer and the parameters of the Softmax transformation in step 1.5 specifically includes: updating the parameters of feature mapping, the parameters of LSTM, the parameters of multi-object time encoding, and the parameters of attention network.

5. The edge thermal object cache admission method of claim 1, wherein, Step 2 specifically includes: Step 2.1: constructing a state space, the state space comprising a plurality of states, the states comprising information Str of a user request, request object information Sth of a historical request corresponding to the user request, and a sequence of initiation times of the historical request ; Str = (id, size, ), id represents a unique identifier of the request object corresponding to the user request, size represents the size of the request object corresponding to the user request, represents the initiation time of the user request; wherein, , represents a unique identifier of a request object of a historical request at a time , represents a unique identifier of a request object of a historical request at a time , represents a size of a request object of a historical request at a time , represents a size of a request object of a historical request at a time , Step 2.2: inputting the state in the state space into the trained time series model to obtain the context information of the state; Step 2.3: passing the context information of the state through the full connection layer and the Sigmoid transformation to obtain the admission decision At, which is specifically represented by the following formula: ； wherein, contextual information representing the state, dense denotes a fully connected layer; Step 2.4: in the case of admission decision At=1, it represents that the target request object of the user request is allowed to access the hot object cache of the content distribution network, and in the case of admission decision At=0, it represents that the target request object of the user request is rejected to access the hot object cache of the content distribution network.

6. The edge thermal object cache admission method of claim 5, wherein, Step 3 specifically includes: Step 3.1: in the case of admission decision At=1, the target request object of the user request is inserted into the hot object cache of the content distribution network, and in the case of admission decision At=0, a shadow request object of the user request is created, which includes the unique identifier and size of the request object corresponding to the user request. The shadow request object is inserted into the hot object cache of the content distribution network; Step 3.2: setting a triple for the target request object or the shadow request object, the triple is represented as (S,A,R), wherein S represents the input state, A represents the admission decision, and R represents the reward. The triple is stored in the buffer; Step 3.3: obtaining the next state in the state space, updating the reward R of the target request object or the shadow request object based on the request object requested by the user in the state, and in the case that the target request object or the shadow request object meets the preset eviction condition, evicting the target request object or the shadow request object from the hot object cache and moving the triple of the target request object or the triple of the shadow request object from the buffer to the replay pool; wherein the eviction condition comprises that the request object in the hot object cache has the longest storage time or the request object in the hot object cache has the lowest hit frequency; Step 3.4: when the number of triples in the replay pool reaches a preset threshold, updating the parameters in the trained time sequence model, the parameters of the full connection layer and the parameters of the Sigmoid transformation based on the triples in the replay pool according to the soft actor-critic SAC algorithm.

7. The edge thermal object cache admission method of claim 6, wherein, Step 3.3 specifically comprises: in the state space, obtaining the next state, when the request object requested by the user in the state is the target request object, that is, when the target request object is hit, the value of R is increased by 1, in the case that the target request object is not hit until the preset eviction condition is met, the value of R is set to -1, and in the case that the target request object meets the preset eviction condition, the target request object is evicted from the hot object cache, and the triple of the target request object is moved from the buffer to the replay pool; when the request object requested by the user in the state is the shadow request object, that is, when the shadow request object is accessed, the value of R is reduced by 1, in the case that the shadow request object is not accessed until the preset eviction condition is met, the value of R is set to 1, and in the case that the shadow request object meets the preset eviction condition, the shadow request object is evicted from the hot object cache, and the triple of the shadow request object is moved from the buffer to the replay pool.

Images