An edge hot object cache admission method

Abstract

The application provides an edge hot object cache admission method, relates to the field of information technology, and constructs a time sequence model, the time sequence model includes feature mapping, multi-object time coding, LSTM and attention network, the time sequence model is supervised training, and the time sequence model that has completed training is obtained, wherein the application realizes position coding of a request sequence through a multi-object time coding method, so that the algorithm can learn two-level sequence characteristics of an access sequence, and the learning ability of the model is significantly improved. The application sets a buffer, inserts a corresponding target request into a hot object cache, creates a shadow request object and inserts it into the hot object cache, then calculates the corresponding rewards respectively, and based on the state, admission decision and reward, the trained time sequence model is subjected to semi-online reinforcement learning, so that the model can maximize the cumulative reward for a long time, and the hit rate and other indicators of the hot object cache are significantly improved.

Description

Technical Field

[0001] This invention relates to the field of information technology, specifically to an edge hot object caching admission method. Background Technology

[0002] With the rise of the computing power network concept, various computing, storage, and network resources within the network are uniformly orchestrated and dynamically scheduled, forming a cross-domain, heterogeneous, and on-demand elastically scalable distributed computing platform. Within this framework, the Content Delivery Network (CDN) caching system serves as both a component of the distributed storage nodes and undertakes the dual tasks of reducing the network's core bandwidth load and improving edge computing response performance. A typical CDN solution deploys "hot object caching + disk caching" on various nodes within the computing power network. The former stores the most frequently accessed data, while the latter is used for persistent storage. When a user request arrives, the hot object cache on the edge node closest to the user responds first. If a request is not found, it can be coordinated with the local disk cache or other nodes in the computing power network to access backend storage. Through intelligent scheduling and admission decisions, the CDN caching system optimizes data transmission latency and reduces the load on the core network and computing centers, improving overall resource utilization efficiency. However, the hot object cache has limited capacity and cannot cache all requested data indefinitely. Therefore, when a user's request is not found in the hot object cache, it is necessary to selectively decide whether to cache the current object in the hot object cache. This process is commonly referred to as the hot object cache admission decision.

[0003] Chinese patent "CN202410685449.X Cache Admission Method and System Based on Machine Learning in Variable Cache Space Scenarios" provides a hot object cache admission method: When a cache miss occurs, the system first temporarily stores the user's most recent K (preferably 19) access records and current access information into a feature buffer window; all records in the window are sequentially subjected to one-hot encoding dimensionality reduction, embedding processing, and tanh normalization to obtain a feature vector of uniform length; then, the vector is input into an offline-trained LSTM model to predict the "reuse distance" classification label; finally, combined with the current available cache capacity threshold, the intermediate parameter A is calculated. When A ≥ 0 (the predicted reuse distance is within the carrying capacity range), the data is allowed to be written to the hot object cache; otherwise, writing is rejected, thus achieving high-accuracy admission decision even when the cache capacity changes dynamically.

[0004] Existing hot object caching admission methods are mainly divided into two categories: hotness information prediction and admission threshold judgment.

[0005] The technical solution described in Chinese patent "CN202410685449.X Cache Admission Method and System Based on Machine Learning in Variable Cache Space Scenarios" relies on a high-dimensional feature set, including various user access markers, object information markers, and system load markers, to ensure prediction accuracy during model training and online decision-making. If applied to a general dataset containing only basic attributes (such as id, size, and time), the lack of these high-dimensional features leads to a significant drop in model performance. Furthermore, the available feature dimensions and distributions vary across different platforms or business scenarios, making this method highly targeted to specific datasets and lacking cross-scenario and cross-platform versatility, thus limiting its applicability.

[0006] Threshold-based admission methods require determining optimized thresholds for object metrics (such as object size, object frequency, and access time). An object is only admitted to the cache if its metrics meet the corresponding threshold. Representative methods include AdaptSize, HillClimbing, Percentile, and Darwin, which improve the intelligence of caching strategies to some extent. However, these strategies still rely on manually defined feature metrics, cannot learn complex dependencies in request patterns, and cannot support information-based access decisions. Summary of the Invention

[0007] To address the shortcomings of existing technologies, the present invention aims to propose an edge hot object caching admission method, comprising:

[0008] Step 1: Obtain the training request and the corresponding historical request. Based on the first information of the training request and the second information of the historical request, train the time series model to obtain the trained time series model.

[0009] The first information includes the unique identifier of the request object corresponding to the training request, the size of the request object corresponding to the training request, and the initiation time t of the training request. The second information includes the unique identifier of the request object corresponding to the historical request, the size of the request object corresponding to the historical request, and the initiation time of the historical request. The initiation time of all historical requests is represented as {tm,…,t-1}, where m represents the number of historical requests.

[0010] The temporal model includes feature mapping, multi-object temporal coding, LSTM, and attention network;

[0011] Step 2: Construct the state space, and pass the states in the state space through the trained temporal model, fully connected layer and Sigmoid transformation in sequence to obtain the admission decision, and then determine whether the target request of the user in the state is allowed to be admitted to the hot object cache of the content delivery network.

[0012] The states in the state space include the user request information Str, the request object information Sth of the historical requests corresponding to the user request, and the initiation time sequence Ste of the historical requests.

[0013] Step 3: Set rewards, update rewards based on states in the state space, and then continuously optimize and update the parameters of the trained temporal model, the parameters of the fully connected layer, and the parameters of the sigmoid transform based on states, admission decisions, and rewards.

[0014] Step 4: Based on the updated temporal model, fully connected layer, and Sigmoid transformation, calculate the admission decision for the next state, and then determine whether the target request of the user request in the state is allowed to be admitted to the hot object cache of the content delivery network, and then return to execute step 3.

[0015] Optionally, step 1 specifically includes:

[0016] Step 1.1: Obtain the first information of the training request, as well as multiple historical requests corresponding to the training request, and obtain the second information of each historical request;

[0017] Step 1.2: Offline annotation of the second information is performed using the Belady algorithm to obtain reuse distance labels. The reuse distance labels include request distance labels or stack distance labels. The request distance label represents the number of request objects between two requests that return the same request object in response. The stack distance label represents the size distance between two requests that return the same request object in response. All the second information corresponding to the training request and the reuse distance labels are used to form training samples. Multiple training samples are used to form a training sample set.

[0018] Step 1.3: Input all the second information corresponding to the training requests in the training samples into the time series model to obtain the context information;

[0019] Step 1.4: Transform the context information through a fully connected layer and Softmax to obtain the predicted reuse distance, specifically expressed by the following formula:

[0020] D = Softmax(dense(Gt));

[0021] Where D represents the predicted reuse distance, dense represents the fully connected layer, and Gt represents the context information;

[0022] Step 1.5: Based on the predicted reuse distance and reuse distance labels, update the parameters in the time series model, the parameters of the fully connected layer, and the parameters of the Softmax transform. Iterate the parameters of the time series model multiple times based on the training sample set to obtain the trained time series model.

[0023] Optionally, step 1.3 specifically includes:

[0024] Step 1.3.1: For each piece of second information, perform feature mapping on the unique identifier of the request object and the size of the request object in the second information to obtain the identifier embedding vector and the size embedding vector;

[0025] Step 1.3.2: Input the identifier embedding vector and size embedding vector of all the second information into the LSTM to obtain the hidden state vector H and the hidden state H at time t. t ;

[0026] Specifically, the identifier embedding vector and size embedding vector corresponding to time tm are input into the LSTM to obtain the cell state C at time tm. t-m The hidden state H of time tm t-m Specifically, it is expressed by the following formula:

[0027] (C t-m H t-m )=LSTM(E(id t-m E(size) t-m ));

[0028] Among them, E(id) t-m E(size) represents the identifier embedding vector corresponding to time tm. t-m ) represents the size embedding vector corresponding to time tm;

[0029] Cellular state C at time tm t-m The identifier embedding vector and size embedding vector corresponding to time t-m+1 are input into the LSTM to obtain the cell state C at time t-m+1. t-m+1 And the hidden state H at time t-m+1 t-m+1 Specifically, it is expressed by the following formula:

[0030] (C t-m+1 H t-m+1 )=LSTM([E(id t-m+1 E(size) t-m+1 )],C t-m );

[0031] Among them, E(id) t-m+1 E(size) represents the identifier embedding vector corresponding to time t-m+1. t-m+1 ) represents the size embedding vector corresponding to time t-m+1;

[0032] Similarly, the cell state from the previous time step, the corresponding identifier embedding vector and size embedding vector for the current time step are input into the LSTM until time t, at which point the hidden state H at time t is obtained from the LSTM output. t Simultaneously, the hidden states from time tm to time t-1 are obtained. These hidden states from time tm to time t-1 form the hidden state vector H, which is specifically represented by the following formula:

[0033] H =(H t-m ,…,H t-1 );

[0034] Step 1.3.3: For each time in {tm,…,t-1} Multi-object time encoding is performed to obtain the time code for each time point, which is represented by the following formula:

[0035] ;

[0036] ;

[0037] ;

[0038] Where n represents time The corresponding historical request's sequential position within all historical requests, where p represents request density and d... TE Encoding dimension for time, This indicates the scale at position 0. This indicates the scale of position i. Indicates position The scaling factor is L, which represents a predefined time scale constant.

[0039] Step 1.3.4: Based on the temporal encoding and hidden state vector H at each time step in {tm,…,t-1}, calculate the key vector K and value vector V of the attention network, specifically through the following formula:

[0040] (K,V)=H+[MO-TE(tm,m,0),..,MO-TE(t-1,m,m)];

[0041] Where MO-TE(tm,m,0) represents the time code of time tm, and MO-TE(t-1,m,m) represents the time code of time t-1;

[0042] Step 1.3.5: H of the hidden state at time t t As the query vector Q, the query vector Q, the key vector K, and the value vector V are input into the attention network to obtain the context information Gt, which is specifically represented by the following formula:

[0043] Gt = attention(K,Q,V);

[0044] Here, attention refers to the attention network.

[0045] Optionally, step 1.5 updates the parameters in the time series model, the parameters of the fully connected layer, and the parameters of the Softmax transform, specifically including:

[0046] The parameters of the feature map, LSTM, multi-object temporal encoding, and attention network are updated.

[0047] Optionally, step 2 specifically includes:

[0048] Step 2.1: Construct a state space containing multiple states. Each state includes information about the user request (Str), request object information (Sth) of the historical requests corresponding to the user request, and the initiation time sequence of the historical requests. ;

[0049] Where Str=(id,size, ), where id represents the unique identifier of the request object corresponding to the user request, and size represents the size of the request object corresponding to the user request. Indicates the time when the user request was initiated;

[0050] in, , Indicates time The unique identifier of the request object in the history of requests. Indicates time The unique identifier of the request object in the history of requests. Indicates time The size of the request object in the history of requests. Indicates time The size of the request object in the history of requests;

[0051] Step 2.2: Input the states in the state space into the trained time series model to obtain the context information of the states;

[0052] Step 2.3: Pass the context information of the state through a fully connected layer and a Sigmoid transformation to obtain the admission decision At, which is specifically expressed by the following formula:

[0053] ;

[0054] in, Contextual information representing the state; dense indicates a fully connected layer.

[0055] Step 2.4: When the admission decision At=1, the target request object representing the user's request is allowed to be admitted to the hot object cache of the content delivery network. When the admission decision At=0, the target request object representing the user's request is denied admission to the hot object cache of the content delivery network.

[0056] Optionally, step 3 specifically includes:

[0057] Step 3.1: When the admission decision At=1, insert the target request object of the user request into the hot object cache of the content delivery network. When the admission decision At=0, create a shadow request object of the user request. The shadow request object includes the unique identifier and size of the request object corresponding to the user request. Insert the shadow request object into the hot object cache of the content delivery network.

[0058] Step 3.2: Set up a triple for the target request object or shadow request object. The triple is represented as (S, A, R), where S represents the input state, A represents the admission decision, and R represents the reward. Store the triple in the buffer.

[0059] Step 3.3: Obtain the next state in the state space. Based on the request object requested by the user in the state, update the reward R of the target request object or shadow request object. If the target request object or shadow request object meets the preset eviction conditions, evict the target request object or shadow request object from the hot object cache and move the triple of the target request object or shadow request object from the buffer to the replay pool.

[0060] The eviction conditions include the object being requested for the longest time in the hot object cache, or the object being requested for the lowest frequency of being hit in the hot object cache.

[0061] Step 3.4: When the number of triples in the replay pool reaches a preset threshold, the parameters of the trained temporal model, the parameters of the fully connected layer, and the parameters of the Sigmoid transform are optimized and updated based on the triples in the replay pool, according to the soft actor-critic SAC algorithm.

[0062] Optionally, step 3.3 specifically includes:

[0063] In the state space, obtain the next state. When the user request object in the state is the target request object, that is, when the target request object is hit, the value of R is incremented by 1. If the target request object is not hit until the preset eviction condition is met, the value of R is set to -1. If the target request object meets the preset eviction condition, the target request object is evicted from the hot object cache, and the triple of the target request object is moved from the buffer to the replay pool.

[0064] When the user request in the state is a shadow request object, that is, when the shadow request object is accessed, the value of R is decremented by 1. If the shadow request object is not accessed until the preset eviction condition is met, the value of R is set to 1. If the shadow request object meets the preset eviction condition, the shadow request object is evicted from the hot object cache, and the triple of the shadow request object is moved from the buffer to the replay pool.

[0065] The beneficial effects of adopting the above technical solution are as follows:

[0066] This invention constructs a temporal model, which includes feature mapping, multi-object temporal encoding, LSTM, and an attention network. Supervised training is performed on the temporal model to obtain a trained temporal model. Specifically, this invention uses multi-object temporal encoding to encode the position of the request sequence, enabling the algorithm to learn the two-level sequence features of the access sequence, significantly improving the model's learning ability. This invention sets up a buffer and, for admission cases, inserts the corresponding target request into the hot object cache; for rejection cases, creates a shadow request object and inserts it into the hot object cache, then calculates the corresponding rewards. Based on the state, admission decision, and reward, semi-online reinforcement learning is performed on the trained temporal model. Through the agent decision-making process, the model can stably maximize the cumulative reward in the long term, significantly improving metrics such as the hit rate of the hot object cache. Attached Figure Description

[0067] Figure 1 This is a schematic diagram of the structure of an edge hot object caching admission method in an embodiment of the present invention;

[0068] Figure 2 This is a flowchart illustrating an edge hot object caching admission method according to an embodiment of the present invention;

[0069] Figure 3 This is a schematic diagram of a conventional position encoding method in an embodiment of the present invention;

[0070] Figure 4 This is a schematic diagram of multi-object time encoding in an embodiment of the present invention;

[0071] Figure 5 This is a schematic diagram of agent decision-making in an embodiment of the present invention. Detailed Implementation

[0072] The specific embodiments of the present invention will be described in further detail below with reference to the accompanying drawings and examples. The following examples are for illustrative purposes only and are not intended to limit the scope of the invention.

[0073] To more clearly describe this invention, the application scenarios of Content Delivery Networks (CDNs) are first introduced below. A CDN is a distributed network of servers deployed globally to accelerate content transmission and delivery services on the Internet. When a user initiates a content request, the CDN selects the nearest edge server to the user's geographical location to provide the content service, rather than loading it from the origin server website. This reduces bandwidth waste and latency, while also improving user experience and lowering service provider costs.

[0074] Content Delivery Network (CDN) storage structures typically consist of two layers: a hot object cache and a disk cache. The hot object cache offers advantages such as fast read / write speeds and no data loss, but generally has a smaller capacity. The disk cache offers advantages such as large capacity, but slower read / write speeds, and its lifespan is limited by the number of read / write cycles. However, CDN edge servers have limited capacity and cannot store all content from the origin server. Therefore, it's necessary to select content that is frequently requested in the short term based on edge user behavior. When a user makes a request, the CDN first checks if the requested object is cached in the hot object cache. If it is, the object is returned directly to the user. If not, it checks the disk cache. If the object is cached, it is returned to the user, and a decision is made on whether to promote it to the hot object cache for future requests. If the object is not cached, the CDN requests it from the origin server and decides whether to cache it in the hot object cache. The process of deciding whether to cache an object in the hot object cache when it is not found is called the hot object cache admission decision.

[0075] To address the problems of existing technologies, this invention provides an edge-hot object caching admission method. To optimize the caching admission strategy of content delivery networks, this invention proposes a dual-head model architecture. This model combines multi-object temporal encoding to extract complex temporal features and optimizes the caching admission strategy through reinforcement learning with a buffer mechanism. The overall architecture of the model is shown in Figure 1. This invention includes a temporal model, fully connected layers and Softmax transform, and fully connected layers and Sigmoid transform. First, the model consisting of the temporal model, fully connected layers, and Softmax transform is subjected to supervised training. After obtaining the trained temporal model, semi-online reinforcement learning is performed based on the trained temporal model, fully connected layers, and Sigmoid transform. Specifically, combined with... Figure 2 This may include the following steps:

[0076] Step 1: Obtain the training request and the corresponding historical request. Based on the first information of the training request and the second information of the historical request, train the time series model to obtain the trained time series model.

[0077] The first information includes the unique identifier of the request object corresponding to the training request, the size of the request object corresponding to the training request, and the initiation time t of the training request. The second information includes the unique identifier of the request object corresponding to the historical request, the size of the request object corresponding to the historical request, and the initiation time of the historical request. The initiation time of all historical requests is represented as {tm,…,t-1}, where m represents the number of historical requests.

[0078] It is understandable that the historical requests corresponding to the training request are multiple requests prior to the time t of the training request.

[0079] The temporal model includes feature mapping, multi-object temporal encoding, LSTM, and attention network.

[0080] This invention designs a multi-object time coding system to address the problem that existing location coding cannot effectively represent irregular multi-object access sequences in a content delivery network environment. The following section provides a detailed description of the problems existing in the prior art and the multi-object time coding system provided by this invention.

[0081] Because request sequences (i.e., sequences of multiple requests) often exhibit dependencies, meaning that access patterns for certain objects are influenced by historical requests or user group preferences, the Attention mechanism, while possessing powerful historical information extraction and learning capabilities when processing sequence data, primarily targets single-dimensional sequences. Content Delivery Network (CDN) request access sequences typically exhibit irregular multi-object access, where multiple objects are accessed sequentially within the same timeframe, and the number of accessed objects varies across different time points. This access pattern forms two layers of sequence relationships in the time dimension: an intra-time sequence, representing the sequential order of multiple accessed objects within a given moment; and an inter-time sequence, representing the access order across different time points. Existing methods for irregularly sampled sequence data typically transform them into regular sequence data by supplementing missing data. However, traditional positional encoding methods only focus on a single time series and cannot effectively represent irregular multi-object access sequences in a CDN environment, resulting in insufficient modeling of the relative order and access density of requests. For example, ... Figure 3 As shown, there are a total of 25 accessed objects during the 6 time periods from t1 to t6 (i.e., Figure 3 R in x(x=1,…,8), and the number of accessed objects per second is uneven, but has sequential characteristics, exhibiting obvious temporal fluctuations. If position encoding is performed only according to the access order, the encoded sequence will ignore the time information between t1 and t6, making it impossible to distinguish access patterns at different time points. If position encoding is performed according to the access time, multiple accessed objects at the same time will be considered to have arrived simultaneously, thus losing the sequence information within that time moment and affecting the learning of access dependencies.

[0082] To address these issues, this invention proposes Multi-Object Time Encoding (MO-TE), such as... Figure 4 As shown, this invention expresses the two-layer sequence relationship as a density-based single-dimensional irregular sequence, and then extends the temporal encoding from processing single-dimensional regular sequences to processing irregular density sequences, which can better capture the temporal features in content delivery network requests. MO-TE designs a density-aware temporal encoding method that satisfies:

[0083] Property 1 (Absolute Distance): If multiple requests at the same time have a sequential relationship, the encoded values ​​should be different.

[0084] Property 2 (relative distance): If the time interval between high-density access requests is short, their encoded vectors should have a stronger correlation.

[0085] Based on this, step 1 of the present invention may specifically include the following steps:

[0086] Step 1.1: Obtain the first information of the training request, as well as multiple historical requests corresponding to the training request, and obtain the second information of each historical request;

[0087] Step 1.2: Offline annotation of the second information is performed using the Belady algorithm to obtain reuse distance labels. The reuse distance labels include request distance labels or stack distance labels. The request distance label represents the number of request objects between two requests that return the same request object in response. The stack distance label represents the size distance between two requests that return the same request object in response. All the second information corresponding to the training request and the reuse distance labels are used to form training samples. Multiple training samples are used to form a training sample set.

[0088] Step 1.3: Input all the second information corresponding to the training requests in the training samples into the time series model to obtain the context information;

[0089] Step 1.3.1: For each piece of second information, perform feature mapping on the unique identifier of the request object and the size of the request object in the second information to obtain the identifier embedding vector and the size embedding vector;

[0090] Step 1.3.2: Input the identifier embedding vector and size embedding vector of all second information into the LSTM to capture long-term dependencies of access patterns, and obtain the hidden state vector H and the hidden state H at time t. t ;

[0091] Specifically, the identifier embedding vector and size embedding vector corresponding to time tm are input into the LSTM to obtain the cell state C at time tm. t-m The hidden state H of time tm t-m Specifically, it is expressed by the following formula:

[0092] (C t-m H t-m )=LSTM(E(id t-m E(size) t-m ));

[0093] Among them, E(id) t-m E(size) represents the identifier embedding vector corresponding to time tm. t-m ) represents the size embedding vector corresponding to time tm;

[0094] Cellular state C at time tm t-m The identifier embedding vector and size embedding vector corresponding to time t-m+1 are input into the LSTM to obtain the cell state C at time t-m+1. t-m+1 And the hidden state H at time t-m+1 t-m+1 Specifically, it is expressed by the following formula:

[0095] (C t-m+1 H t-m+1 )=LSTM([E(id t-m+1 E(size) t-m+1 )],C t-m );

[0096] Among them, E(id) t-m+1 E(size) represents the identifier embedding vector corresponding to time t-m+1. t-m+1 ) represents the size embedding vector corresponding to time t-m+1;

[0097] Similarly, the cell state from the previous time step, the corresponding identifier embedding vector and size embedding vector for the current time step are input into the LSTM until time t, at which point the hidden state H at time t is obtained from the LSTM output. t Simultaneously, the hidden states from time tm to time t-1 are obtained. These hidden states from time tm to time t-1 form the hidden state vector H, which is specifically represented by the following formula:

[0098] H =(Ht-m ,…,H t-1 );

[0099] Combination Figure 1 For example, embed the identifier corresponding to time t-1 into the vector E(id) t-1 ), the size embedding vector E(size) corresponding to time t-1 t-1 Cell state C at time t-2 t-2 Inputting the LSTM, we obtain the hidden state H at time t-1. t-1 Embed the identifier corresponding to time t into vector E(id) t ), the size embedding vector E(size) corresponding to time t t Cell state C at time t-1 t-1 Inputting an LSTM, we obtain the hidden state H at time t. t .

[0100] Step 1.3.3: For each time in {tm,…,t-1} Multi-object time encoding is performed to enhance the ability to model time information between multiple historical requests, resulting in a time code for each time point, specifically expressed by the following formula:

[0101] ;

[0102] ;

[0103] ;

[0104] Where n represents time The corresponding historical request's sequential position within all historical requests, where p represents request density and d... TE Encoding dimension for time, and All of these are frequency control factors. This indicates the scale at position 0. This indicates the scale of position i. Indicates position The scaling factor is L, which represents a predefined time scale constant.

[0105] Step 1.3.4: Based on the temporal encoding and hidden state vector H at each time step in {tm,…,t-1}, calculate the key vector K and value vector V of the attention network, specifically through the following formula:

[0106] (K,V)=H+[MO-TE(tm,m,0),..,MO-TE(t-1,m,m)];

[0107] Where MO-TE(tm,m,0) represents the time code of time tm, and MO-TE(t-1,m,m) represents the time code of time t-1;

[0108] Step 1.3.5: H of the hidden state at time t t As the query vector Q, the query vector Q, the key vector K, and the value vector V are input into the attention network to obtain the context information Gt, which is specifically represented by the following formula:

[0109] Gt = attention(K,Q,V);

[0110] Here, attention refers to the attention network.

[0111] Step 1.4: Transform the context information through a fully connected layer (Linear) and a Softmax layer to obtain the predicted reuse distance, specifically expressed by the following formula:

[0112] D = Softmax(dense(Gt));

[0113] Where D represents the predicted reuse distance, dense represents the fully connected layer, and Gt represents the context information;

[0114] Step 1.5: Based on the predicted reuse distance and reuse distance label, update the parameters in the temporal model, the parameters of the fully connected layer, and the parameters of the Softmax transform. Specifically, update the parameters of the feature map, the LSTM, the multi-object temporal encoding, and the attention network. Iterate the parameters of the temporal model multiple times based on the training sample set to obtain the trained temporal model.

[0115] Thus, this invention completes supervised training of the temporal model. Next, semi-online reinforcement learning is performed based on the trained temporal model. The semi-online reinforcement learning in this invention is an improvement upon reinforcement learning. Reinforcement learning, with its powerful decision-making capabilities, is typically used to handle sequential decision problems, i.e., the process by which an agent maximizes long-term cumulative rewards through multiple rounds of interaction between decision-making, perception, and reward in an environment. However, in the hot object caching admission decision scenario, this problem cannot be fully modeled as a typical sequential decision problem because the admission decision for cache requests suffers from a reward delay problem; that is, the validity feedback of the current decision cannot be obtained immediately. Combined with... Figure 5 , Figure 5 S t S t-1 ,….,S t-nLet Rt represent the input state, Ast represent the admission decision, and Rt represent the reward. When faced with a request R1, the agent needs to decide whether to admit it into cache. However, due to the buffering space and the sequential nature of re-access, the agent cannot immediately obtain the benefit of this decision. The effectiveness of the decision will only be revealed when a future re-access occurs, for example:

[0116] Admission criteria: It can only be determined whether the decision has improved the cache hit rate if the object is accessed again in the future. Otherwise, if the object is not accessed again before being evicted from the hot object cache, the admission decision is invalid and should be penalized.

[0117] If eviction occurs: the rationality of the current eviction decision can only be measured if the hot object is not accessed again within the valid access capacity of the cache. The valid access in this process depends on the admission decisions of all subsequent objects.

[0118] Therefore, the hot object caching admission decision scenario deviates somewhat from the standard MDP (Markov Decision Process) in reinforcement learning. The actual reward for the admission or expulsion decision made by the agent at the current moment will only be reflected at some uncertain point in the future. The core of this problem lies in the fact that the cache capacity and the agent's dynamic decisions affect the lifecycle of objects, making the delay of reward signals more complex.

[0119] To address the reward delay problem faced by reinforcement learning in a hot object cache admission environment, this invention designs a buffer mechanism to optimize the training process of reinforcement learning. The buffer mainly maintains decision data that the agent has not yet received the corresponding reward after making a decision, as well as decision data that has not been logically evicted from the hot object cache. Specifically, it is implemented through the following steps:

[0120] Step 2: Construct the state space, and pass the states in the state space through the trained temporal model, fully connected layer and Sigmoid transformation in sequence to obtain the admission decision, and then determine whether the target request of the user in the state is allowed to be admitted to the hot object cache of the content delivery network.

[0121] Step 2.1: Construct a state space containing multiple states. Each state includes information about the user request (Str), request object information (Sth) of the historical requests corresponding to the user request, and the initiation time sequence of the historical requests. ;

[0122] Where Str=(id,size, ), where id represents the unique identifier of the request object corresponding to the user request, and size represents the size of the request object corresponding to the user request. Indicates the time when the user request was initiated;

[0123] in, , Indicates time The unique identifier of the request object in the history of requests. Indicates time The unique identifier of the request object in the history of requests. Indicates time The size of the request object in the history of requests. Indicates time The size of the request object in the history of requests;

[0124] Step 2.2: Input the states in the state space into the trained time series model to obtain the context information of the states;

[0125] Step 2.3: Transform the context information of the state through a fully connected layer (Linear) and a Sigmoid layer to obtain the admission decision At, which is specifically expressed by the following formula:

[0126] ;

[0127] in, Contextual information representing the state; dense indicates a fully connected layer.

[0128] Step 2.4: When the admission decision At=1, the target request object representing the user's request is allowed to be admitted to the hot object cache of the content delivery network. When the admission decision At=0, the target request object representing the user's request is denied admission to the hot object cache of the content delivery network.

[0129] The states in the state space include the user request information Str, the request object information Sth of the historical requests corresponding to the user request, and the initiation time sequence Ste of the historical requests.

[0130] Step 3: Set rewards, update rewards based on states in the state space, and then continuously optimize and update the parameters of the trained temporal model, the parameters of the fully connected layer, and the parameters of the sigmoid transform based on states, admission decisions, and rewards.

[0131] Step 3.1: When the admission decision At=1, insert the target request object of the user request into the hot object cache of the content delivery network. When the admission decision At=0, create a shadow request object of the user request. The shadow request object includes the unique identifier and size of the request object corresponding to the user request. Insert the shadow request object into the hot object cache of the content delivery network.

[0132] In this system, shadow request objects do not occupy actual storage space in the hot object cache, but they fully comply with the hot object cache's cache management mechanism, including being evicted according to the hot object cache's eviction policy. The shadow request object simulates the cache behavior of the object if it were admitted, and the triple of the request is also stored in the buffer. In this way, a one-to-one mapping relationship is formed between all request objects in the hot object cache (including rejected shadow request objects) and the triples in the buffer.

[0133] Step 3.2: Set up a triple for the target request object or shadow request object. The triple is represented as (S, A, R), where S represents the input state, A represents the admission decision, and R represents the reward. Store the triple in the buffer.

[0134] Step 3.3: Obtain the next state in the state space. Based on the request object requested by the user in the state, update the reward R of the target request object or shadow request object. If the target request object or shadow request object meets the preset eviction conditions, evict the target request object or shadow request object from the hot object cache and move the triple of the target request object or shadow request object from the buffer to the replay pool.

[0135] The eviction conditions include the object being requested for the longest time in the hot object cache, or the object being requested for the lowest frequency of being hit in the hot object cache.

[0136] Specifically, the next state is obtained in the state space. When the user request object in the state is the target request object, that is, when the target request object is hit, the value of R is incremented by 1. If the target request object is not hit until the preset eviction condition is met, the value of R is set to -1. If the target request object meets the preset eviction condition, the target request object is evicted from the hot object cache, and the triple of the target request object is moved from the buffer to the replay pool.

[0137] When the user request in the state is a shadow request object, that is, when the shadow request object is accessed, the value of R is decremented by 1. If the shadow request object is not accessed until the preset eviction condition is met, the value of R is set to 1. If the shadow request object meets the preset eviction condition, the shadow request object is evicted from the hot object cache, and the triple of the shadow request object is moved from the buffer to the replay pool.

[0138] Step 3.4: When the number of triples in the replay pool reaches a preset threshold, the parameters of the trained temporal model, the parameters of the fully connected layer, and the parameters of the Sigmoid transform are optimized and updated based on the triples in the replay pool, according to the soft actor-critic SAC algorithm.

[0139] This buffer mechanism effectively alleviates the reward delay problem in reinforcement learning, enabling agents to learn cache admission policies more stably, improving the hit rate of hot object caches and overall system performance.

[0140] Step 4: Based on the updated temporal model, fully connected layer, and Sigmoid transformation, calculate the admission decision for the next state, and then determine whether the target request of the user request in the state is allowed to be admitted to the hot object cache of the content delivery network, and then return to execute step 3.

[0141] This invention represents multi-object time series as density-based irregular sequences through a multi-object time encoding mechanism, thereby enabling more accurate modeling of complex dependencies between requesting objects; this invention solves the delayed reward problem in reinforcement learning through buffering, enabling the model to make correct decisions based on historical information; this invention improves the model's decision-making ability and long-term stability through a dual-head model and two-stage training design.

[0142] This invention implements positional encoding of request sequences through a multi-object temporal encoding method, enabling the algorithm to learn the two-level sequence features of the access sequence and significantly improving the model's learning ability; by modeling the agent's decision-making process through a buffer, the model can stably maximize the cumulative reward in the long term, significantly improving indicators such as the hit rate of hot object cache.

[0143] The above description is merely a preferred embodiment of this disclosure and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of the invention involved in the embodiments of this disclosure is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described inventive concept. For example, technical solutions formed by substituting the above-described features with (but not limited to) technical features with similar functions disclosed in the embodiments of this disclosure.

Claims

1. An edge hot object caching admission method, characterized in that, include: Step 1: Obtain the training request and the corresponding historical request. Based on the first information of the training request and the second information of the historical request, train the time series model to obtain the trained time series model. The first information includes the unique identifier of the request object corresponding to the training request, the size of the request object corresponding to the training request, and the initiation time t of the training request. The second information includes the unique identifier of the request object corresponding to the historical request, the size of the request object corresponding to the historical request, and the initiation time of the historical request. The initiation time of all historical requests is represented as {tm,…,t-1}, where m represents the number of historical requests. The temporal model includes feature mapping, multi-object temporal coding, LSTM, and attention network; Step 2: Construct the state space, and pass the states in the state space through the trained temporal model, fully connected layer and Sigmoid transformation in sequence to obtain the admission decision, and then determine whether the target request of the user in the state is allowed to be admitted to the hot object cache of the content delivery network. The states in the state space include the user request information Str, the request object information Sth of the historical requests corresponding to the user request, and the initiation time sequence Ste of the historical requests. Step 2.1: Construct a state space containing multiple states. Each state includes information about the user request (Str), request object information (Sth) of the historical requests corresponding to the user request, and the initiation time sequence of the historical requests. ; Where Str=(id,size, ), where id represents the unique identifier of the request object corresponding to the user request, and size represents the size of the request object corresponding to the user request. Indicates the time when the user request was initiated; in, , Indicates time The unique identifier of the request object in the history of requests. Indicates time The unique identifier of the request object in the history of requests. Indicates time The size of the request object in the history of requests. Indicates time The size of the request object in the history of requests; Step 2.2: Input the states in the state space into the trained time series model to obtain the context information of the states; Step 2.3: Pass the context information of the state through a fully connected layer and a Sigmoid transformation to obtain the admission decision At, which is specifically expressed by the following formula: ； in, Contextual information representing the state; dense indicates a fully connected layer. Step 2.4: When the admission decision At=1, the target request object representing the user's request is allowed to be admitted to the hot object cache of the content delivery network; when the admission decision At=0, the target request object representing the user's request is denied admission to the hot object cache of the content delivery network. Step 3: Set rewards and update rewards based on the states in the state space. Then, based on the states, admission decisions, and rewards, continuously optimize and update the parameters of the trained temporal model, the parameters of the fully connected layers, and the parameters of the Sigmoid transform. Step 4: Based on the updated temporal model, fully connected layer, and Sigmoid transformation, calculate the admission decision for the next state, and then determine whether the target request of the user request in the state is allowed to be admitted to the hot object cache of the content delivery network, and then return to execute step 3.

2. The edge hot object caching admission method according to claim 1, characterized in that, Step 1 specifically includes: Step 1.1: Obtain the first information of the training request, as well as multiple historical requests corresponding to the training request, and obtain the second information of each historical request; Step 1.2: Offline annotation of the second information is performed using the Belady algorithm to obtain reuse distance labels. The reuse distance labels include request distance labels or stack distance labels. The request distance label represents the number of request objects between two requests that return the same request object in response. The stack distance label represents the size distance between two requests that return the same request object in response. All the second information corresponding to the training request and the reuse distance labels are used to form training samples. Multiple training samples are used to form a training sample set. Step 1.3: Input all the second information corresponding to the training requests in the training samples into the time series model to obtain the context information; Step 1.4: Transform the context information through a fully connected layer and Softmax to obtain the predicted reuse distance, specifically expressed by the following formula: D = Softmax(dense(Gt)); Where D represents the predicted reuse distance, dense represents the fully connected layer, and Gt represents the context information; Step 1.5: Based on the predicted reuse distance and reuse distance labels, update the parameters in the time series model, the parameters of the fully connected layer, and the parameters of the Softmax transform. Iterate the parameters of the time series model multiple times based on the training sample set to obtain the trained time series model.

3. The edge hot object caching admission method according to claim 2, characterized in that, Step 1.3 specifically includes: Step 1.3.1: For each piece of second information, perform feature mapping on the unique identifier of the request object and the size of the request object in the second information to obtain the identifier embedding vector and the size embedding vector; Step 1.3.2: Input the identifier embedding vector and size embedding vector of all the second information into the LSTM to obtain the hidden state vector H and the hidden state H at time t. t ; Specifically, the identifier embedding vector and size embedding vector corresponding to time tm are input into the LSTM to obtain the cell state C at time tm. t-m The hidden state H of time tm t-m Specifically, it is expressed by the following formula: (C t-m ,H t-m )=LSTM(E(id t-m );E(size t-m )); Among them, E(id) t-m E(size) represents the identifier embedding vector corresponding to time tm. t-m ) represents the size embedding vector corresponding to time tm; Cellular state C at time tm t-m The identifier embedding vector and size embedding vector corresponding to time t-m+1 are input into the LSTM to obtain the cell state C at time t-m+1. t-m+1 And the hidden state H at time t-m+1 t-m+1 Specifically, it is expressed by the following formula: (C t-m+1 ,H t-m+1 )=LSTM([E(id t-m+1 );E(size t-m+1 )],C t-m ); Among them, E(id) t-m+1 E(size) represents the identifier embedding vector corresponding to time t-m+1. t-m+1 ) represents the size embedding vector corresponding to time t-m+1; Similarly, the cell state from the previous time step, the corresponding identifier embedding vector and size embedding vector from the current time step are input into the LSTM until time t, at which point the hidden state H at time t is obtained from the LSTM output. t Simultaneously, the hidden states from time tm to time t-1 are obtained. These hidden states from time tm to time t-1 form the hidden state vector H, which is specifically represented by the following formula: H =(H t-m ,…,H t-1 ); Step 1.3.3: For each time in {tm,…,t-1} Multi-object time encoding is performed to obtain the time code for each time point, which is represented by the following formula: ； ； ； Where n represents time The corresponding historical request's sequential position within all historical requests, where p represents request density and d... TE Encoding dimension for time, This indicates the scale at position 0. This indicates the scale of position i. Indicates position The scaling factor is L, which represents a predefined time scale constant. Step 1.3.4: Based on the temporal encoding and hidden state vector H at each time step in {tm,…,t-1}, calculate the key vector K and value vector V of the attention network, specifically through the following formula: (K,V)=H+[MO-TE(tm,m,0),..,MO-TE(t-1,m,m)]; Where MO-TE(tm,m,0) represents the time code of time tm, and MO-TE(t-1,m,m) represents the time code of time t-1; Step 1.3.5: H of the hidden state at time t t As the query vector Q, the query vector Q, the key vector K, and the value vector V are input into the attention network to obtain the context information Gt, which is specifically represented by the following formula: Gt = attention(K,Q,V); Here, attention refers to the attention network.

4. The edge hot object caching admission method according to claim 2, characterized in that, Step 1.5 updates the parameters in the time series model, the parameters of the fully connected layer, and the parameters of the Softmax transform, specifically including: The parameters of the feature map, LSTM, multi-object temporal encoding, and attention network are updated.

5. The edge hot object caching admission method according to claim 1, characterized in that, Step 3 specifically includes: Step 3.1: When the admission decision At=1, insert the target request object of the user request into the hot object cache of the content delivery network. When the admission decision At=0, create a shadow request object of the user request. The shadow request object includes the unique identifier and size of the request object corresponding to the user request. Insert the shadow request object into the hot object cache of the content delivery network. Step 3.2: Set up a triple for the target request object or shadow request object. The triple is represented as (S, A, R), where S represents the input state, A represents the admission decision, and R represents the reward. Store the triple in the buffer. Step 3.3: Obtain the next state in the state space. Based on the request object requested by the user in the state, update the reward R of the target request object or shadow request object. If the target request object or shadow request object meets the preset eviction conditions, evict the target request object or shadow request object from the hot object cache and move the triple of the target request object or shadow request object from the buffer to the replay pool. The eviction conditions include the object being requested for the longest time in the hot object cache, or the object being requested for the lowest frequency of being hit in the hot object cache. Step 3.4: When the number of triples in the replay pool reaches a preset threshold, the parameters of the trained temporal model, the parameters of the fully connected layer, and the parameters of the Sigmoid transform are optimized and updated based on the triples in the replay pool, according to the soft actor-critic SAC algorithm.

6. The edge hot object caching admission method according to claim 5, characterized in that, Step 3.3 specifically includes: In the state space, obtain the next state. When the user request object in the state is the target request object, that is, when the target request object is hit, the value of R is incremented by 1. If the target request object is not hit until the preset eviction condition is met, the value of R is set to -1. If the target request object meets the preset eviction condition, the target request object is evicted from the hot object cache, and the triple of the target request object is moved from the buffer to the replay pool. When the user request in the state is a shadow request object, that is, when the shadow request object is accessed, the value of R is decremented by 1. If the shadow request object is not accessed until the preset eviction condition is met, the value of R is set to 1. If the shadow request object meets the preset eviction condition, the shadow request object is evicted from the hot object cache, and the triple of the shadow request object is moved from the buffer to the replay pool.

Images