A multi-level cache method and system based on structure-enhanced prediction and reinforcement learning

Abstract

A multi-level caching method and system based on structural reinforcement prediction and reinforcement learning includes: extracting cacheable data objects, dependencies, and timestamps from logs to construct a data dependency graph; constructing and training a structural reinforcement Transformer model, extracting access sequences from the logs to generate an input matrix, calculating the structural bias of every two data objects in the graph, generating a structural bias matrix, and then inputting the matrix into the Transformer, injecting it into self-attention computation, predicting the popularity of all data objects, and outputting the K most popular data objects; modeling the multi-level cache scheduling problem as a Markov decision process, using cache level, predicted popularity, access intensity, and migration cost to constitute the state space of each candidate data object, and then generating and executing cache migration actions for data objects through a reinforcement learning strategy, optimizing and updating the strategy based on rewards. This invention relates to the field of artificial intelligence technology and can achieve adaptive optimization of multi-level cache scheduling under dynamic loads.

Description

Technical Field

[0001] This invention belongs to the fields of artificial intelligence, cloud computing and big data technology, and specifically relates to a multi-level caching method and system based on structural reinforcement prediction and reinforcement learning. Background Technology

[0002] Distributed caching is a technical architecture that improves system access performance by adding a high-speed caching layer in front of backend storage. It is widely used in big data and high-concurrency scenarios. It achieves data distribution and expansion through sharding and consistent hashing, and often uses strategies such as LRU and LFU for data replacement, combined with expiration control and write-back or write-through mechanisms to ensure data consistency.

[0003] Current distributed caching technologies can be summarized as follows:

[0004] (1) Cache architecture and strategy optimization oriented towards resources and performance, including multi-level cache collaboration, adaptive cache selection based on data size or type, and tag / policy-based classification caching to improve hit rate and reduce cache pollution.

[0005] (2) Enhanced systematic capabilities for engineering implementation, including unified component encapsulation and configuration templates to achieve rapid adaptation of heterogeneous caches, and collaborative optimization combining sharding, prefetching and metadata management, improving overall service quality and scalability through sharding granularity adjustment and caching strategy linkage.

[0006] Patent CN202511136454.X provides an adaptive sharding and caching collaborative optimization method and system for distributed data storage, including: constructing a spatiotemporal joint sharding model based on input data, using time-dimensional processing and correlation quantification; employing a cache prefetch engine to prefetch and cache the sharded data of the spatiotemporal joint sharding model to obtain multi-level cache scheduling data; and adopting a distributed metadata management architecture to adjust the sharding granularity of the spatiotemporal joint sharding model and the caching strategy for generating multi-level cache scheduling data, thereby completing the adaptive sharding and caching collaborative optimization of distributed data storage. This invention employs a designed intelligent cache prefetch engine to reduce frequent access to low-speed storage media during computation, improving cache prefetching efficiency; and uses a distributed metadata management architecture to complete the adaptive sharding and caching collaborative optimization of distributed data storage, eliminating the resource scheduling lag problem caused by the independent operation of sharding and caching in traditional solutions. The prediction model of this scheme is too simple, mainly relying on LSTM+DAG+rule triggering, which makes it difficult to fully characterize complex dependencies, long-tail objects and dynamic contexts. Furthermore, the decision-making mechanism is heuristic, and the prediction results directly serve the prefetching and elimination rules. It lacks adaptive optimization driven by long-term benefits and is also heavily dependent on the sharding system.

[0007] Therefore, how to achieve adaptive optimization of multi-level cache scheduling under dynamic load, so as to be applicable to complex dependency scenarios, has become a key technical issue of concern to technical personnel. Summary of the Invention

[0008] In view of this, the purpose of this invention is to provide a multi-level caching method and system based on structural reinforcement prediction and reinforcement learning, which can achieve adaptive optimization of multi-level cache scheduling under dynamic load, thus making it suitable for complex dependency scenarios.

[0009] To achieve the above objectives, this invention provides a multi-level caching method based on structural enhancement prediction and reinforcement learning, comprising:

[0010] Step 1: Extract cacheable data objects, dependencies between data objects, and access timestamp information from the collected access logs to construct a data dependency graph: ,in, Represents a collection of cacheable data objects. Represents the set of directed dependent edges. Represents a set of timestamps. Represents the set of dependency strength weights;

[0011] Step 2: Build and train a structured augmentation Transformer model to extract ordered access sequences from the access logs. , It is the sequence length. , … They represent the 1st, 2nd, ..., The cacheable data object corresponding to each access will be part of the ordered access sequence. The semantic representations and temporal information of all cacheable data objects are fused into an input matrix. At the same time, calculate The structural bias of each pair of cacheable data objects in the data dependency graph is used to generate a structural bias matrix. Then input matrix and structural bias matrix Inputting a structure-enhanced Transformer model, by injecting the structure bias matrix into the self-attention computation process, predicts... The popularity of all cacheable data objects is calculated, and finally the K cacheable data objects with the highest predicted popularity and their corresponding predicted popularity are output. The K cacheable data objects are the candidate cacheable data objects.

[0012] Step 3: Model the multi-level cache scheduling problem as a Markov decision process. The cache level, predicted popularity, access intensity, and migration cost constitute the state space of each candidate cacheable data object. Then, the cache migration actions of all candidate cacheable data objects are generated and executed through a reinforcement learning strategy. Finally, the reinforcement learning strategy is optimized and updated based on the reward value.

[0013] To achieve the above objectives, the present invention also provides a multi-level caching system based on structural enhancement prediction and reinforcement learning, comprising:

[0014] A dependency graph construction device is used to extract cacheable data objects, dependencies between data objects, and access timestamp information from collected access logs to construct a data dependency graph. ,in, Represents a collection of cacheable data objects. Represents the set of directed dependent edges. Represents a set of timestamps. Represents the set of dependency strength weights;

[0015] A heat prediction device is used to build and train a structurally augmented Transformer model to extract ordered access sequences from access logs. , It is the sequence length. , … They represent the 1st, 2nd, ..., The cacheable data object corresponding to each access will be part of the ordered access sequence. The semantic representations and temporal information of all cacheable data objects are fused into an input matrix. At the same time, calculate The structural bias of each pair of cacheable data objects in the data dependency graph is used to generate a structural bias matrix. Then input matrix and structural bias matrix Inputting a structure-enhanced Transformer model, by injecting the structure bias matrix into the self-attention computation process, predicts... The popularity of all cacheable data objects is calculated, and finally the K cacheable data objects with the highest predicted popularity and their corresponding predicted popularity are output. The K cacheable data objects are the candidate cacheable data objects.

[0016] The reinforcement learning device is used to model the multi-level cache scheduling problem as a Markov decision process. The cache level, predicted popularity, access intensity and migration cost constitute the state space of each candidate cacheable data object. Then, the cache migration actions of all candidate cacheable data objects are generated and executed through the reinforcement learning policy. Finally, the reinforcement learning policy is optimized and updated based on the reward value.

[0017] To achieve the above objectives, the present invention also provides a computing device, comprising:

[0018] Memory and processor;

[0019] The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions. When the computer-executable instructions are executed by the processor, they implement the steps of the multi-level caching method based on structural reinforcement prediction and reinforcement learning.

[0020] To achieve the above objectives, the present invention also provides a computer-readable storage medium storing computer-executable instructions that, when executed by a processor, implement the steps of the multi-level caching method based on structural reinforcement prediction and reinforcement learning.

[0021] Compared with existing technologies, the beneficial effects of this invention are as follows: This invention uses a structure-enhanced temporal Transformer model to globally model data access sequences, which can characterize complex dependencies and long-term temporal features, enabling forward-looking prediction of potential hot data. Simultaneously, it models the cache scheduling process as a reinforcement learning decision problem, performing adaptive hierarchical adjustments and resource allocation based on the prediction results, forming a prediction-scheduling closed-loop optimization mechanism. Compared with rule-based or static strategy-based solutions, this invention possesses dynamic self-learning capabilities, adapting to load fluctuations and business changes, improving cache hit rate and resource utilization efficiency. Furthermore, by injecting dependency structure information into the attention mechanism, this invention eliminates reliance on single access frequency statistics for hot data prediction. The reinforcement learning scheduling mechanism of this invention solves the long-term benefit optimization problem of inter-layer migration, maintaining stable performance improvement under load changes, data pattern evolution, and cache size adjustments. Attached Figure Description

[0022] Figure 1 This is a flowchart illustrating a multi-level caching method based on structural enhancement prediction and reinforcement learning, as shown in an exemplary embodiment of the present invention.

[0023] Figure 2 This is an exemplary embodiment of the present invention. Figure 1 The flowchart for the specific steps in step three.

[0024] Figure 3This is a schematic diagram of a multi-level caching system based on structural enhancement prediction and reinforcement learning, as illustrated in an exemplary embodiment of the present invention.

[0025] Figure 4 This is a schematic diagram of the structure of a computer device shown in an exemplary embodiment of the present invention. Detailed Implementation

[0026] To make the objectives, technical solutions, and advantages of the present invention clearer, the present invention will be further described in detail below with reference to the accompanying drawings.

[0027] In distributed data warehouses and large-scale analytical workloads, cache hit rate and end-to-end access latency depend not only on the local access frequency of objects but also on the synergistic effect of data lineage and computational dependencies. Typically, access to a triggering object (such as a fact table or key intermediate results) can induce a cascading read of a group of dependent objects within a short period. If the scheduling strategy relies solely on sliding window frequency statistics or rule-based replacement mechanisms, the system often completes migration and prefetching only after hotspots have formed, resulting in significant response lag and amplified migration overhead. To address these issues, this invention constructs a closed-loop system in the control plane—"structure enhancement prediction - strategy decision - migration execution - performance feedback"—without altering the cache data plane read / write path and consistency control. It utilizes a structure-enhanced temporal Transformer model (Chinese name: deep neural network model based on self-attention mechanism) to learn access intentions and output heat distribution. Simultaneously, the prediction results, cache water level, and migration cost are incorporated into a reinforcement learning (RL) scheduler to optimize long-term benefits of cross-level migration. This design enables the cache system to possess forward-looking scheduling capabilities and achieves adaptive convergence through online feedback.

[0028] like Figure 1 As shown, this invention relates to a multi-level caching method based on structural enhancement prediction and reinforcement learning, comprising:

[0029] Step 1: Extract cacheable data objects, dependencies between data objects, and access timestamp information from the collected access logs to construct a data dependency graph: ,in, This represents a collection of cacheable data objects (such as tables, partitions, files, object blocks, etc.). Represents the set of directed dependent edges. Represents a set of timestamps. Represents the set of dependency strength weights;

[0030] right any edge in : , Indicates at time What happened (Right now arrive The dependency relationship is defined by the dependency strength weight. , The value can be calculated from the frequency of references, job scheduling frequency, or the intensity of data change propagation, and time decay is introduced to reduce the sticky effect of historical structure on current decisions;

[0031] Step 2: Build and train a structured augmentation Transformer model to extract ordered access sequences from the access logs. , It is the sequence length. , … They represent the 1st, 2nd, ..., The cacheable data object corresponding to each access will be part of the ordered access sequence. The semantic representations and temporal information of all cacheable data objects are fused into an input matrix. At the same time, calculate The structural bias of each pair of cacheable data objects in the data dependency graph is used to generate a structural bias matrix. Then input matrix and structural bias matrix Inputting a structure-enhanced Transformer model, by injecting the structure bias matrix into the self-attention computation process, predicts... The popularity of all cacheable data objects is calculated, and finally the K cacheable data objects with the highest predicted popularity and their corresponding predicted popularity are output. K is a preset positive integer, and the K cacheable data objects are the candidate cacheable data objects.

[0032] Step 3: Model the multi-level cache scheduling problem as a Markov Decision Process (MDP). Construct the state space of each candidate cacheable data object by considering cache level, predicted popularity, access intensity, and migration cost. Then, generate and execute cache migration actions for all candidate cacheable data objects through a reinforcement learning strategy. Finally, optimize and update the reinforcement learning strategy based on the reward value. The cache levels can be set as L1 (memory), L2 (SSD), L3 (HDD / object storage), etc.

[0033] The data dependency extraction in step one can reuse the existing metadata management module in the distributed caching system and data warehouse architecture. Extracting cacheable data objects, dependencies between data objects, and access timestamp information from the collected access logs can further include:

[0034] For SQL logs, a parser is used to construct an Abstract Syntax Tree (AST), and the dependencies between cacheable data objects are extracted from projection, join, subquery, and write statements; for ETL jobs, the input and output edges are directly parsed into the Directed Acyclic Graph (DAG); and existing data lineage systems are synchronized in batches through an interface.

[0035] The data dependency graph is stored using a graph database to support path lookup, or an adjacency list key-value structure can be used to improve online computation efficiency. To meet the prediction latency requirements, the induced subgraph of the nodes involved in the session is cached, and the affected edge weights and timestamps are updated incrementally.

[0036] Figure 1 In step two, since the data dependency graph itself is not a tensor structure that can be directly input into a temporal Transformer, this invention adopts a structure bias injection method to inject any object pair within the sequence. Structural correlation is encoded as a bias term This is then incorporated into the attention scoring calculation, enabling the model to explicitly perceive lineage topology while learning temporal dependencies. Step two further includes:

[0037] Step 21: Aggregate the raw access logs according to user, session, or preset time window to form an ordered access sequence. ;

[0038] Step 22, according to The semantic representation and temporal information of each cacheable data object in the dataset are used to generate the embedding vector and position vector for each cacheable data object, and then... The embedding vectors and position vectors of all cacheable data objects are fused into the input matrix. ;

[0039] Step 23, Calculation The structural bias of each pair of cacheable data objects in the data dependency graph is used to generate a structural bias matrix. : ,in, yes The first in Line number Column elements are also known as cacheable data objects. and cacheable data objects Structural bias between them It is a data dependency graph The shortest path length, yes Dependency strength weight, It is the time decay factor. It is the attenuation coefficient. It is a data dependency graph The time interval between the latest update time of the corresponding directed edge and the current time;

[0040] In the structural information modeling stage, considering that the data dependency graph itself is not a regular tensor structure, this invention does not directly use a graph neural network to perform multiple rounds of propagation on the entire graph. Instead, it focuses on the access sequence in the current prediction window. Dynamically construct the structure bias matrix To explicitly encode the lineage topology between cacheable data object pairs, As a batch-level additional input, it follows the input matrix. By feeding them together into the model inference, this strategy avoids the multi-round neighborhood propagation of graph neural networks, reduces online computational complexity, and retains the Transformer's global modeling ability for long-range dependencies.

[0041] Step 24: Input matrix and structural bias matrix Input structure enhancement for Transformer models: During the attention computation phase, the input matrix is... The query matrix is ​​obtained through three sets of learnable linear projections. Key matrix Sum matrix : , , , , , The trainable attention projection parameter matrix is ​​then used. Finally, based on the standard attention mechanism, the structural bias matrix is... By injecting an additive bias into the attention scoring stage, structure-enhanced attention is obtained. , This represents the original attention score matrix obtained based on temporal context learning. It is the structural bias from the data dependency graph; both have the same dimension. It is the dimension of the key vector, used to scale the dot product attention, and finally calculated. The predicted popularity of each cacheable data object is calculated, and based on this, the K cacheable data objects with the highest predicted popularity and their predicted popularity are selected and output.

[0042] If a set of cacheable data object pairs are closer in distance, have stronger dependencies, or are updated more recently on the data dependency graph, then The larger the value, the higher the attention score at the corresponding location. Before normalization, the bias is explicitly raised, so that while the model learns the temporal relevance of access, it tends to focus more on cacheable data object pairs that are truly related in the topology. Conversely, for cacheable data object pairs with long paths, weak correlations, or outdated dependencies, the bias is smaller or even suppressed, thus avoiding attention being interfered with by irrelevant structural noise.

[0043] In training the structurally enhanced Transformer model, this invention can also use next-visit prediction as the primary supervision signal: , It is the loss value of the access prediction task. It is a moment An ordered access sequence, It is a moment Access to cacheable data objects, predicting probabilities Represented by the session After linear transformation get: , It is the weight matrix of a linear output layer. It is The session representation is obtained by pooling the sequence representation H after structural enhancement Transformer encoding;

[0044] Meanwhile, this invention also introduces contrastive learning to enhance the semantic discriminativeness of long-tail objects: object pairs that co-occur in the same conversation and are structurally strongly related are used as positive samples, while objects that are structurally distant or weakly related are used as negative samples. The embedding space is constrained by the Information Noise-Contrastive Estimation (InfoNCE) loss, and the contrastive learning loss value is calculated. : ,in, Represents a cacheable data object The representation vector, Indicates and The representation vector of the cacheable data object that constitutes a positive sample pair. Let represent the representation vector of the k-th candidate object in the comparison set. This represents a temperature coefficient, used to adjust the smoothness of the similarity distribution. This represents the index of the candidate sample in the denominator that participates in the normalization calculation;

[0045] Finally, calculate the overall loss: , It is a loss balancing coefficient used to control the loss value of the access prediction task. Comparative learning loss value The relative contribution to the overall optimization objective.

[0046] This invention outputs Top-K hotspot candidates and their predicted popularity through online inference, driving subsequent policy decisions rather than directly executing transfers. Considering that access logs exhibit a significant long-tail distribution in real-world scenarios, the representation learning of low-frequency objects is easily affected by noise and sparsity. This mechanism provides additional sample construction and comparison heads during the training phase, without altering the online inference path, but significantly improves the stability of prediction confidence, thereby reducing the exploration risk and false transfer probability of the scheduling policy.

[0047] Step 22 may further include:

[0048] Step 221, according to The semantic representation of each cacheable data object in the graph is used to generate an embedding vector for each cacheable data object: for the data dependency graph. A collection of cacheable data objects Each cacheable data object in the dataset is assigned a unique index, and then processed through trainable parameters. This maps all discrete, cacheable data objects to continuous vectors. It is a trainable embedding matrix initialized for a set of cacheable data objects, whose parameters can be optimized during the training process. The embedding vector of each cacheable data object in the data is obtained from... Extracted from the data, where d is the vector dimension of the cacheable data object. express The number of cacheable data objects in the database;

[0049] Step 222, according to For each cacheable data object in the dataset, the temporal information is used to generate a location vector for each cacheable data object: to preserve... Information on the access order of all cacheable data objects, and setting a position encoding matrix. , The row vector yes The location vector of the cacheable data object accessed for the i-th time;

[0050] Step 223, Generate The input vector for each cacheable data object: , It is the first The input vector of the cacheable data object accessed next time will The input vectors of all accessed cacheable data objects are stacked in the order of access to obtain the input matrix. : .

[0051] The row dimension corresponds to the position in the access sequence, and the column dimension corresponds to the representation dimension of each cacheable data object in the semantic space. In other words, the input matrix... Essentially, it is the result of fusing the semantic representation of cacheable data objects with sequence position information, used to characterize the temporal access context of the current session.

[0052] like Figure 2 As shown, Figure 1 Step three may further include:

[0053] Step 31: Model the multi-level cache scheduling problem as a Markov decision process, and generate a cache state vector for each candidate cacheable data object at the current time: ={ , , , }, It is a candidate cacheable data object The state vector at time t, It is a candidate cacheable data object The cache water level of the cache layer at time t. It is a candidate cacheable data object The predicted popularity at time t It is a candidate cacheable data object The intensity of access at time t, It is a candidate cacheable data object The migration cost at time t;

[0054] In step 31, the predicted popularity comes from the output of step two; the cache level comes from the cache monitoring module and resource statistics interface, directly affecting the feasibility of migration actions and replacement pressure. This indicates the resource usage status of the caching layer at time t, reflecting the remaining capacity and congestion level of different layers. If a candidate cacheable data object is selected... Cache layer The total capacity of the cache is The used capacity at time t is The formula for calculating its buffer water level is as follows: Access intensity is used to characterize the request pressure currently faced by the system, reflecting the intensity of object access, arrival rate, and request distribution per unit time. The calculation formula is as follows: ,in For the interval [t, t+ Candidate cacheable data objects arriving within ] The number of access requests, derived from the cache request monitoring interface, is a direct observation of the system's current workload and is used to determine whether migration, prefetching, or protection actions are appropriate. Configured based on actual business needs; migration costs This represents the cost of migrating objects between different caching layers. It includes both the I / O and network overhead of the data movement itself, as well as the latency disturbances and resource consumption that may be introduced during the migration process. It can be determined by the size of the candidate cacheable data objects and the inter-layer migration rate. The calculation yielded: , Candidate cacheable data objects Size, This indicates other time delay disturbances and delays;

[0055] Step 32: Generate corresponding scheduling actions for each candidate cacheable data object's cache state vector at the current time using a reinforcement learning strategy, and form a scheduling action set by the scheduling actions of all candidate cacheable data objects: , It is the set of scheduling actions at time t, including scheduling actions such as Promote, Demote, Retain, Evict, and Pin. Each candidate cacheable data object corresponds to a scheduling action. Promote / Demote indicates the inter-layer migration direction. Promote and Demote represent migration up and migration down, respectively. Retain means to maintain the resident layer. Evict means to allow replacement or eviction. Pin means short-term resident protection.

[0056] Step 33: Convert the cache state vector of each candidate cacheable data object into a priority attribute of the candidate cacheable data object, and record the generated scheduling action in the candidate cacheable data object attribute. Then, the scheduling plugin executes the scheduling actions of all candidate cacheable data objects in priority order according to the corresponding attributes. The scheduling plugin is responsible for the data transfer and replacement strategy integration. The calculation formula for the priority attribute of each candidate cacheable data object is as follows: ,in, It is a candidate cacheable data object Priority attribute , , These are weighting coefficients used to adjust the relative contributions of the three types of factors in object priority calculation. Indicated based on reward function With long-term return optimization goals The learned object-level value estimate is used to characterize the long-term cumulative benefit corresponding to performing scheduling operations such as retention, prefetching, and promotion on candidate cacheable data objects in the current cache state. The specific calculation formula can be: That is, it is represented as the scheduling action generated given the current state and policy. The expected reward of the system's future returns under the given conditions. It is the immediate reward obtained after performing the action at time t. This represents the immediate reward that can be obtained after the scheduling action is completed at time t+l in the future. This indicates the discount weight of the reward at future time t+l. This indicates the calculation of the expected value.

[0057] The larger, the more it means The higher the priority, the higher the cache should be cached and prefetched to the higher-level cache. Due to the simultaneous access of data by multiple tenants and multiple processes, in order to avoid the oscillation between cache layers caused by the rapid heating and cooling of local cache data, step 33 records the priority and action in the attribute, and then lets the scheduling plugin execute the scheduling action according to the priority and queue order, transforming the hard control of cache prediction scheduling into a soft drive.

[0058] Step 34: Collect monitoring indicators and calculate reward values: , It is the immediate reward obtained after performing the action at time t. This represents the overall cache hit rate at time t, used to measure the impact of all current scheduling actions on cache hit benefits. This represents the end-to-end service latency at time t, used to characterize the impact of all current scheduling actions on response performance. This represents the migration cost of all candidate cacheable data objects at time t. , , These are weighting coefficients used to adjust the relative contributions of the three types of indicators in the reward function, and to optimize and update the reinforcement learning strategy based on the reward. The strategy optimization objective is set as follows: The aim is to adjust the strategy network parameters. This maximizes the expected cumulative reward from the discounts received throughout the entire scheduling process. This represents the expected value of long-term returns under the randomness of strategy and environment. (0,1) represents the discount weight corresponding to the expected reward at time t, used to balance current and future returns, thereby forming a continuously converging closed-loop control. This means calculating the mathematical expectation, which is to average the randomness of long-term returns during strategy execution and environmental evolution.

[0059] The reward function establishes a trade-off between the hit reward and the execution cost.

[0060] In scenarios involving multiple business units or multiple data centers, raw access logs may be restricted by compliance and cannot be centralized. This invention can also employ federated learning to achieve cross-domain collaborative modeling. After each site completes the construction of the data dependency graph, generation of the structural bias matrix, and model training locally, it uploads the model parameters or gradients to the aggregation server. The aggregation server performs weighted aggregation and publishes the global model, which is then received by each site for local inference and scheduling. Scheduling operations are still performed locally, avoiding cross-domain data flow and object migration.

[0061] In summary, this invention explicitly integrates data dependency topology into the attention computation of the temporal Transformer using a structural bias injection approach, and enhances the stability of long-tail object representations through contrastive learning. Furthermore, it incorporates the prediction results and cache runtime state into a reinforcement learning scheduling framework, forming a closed-loop control system of prediction-decision-execution-feedback. This system maintains low invasiveness to the cached data plane in engineering, supports multi-site federated expansion and lightweight deployment, and is suitable for distributed data access scenarios with high concurrency, significant dependencies, and dynamically evolving loads.

[0062] like Figure 3 As shown, this invention relates to a multi-level caching system based on structural reinforcement prediction and reinforcement learning, comprising:

[0063] A dependency graph construction device is used to extract cacheable data objects, dependencies between data objects, and access timestamp information from collected access logs to construct a data dependency graph. ,in, This represents a collection of cacheable data objects (such as tables, partitions, files, object blocks, etc.). Represents the set of directed dependent edges. Represents a set of timestamps. Represents the set of dependency strength weights;

[0064] A heat prediction device is used to build and train a structurally augmented Transformer model to extract ordered access sequences from access logs. , It is the sequence length. , … They represent the 1st, 2nd, ..., The cacheable data object corresponding to each access will be part of the ordered access sequence. The semantic representations and temporal information of all cacheable data objects are fused into an input matrix. At the same time, calculate The structural bias of each pair of cacheable data objects in the data dependency graph is used to generate a structural bias matrix. Then input matrix and structural bias matrix Inputting a structure-enhanced Transformer model, by injecting the structure bias matrix into the self-attention computation process, predicts... The popularity of all cacheable data objects is calculated, and finally the K cacheable data objects with the highest predicted popularity and their corresponding predicted popularity are output. K is a preset positive integer, and the K cacheable data objects are the candidate cacheable data objects.

[0065] The reinforcement learning device is used to model the multi-level cache scheduling problem as a Markov decision process. The cache level, predicted popularity, access intensity and migration cost constitute the state space of each candidate cacheable data object. Then, the cache migration actions of all candidate cacheable data objects are generated and executed through the reinforcement learning policy. Finally, the reinforcement learning policy is optimized and updated according to the reward value. The cache level can be set as L1 (memory), L2 (SSD), L3 (HDD / object storage), etc.

[0066] See Figure 4 , Figure 4 This is a structural block diagram of a computing device 400 illustrated in an exemplary embodiment of this specification. The components of the computing device 400 include, but are not limited to, a memory 410 and a processor 420. The processor 420 is connected to the memory 410 via a bus 430, and a database 450 is used to store data.

[0067] The computing device 400 also includes an access device 440, which enables the computing device 400 to communicate via one or more networks 460. Examples of these networks include Public Switched Telephone Network (PSTN), Local Area Network (LAN), Wide Area Network (WAN), Personal Area Network (PAN), or combinations of communication networks such as the Internet. The access device 440 may include one or more of any type of wired or wireless network interface (e.g., Network Interface Card (NIC)), such as an IEEE 802.11 Wireless Local Area Network (WLAN) interface, a Wi-MAX (Worldwide Interoperability for Microwave Access) interface, an Ethernet interface, a Universal Serial Bus (USB) interface, a cellular network interface, a Bluetooth interface, a Near Field Communication (NFC) interface, and so on.

[0068] In one embodiment of this specification, the aforementioned components of the computing device 400 and Figure 4 Other components, not shown, can also be connected to each other, for example, via a bus. It should be understood that... Figure 4 The block diagram of the computing device shown is for illustrative purposes only and is not intended to limit the scope of this specification. Those skilled in the art can add or replace other components as needed.

[0069] The computing device 400 can be any type of stationary or mobile computing device, including mobile computers or mobile computing devices (e.g., tablet computers, personal digital assistants, laptop computers, notebook computers, netbooks, etc.), mobile phones (e.g., smartphones), wearable computing devices (e.g., smartwatches, smart glasses, etc.) or other types of mobile devices, or stationary computing devices such as desktop computers or PCs. The computing device 400 can also be a mobile or stationary server or cloud server, etc.

[0070] The processor 420 is configured to execute the following computer-executable instructions, which, when executed by the processor, implement the steps of the multi-level caching method based on structural reinforcement prediction and reinforcement learning described above.

[0071] The above is an illustrative scheme of a computing device according to this embodiment. It should be noted that the technical solution of this computing device belongs to the same concept as the technical solution of the multi-level caching method based on structural enhancement prediction and reinforcement learning described above. For details not described in detail in the technical solution of the computing device, please refer to the description of the technical solution of the multi-level caching method based on structural enhancement prediction and reinforcement learning described above.

[0072] An embodiment of this specification also provides a computer-readable storage medium storing computer-executable instructions that, when executed by a processor, implement the steps of the multi-level caching method based on structural reinforcement prediction and reinforcement learning described above.

[0073] The above is an illustrative scheme of a computer-readable storage medium according to this embodiment. It should be noted that the technical solution of this storage medium belongs to the same concept as the multi-level caching method based on structure enhancement prediction and reinforcement learning described above. For details not described in detail in the technical solution of the storage medium, please refer to the description of the technical solution of the multi-level caching method or system based on structure enhancement prediction and reinforcement learning described above.

[0074] An embodiment of this specification also provides a computer program, wherein when the computer program is executed in a computer, it causes the computer to perform the steps of the multi-level caching method based on structural reinforcement prediction and reinforcement learning described above.

[0075] The above is an illustrative example of a computer program in this embodiment. It should be noted that the technical solution of this computer program belongs to the same concept as the technical solution of the multi-level caching method based on structural reinforcement prediction and reinforcement learning described above. Details not described in detail in the computer program's technical solution can be found in the description of the technical solution of the multi-level caching method or system based on structural reinforcement prediction and reinforcement learning described above.

[0076] The foregoing has described specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired result. In some embodiments, multitasking and parallel processing are possible or may be advantageous.

[0077] The computer instructions include computer program code, which may be in the form of source code, object code, executable file, or some intermediate form. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording media, USB flash drive, portable hard drive, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc.

[0078] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that the embodiments in this specification are not limited to the described order of actions, because according to the embodiments in this specification, some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in this specification are all preferred embodiments, and the actions and modules involved are not necessarily essential to the embodiments in this specification.

[0079] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A multi-level caching method based on structural reinforcement prediction and reinforcement learning, characterized in that, Including: Step 1: Extract cacheable data objects, dependencies between data objects, and access timestamp information from the collected access logs to construct a data dependency graph: ,in, Represents a collection of cacheable data objects. Represents the set of directed dependent edges. Represents a set of timestamps. Represents the set of dependency strength weights; Step 2: Build and train a structured augmentation Transformer model to extract ordered access sequences from the access logs. , It is the sequence length. , … They represent the 1st, 2nd, ..., The cacheable data object corresponding to each access will be part of the ordered access sequence. The semantic representations and temporal information of all cacheable data objects are fused into an input matrix. At the same time, calculate The structural bias of each pair of cacheable data objects in the data dependency graph is used to generate a structural bias matrix. Then input matrix and structural bias matrix Inputting a structure-enhanced Transformer model, by injecting the structure bias matrix into the self-attention computation process, predicts... The popularity of all cacheable data objects is calculated, and finally the K cacheable data objects with the highest predicted popularity and their corresponding predicted popularity are output. The K cacheable data objects are the candidate cacheable data objects. Step 3: Model the multi-level cache scheduling problem as a Markov decision process. The cache level, predicted popularity, access intensity, and migration cost constitute the state space of each candidate cacheable data object. Then, the cache migration actions of all candidate cacheable data objects are generated and executed through a reinforcement learning strategy. Finally, the reinforcement learning strategy is optimized and updated based on the reward value.

2. The method according to claim 1, characterized in that, Extract cacheable data objects, dependencies between data objects, and access timestamp information from the collected access logs, including: An abstract syntax tree is constructed using a parser for SQL logs, and the dependencies between cacheable data objects are extracted from projection, join, subquery, and write statements; input and output edges are directly parsed for directed acyclic graphs of ETL jobs; and existing data lineage systems are synchronized in batches via interfaces.

3. The method according to claim 1, characterized in that, Step two includes: Step 21: Aggregate the raw access logs according to user, session, or preset time window to form an ordered access sequence. ; Step 22, according to The semantic representation and temporal information of each cacheable data object in the dataset are used to generate the embedding vector and position vector for each cacheable data object, and then... The embedding vectors and position vectors of all cacheable data objects are fused into the input matrix. ; Step 23, Calculation The structural bias of each pair of cacheable data objects in the data dependency graph is used to generate a structural bias matrix. : ,in, yes The first in Line number Column elements are also known as cacheable data objects. and cacheable data objects Structural bias between them It is a data dependency graph The shortest path length, yes Dependency strength weight, It is the time decay factor. It is the attenuation coefficient. It is a data dependency graph The time interval between the latest update time of the corresponding directed edge and the current time; Step 24: Input matrix and structural bias matrix Input structure enhancement for Transformer models: During the attention computation phase, the input matrix is... The query matrix is ​​obtained through three sets of learnable linear projections. Key matrix Sum matrix : , , , , , The trainable attention projection parameter matrix is ​​then used, followed by the structure bias matrix. By injecting an additive bias into the attention scoring stage, structure-enhanced attention is obtained. , It is the dimension of the key vector, which is calculated in the end. The predicted popularity of each cacheable data object is calculated, and based on this, the K cacheable data objects with the highest predicted popularity and their predicted popularity are selected and output.

4. The method according to claim 3, characterized in that, Step 22 includes: Step 221, according to The semantic representation of each cacheable data object in the graph is used to generate an embedding vector for each cacheable data object: for the data dependency graph. A collection of cacheable data objects Each cacheable data object in the dataset is assigned a unique index, and then processed through trainable parameters. This maps all discrete, cacheable data objects to continuous vectors. It is a trainable embedding matrix initialized for a set of cacheable data objects. The embedding vector of each cacheable data object in the data is from Extracted from the data, where d is the vector dimension of the cacheable data object. express The number of cacheable data objects in the database; Step 222, according to For each cacheable data object, the timing information is used to generate a location vector for each cacheable data object: a location encoding matrix is ​​set. , The row vector yes The location vector of the cacheable data object accessed for the i-th time; Step 223, Generate The input vector for each cacheable data object: , It is the first The input vector of the cacheable data object accessed next time will The input vectors of all accessed cacheable data objects are stacked in the order of access to obtain the input matrix. : .

5. The method according to claim 1, characterized in that, When training the structurally enhanced Transformer model, the next visit prediction is used as a supervision signal: , It is the loss value of the access prediction task. It is a moment An ordered access sequence, It is a moment Access to cacheable data objects, predicting probabilities Represented by the session After linear transformation get: , It is the weight matrix of a linear output layer. It is The session representation is obtained by pooling the sequence representation H after structural enhancement Transformer encoding; Simultaneously, contrastive learning is introduced: object pairs that co-occur in the same conversation and have a strong structural association are used as positive samples, while objects with a large structural distance or weak association are used as negative samples. The embedding space is constrained by noise contrast estimation loss, and the contrastive learning loss value is calculated. : ,in, Represents a cacheable data object The representation vector, Indicates and The representation vector of the cacheable data object that constitutes a positive sample pair. Let represent the representation vector of the k-th candidate object in the comparison set. Indicates the temperature coefficient. This represents the index of the candidate sample in the denominator that participates in the normalization calculation; Finally, calculate the overall loss: , It is the loss balance coefficient.

6. The method according to claim 1, characterized in that, Step three includes: Step 31: Model the multi-level cache scheduling problem as a Markov decision process, and generate a cache state vector for each candidate cacheable data object at the current time: ={ , , , }, It is a candidate cacheable data object The state vector at time t, It is a candidate cacheable data object The cache water level of the cache layer at time t. It is a candidate cacheable data object The predicted popularity at time t It is a candidate cacheable data object The intensity of access at time t, It is a candidate cacheable data object The migration cost at time t; Step 32: Generate corresponding scheduling actions for each candidate cacheable data object's cache state vector at the current time using a reinforcement learning strategy, and form a scheduling action set by the scheduling actions of all candidate cacheable data objects: , It is the set of scheduling actions at time t, including Promote, Demote, Retain, Evict, and Pin scheduling actions. Promote and Demote represent the inter-layer migration direction, respectively. Promote and Demote represent migration up and migration down. Retain means maintaining the resident layer. Evict means allowing replacement or eviction. Pin means short-term resident protection. Step 33: Convert the cache state vector of each candidate cacheable data object into a priority attribute of the candidate cacheable data object, and record the generated scheduling action in the candidate cacheable data object attribute. Then, the scheduling plugin executes the scheduling actions of all candidate cacheable data objects in priority order according to the corresponding attributes. The scheduling plugin is responsible for the data transfer and replacement strategy integration. The calculation formula for the priority attribute of each candidate cacheable data object is as follows: ,in, It is a candidate cacheable data object Priority attribute , , These are weighting coefficients used to adjust the relative contributions of the three types of factors in object priority calculation. Indicated based on reward function With long-term return optimization goals The learned object-level value estimation is calculated using the following formula: That is, it is represented as the scheduling action generated given the current state and policy. The expected reward of the system's future returns under the given conditions. It is the immediate reward obtained after performing the action at time t. This represents the immediate reward that can be obtained after the scheduling action is completed at time t+l in the future. This indicates the discount weight of the reward at future time t+l. This indicates the calculation of the expected value. Step 34: Collect monitoring indicators and calculate reward values: , It is the immediate reward obtained after performing the action at time t. This represents the overall cache hit rate at time t. This represents the end-to-end service delay at time t. This represents the migration cost of all candidate cacheable data objects at time t. , , These are weighting coefficients used to adjust the relative contributions of the three types of indicators in the reward function, and to optimize and update the reinforcement learning strategy based on the reward. The strategy optimization objective is set as follows: The aim is to adjust the strategy network parameters. This maximizes the expected cumulative reward from the discounts received throughout the entire scheduling process. This represents the expected value of long-term returns under the randomness of strategy and environment. (0,1) represents the discount weight corresponding to the expected reward at time t. This indicates the calculation of mathematical expectation.

7. The method according to claim 6, characterized in that, In step 31, if a candidate cacheable data object... Cache layer The total capacity of the cache is The used capacity at time t is The formula for calculating the buffer water level is as follows: ; The formula for calculating access strength is as follows: ,in For the interval [t, t+ Candidate cacheable data objects arriving within ] The number of access requests; Migration cost is determined by the size of the candidate cacheable data object and the inter-layer migration rate. The calculation yielded: , Candidate cacheable data objects Size, This indicates other time delay disturbances and delays.

8. The method according to claim 1, characterized in that, Federated learning is used to achieve cross-domain collaborative modeling. After the site completes the construction of data dependency graph, generation of structural bias matrix and model training locally, it uploads the model parameters or gradients to the aggregation server. The aggregation server performs weighted aggregation and publishes the global model, which is then received by each site for local inference and scheduling.

9. A multi-level caching system based on structural reinforcement prediction and reinforcement learning, characterized in that, Including: A dependency graph construction device is used to extract cacheable data objects, dependencies between data objects, and access timestamp information from collected access logs to construct a data dependency graph. ,in, Represents a collection of cacheable data objects. Represents the set of directed dependent edges. Represents a set of timestamps. Represents the set of dependency strength weights; A heat prediction device is used to build and train a structurally augmented Transformer model to extract ordered access sequences from access logs. , It is the sequence length. , … They represent the 1st, 2nd, ..., The cacheable data object corresponding to each access will be part of the ordered access sequence. The semantic representations and temporal information of all cacheable data objects are fused into an input matrix. At the same time, calculate The structural bias of each pair of cacheable data objects in the data dependency graph is used to generate a structural bias matrix. Then input matrix and structural bias matrix Inputting a structure-enhanced Transformer model, by injecting the structure bias matrix into the self-attention computation process, predicts... The popularity of all cacheable data objects is calculated, and finally the K cacheable data objects with the highest predicted popularity and their corresponding predicted popularity are output. The K cacheable data objects are the candidate cacheable data objects. The reinforcement learning device is used to model the multi-level cache scheduling problem as a Markov decision process. The cache level, predicted popularity, access intensity and migration cost constitute the state space of each candidate cacheable data object. Then, the cache migration actions of all candidate cacheable data objects are generated and executed through the reinforcement learning policy. Finally, the reinforcement learning policy is optimized and updated based on the reward value.

10. A computing device, characterized in that, include: Memory and processor; The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions. When the computer-executable instructions are executed by the processor, they implement the steps of the multi-level caching method based on structural enhancement prediction and reinforcement learning as described in any one of claims 1-8.

11. A computer-readable storage medium, characterized in that, It stores computer-executable instructions, which, when executed by a processor, implement the steps of the multi-level caching method based on structural enhancement prediction and reinforcement learning as described in any one of claims 1-8.

Images