Data caching method, data caching system and device, electronic equipment and medium

Abstract

The invention provides a data caching method, a data caching system, a data caching device, electronic equipment and a medium, and relates to the technical field of computers, the method is applied to an intelligent agent, and the method comprises the following steps: determining multiple pieces of corresponding first action information based on multiple pieces of pre-collected first state information; after a first caching strategy configured based on the first action information is applied to the back-end environment, multiple pieces of second state information used for representing the back-end environment are obtained; determining a plurality of pieces of reward information corresponding to the plurality of pieces of second state information, wherein the reward information is used for evaluating the execution effect of the first caching strategy; updating a reinforcement learning network in the intelligent agent according to the first state information, the first action information, the second state information and the reward information; and acquiring third state information of the back-end environment, and determining target action information corresponding to the third state information based on the updated reinforcement learning network to indicate execution of a corresponding caching strategy in the back-end environment. According to the invention, efficient and refined configuration of the caching strategy can be realized.

Description

Technical Field

[0001] This disclosure relates to the field of computer technology, specifically to a data caching method, data caching system, apparatus, electronic device, and medium. Background Technology

[0002] In computer systems, caching technology improves performance by storing copies of frequently accessed data. The core challenge lies in the efficiency of cache eviction policies. Traditional static algorithms (such as Least Recently Used (LRU), Least Frequently Used (LFU), and First In First Out (FIFO)) are widely adopted, but they cannot adapt to dynamic workloads—LRU is susceptible to cache pollution, LFU is poorly adaptable to outdated data, and FIFO ignores access patterns. Related technologies employ caching strategies based on reinforcement learning (RL), which achieve dynamic decision-making through online learning, avoiding dependence on labeled data. However, RL methods in these technologies have a fundamental flaw: their problem modeling is overly simplified, the state space contains only a few indicators (such as file freshness), and the action space is limited to binary choices (caching / not caching). Therefore, existing technologies cannot capture the full picture of caching strategies, resulting in an inability to adaptively generate efficient cache eviction policies in real-world dynamic environments. Summary of the Invention

[0003] This disclosure provides a data caching method, data caching system, apparatus, electronic device, and medium that enable fine-grained configuration of caching strategies.

[0004] In a first aspect, this application provides a data caching method applied to an intelligent agent, the method comprising: Based on multiple pre-collected first state information, multiple corresponding first action information are determined, whereby the first state information is used to characterize the state information of the backend environment. After applying the first caching strategy configured based on the first action information in the backend environment, multiple second state information for characterizing the backend environment are obtained; Determine multiple reward information corresponding to the multiple second state information; wherein, the reward information is used to evaluate the execution effect of the first caching strategy, and each reward information corresponds one-to-one with a second state information; The reinforcement learning network in the agent is updated based on the first state information, the first action information, the second state information, and the reward information. The third state information of the backend environment is obtained, and the target action information corresponding to the third state information is determined based on the updated reinforcement learning network to instruct the execution of the corresponding caching strategy in the backend environment.

[0005] Secondly, this application also provides a data caching system, comprising: The status collection module is used to collect status information in the backend environment. The status information is used to characterize the cache status and load status of the backend environment, which is the environment used to run the data caching device, database and server. An intelligent agent is used to generate first action information containing cached policy parameters through a built-in reinforcement learning network based on the first state information collected by the state collection module, and to update the reinforcement learning network based on the first state information, the first action information, the second state information after the policy corresponding to the first action information is executed, and the reward information corresponding to the second state information, so as to determine the target action information corresponding to the third state information in the backend environment based on the updated reinforcement learning network. The action execution module is used to configure the corresponding caching strategy for action information and instruct the execution of the caching strategy in the backend environment.

[0006] Thirdly, this application also provides a data caching device for use in an intelligent agent, the device comprising: The first determining module is used to determine a plurality of corresponding first action information based on a plurality of pre-collected first state information, wherein the first state information is used to characterize the state information of the backend environment. The first acquisition module is used to acquire multiple second state information for characterizing the backend environment after applying a first caching strategy configured based on the first action information in the backend environment. The second determining module is used to determine multiple reward information corresponding to the multiple second state information; wherein, the reward information is used to evaluate the execution effect of the first caching strategy, and each reward information corresponds one-to-one with a second state information; The network update module is used to update the reinforcement learning network in the agent based on the first state information, the first action information, the second state information, and the reward information. The third determination module is used to obtain the third state information of the backend environment, and determine the target action information corresponding to the third state information based on the updated reinforcement learning network, so as to instruct the execution of the corresponding caching strategy in the backend environment.

[0007] Fourthly, this application provides an electronic device including a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the computer program, when executed by the processor, implements the steps of the method described in the first aspect.

[0008] Fifthly, this application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the method described in the first aspect.

[0009] In a sixth aspect, this application provides a computer program product including computer instructions that, when executed by a processor, implement the steps of the method described in the first aspect.

[0010] This embodiment of the application initially establishes a correspondence between the strategy and the environment by matching the initial state with the action. Then, after applying the first caching strategy configured with the first action information, it collects the second state information and determines the reward information, achieving quantitative feedback on the strategy execution effect. Subsequently, it updates the reinforcement learning network based on the first state, first action, second state, and reward information, completing iterative optimization of the strategy. Finally, it collects the real-time third state information of the backend environment and generates target action information through the updated network, achieving dynamic adaptation of the caching strategy. As can be seen, in this embodiment, the interaction between the agent and the backend environment enables continuous learning and updating, continuously optimizing the reinforcement learning network. The caching strategy is optimized through a "state-action-reward" reinforcement learning loop to adapt to dynamic changes in the backend environment. This overcomes the shortcomings of traditional caching algorithms, such as susceptibility to cache pollution and poor adaptability under dynamic workloads, providing efficient and adaptive cache management and improving the performance and resource utilization of data caching devices. Attached Figure Description

[0011] Figure 1 This is one of the flowcharts illustrating a data caching method provided in an embodiment of this application; Figure 2 This is a second schematic flowchart of a data caching method provided in an embodiment of this application; Figure 3 This is a schematic diagram of the structure of a data caching system provided in an embodiment of this application; Figure 4 This is a comparative diagram of the data caching system of this application embodiment and the backend application and caching system in the prior art; Figure 5 This is a schematic diagram of the structure of a data caching device provided in an embodiment of this application; Figure 6 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0012] The technical solutions of the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this disclosure. Based on the embodiments of this disclosure, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this disclosure.

[0013] Before providing a further detailed description of the embodiments of this application, the relevant technologies and terms involved in the embodiments of this application will be explained.

[0014] Caching is a fundamental and crucial technology for improving performance. Its core idea is to store frequently accessed data copies in a faster storage medium (such as memory). When subsequent requests access the same data again, it can be retrieved directly from the cache, thus avoiding access to slower backend storage systems (such as disk databases or remote services), significantly reducing access latency and alleviating backend load. However, the performance of a caching system largely depends on its management strategy, especially the cache eviction algorithm—the rules that determine which data to evict when the cache space is full. Algorithms based on simple static heuristics used in related technical fields mainly include: Least Recently Used (LRU): This algorithm evicts data items that have not been accessed for the longest time. LRU logic is simple and performs well in many scenarios, but it has a fatal flaw: cache pollution. When the system encounters a one-time, large-scale sequential scan or streaming access (e.g., a single traversal of a large dataset), these newly accessed data items will flood the cache, evicting all the "hot" data that originally had high reuse value but had not been accessed recently, causing a sharp drop in cache efficiency.

[0015] Least Frequently Used (LFU): This algorithm evicts data items with the lowest access frequency. LFU can identify data that has been popular for a long time, but its drawback is poor adaptability. For data that was once popular but is now "outdated," LFU will retain it in the cache for too long; at the same time, for newly emerging data that may become hot, it is difficult to cache it in a timely manner due to its low initial access frequency.

[0016] First-In-First-Out (FIFO): This algorithm evicts the earliest data item that entered the cache. FIFO is the simplest to implement, but it completely ignores data access patterns (whether recent or frequent), so its performance is usually poor under most real-world dynamic workloads.

[0017] The traditional algorithms mentioned above rely on single, fixed rules, while the workload of backend applications today is highly dynamic, complex, and unpredictable. Factors such as user behavior patterns, data popularity, and read / write ratios are constantly changing. Therefore, a caching strategy that performs optimally at one time and under one type of load may become very inefficient at another time or under a different load.

[0018] Currently, attempts to address the limitations of static caching strategies are mainly developing in two directions: rule-based / prediction-based systems and machine learning-based systems, among which: 1. Rule-based and prediction-based "smart" caching: The core logic relies on a pre-written set of rules, making the system rigid and unable to learn and adapt to unknown patterns outside of those rules. Alternatively, there's "predictive" caching focused on specific domains (such as video streaming), optimized through techniques like prefetching and request clustering. Therefore, rule-based and prediction-based "smart" caching is often domain-specific, lacks generality, and doesn't provide a universal framework capable of autonomously generating complete caching strategies.

[0019] 2. Caching strategies based on machine learning (ML), specifically including: (1) Supervised learning methods, such as DeepCache, use Long Short-Term Memory (LSTM) networks to predict the popularity of content, thereby determining which objects to cache. Additionally, dynamic cache allocation is achieved through regression techniques such as Support Vector Regression (SVR). However, a common drawback of these methods is their heavy reliance on large-scale, high-quality labeled datasets for offline training, and their difficulty in generalizing to emerging access patterns not present in the training data.

[0020] (2) Reinforcement Learning (RL) methods: The emergence of reinforcement learning supports online learning without the need for pre-prepared labeled data. For example, RLCache and solutions for the Internet of Things (IoT) or edge networks have successfully verified the feasibility of RL in caching decisions.

[0021] However, despite the existing technology acknowledging the potential of using RL, the following deep-seated problems generally limit its effectiveness in real-world, complex production environments: (1) Problem modeling is too simplistic: In order to reduce the difficulty of the problem, existing RL solutions use an oversimplified definition of Markov Decision Process (MDP). For example, the state space contains only a few indicators, and the action space is limited to the binary choice of "whether to cache", which makes it difficult to capture the full picture of the caching strategy.

[0022] (2) Lack of holistic strategy control: Existing technologies often focus only on a single aspect of cache management, such as replacement or admission decisions, while neglecting comprehensive control over a complete cache strategy. A truly effective strategy requires coordinated adjustment of multiple dimensions, including the selection of the replacement algorithm, the time to live (TTL) of cached objects, and whether to preload them.

[0023] Existing technologies isolate these dimensions, making it difficult to achieve synergistic effects between them. Therefore, embodiments of this application provide a data caching method, data caching system, apparatus, electronic device, and medium to solve the above-mentioned technical problems. The following detailed description, in conjunction with the accompanying drawings, provides specific embodiments and their application scenarios.

[0024] Please see Figure 1 This application provides a data caching method, which is applied to an intelligent agent. The method includes: Step 101: Determine multiple corresponding first action information based on multiple pre-collected first state information, wherein the first state information is used to characterize the state information of the backend environment.

[0025] It should be noted that the intelligent agent in the embodiments of this application can be understood as a reinforcement learning agent embedded in the data caching device, and can adopt reinforcement learning networks, such as Soft Actor-Critic (SAC) algorithm network, Deep Q-Network (DQN), Proximal Policy Optimization (PPO) or Deep Deterministic Policy Gradient (DDPG) algorithm network, etc.

[0026] The aforementioned backend environment can be understood as a running computer system, which may include data caching devices, application servers, databases, and the caching server itself (such as one or more Redis or Memcached instances).

[0027] In some implementations, the agent may include a state collection module, which acts as the agent's sensory organ, continuously collecting and aggregating various raw monitoring data from the backend environment, i.e., first state information. In this embodiment, by inputting the state vector provided by the state collection module into the agent, a composite action vector (i.e., first action information) can be obtained.

[0028] In other embodiments, the first state information may specifically include request patterns and latency from the server application layer, hit / miss rates and memory usage from the server cache layer, and data change frequency from the server database layer. Based on this, the state collection module can clean and normalize the collected data to construct a standardized state vector, and then transmit the state vector to the agent, which then receives the state vector (i.e., the first state information).

[0029] In the above steps, the agent can determine the first state information (i.e., the standardized state vector S) based on the first state information. t The corresponding first action information is calculated. For example, the first state information can be used as input, and the corresponding first action information can be calculated through the nonlinear mapping of a deep neural network. In some implementations, the first state information or state vector... These can be input into the hidden layers of the policy network in an agent, where the network automatically learns the complex relationships between state features and policy parameters. For example, the state vector... (Including cache layer metrics, application layer metrics (high frequency of image resource access, average latency of 45ms), and database layer metrics (medium frequency of write operations)) Inputting the policy network can output a multi-dimensional action vector. That is, the first action information, and each dimension in the first action information corresponds to an adjustable parameter of the caching strategy.

[0030] The aforementioned first action information can be a composite vector defining a complete caching strategy. It overcomes the limitations of existing technologies that fragmented optimization of eviction algorithms, TTL, or preloading, achieving dynamic and adaptive cache management by synchronously controlling multiple strategy dimensions. For example, when the first status information indicates high access frequency for image resources and an increasing cache miss rate, the first action information can specify switching the eviction algorithm to LFU (instead of LRU), extending the TTL for image resources, and simultaneously enabling a preloading strategy for image resources (predicting high-frequency requests based on historical access patterns). As another example, when the status information indicates "frequent database write operations and large system load fluctuations," the first action information can specify using the FIFO eviction algorithm (avoiding redundant retention of outdated data by LFU), shortening the TTL for all cached objects, accelerating data updates, and disabling caching for temporary data (such as session cache) to reduce memory fragmentation.

[0031] As can be seen, in the embodiments of this application, the first action information integrates the policy dimensions (elimination algorithm, TTL, preloading, etc.) into a unified vector, enabling the intelligent agent to globally optimize caching behavior based on real-time state.

[0032] Step 102: After applying the first caching strategy configured based on the first action information in the backend environment, obtain multiple second state information to characterize the backend environment.

[0033] It is understood that the embodiments of this application optimize the caching strategy through a "state-action-reward" reinforcement learning loop. Here, "caching state and load state" are specific examples of state information (including first state information, second state information, and third state information), and not the only form. In practical applications, state information may also include other information related to the effectiveness of the caching strategy, such as data characteristics (e.g., data size, access frequency distribution) and network status (e.g., link bandwidth, latency).

[0034] It is worth mentioning that the above-mentioned second state information This can be understood as a core data set collected in real time by the state collection module after the first caching strategy (configured by the action execution module based on the first action information) is applied in the backend environment, used to dynamically characterize the server's operating status. In this embodiment, the first state information can be matched with the corresponding first action information. After the caching strategy corresponding to the first action information is executed, the backend environment will generate second state information accordingly, and the second state information can correspond to unique reward information, forming a "state-action-feedback" reinforcement learning core data link.

[0035] In this embodiment, the second state information can serve as input for reward information calculation, driving the closed-loop learning mechanism of the agent's optimization strategy. Specifically, the second state information... This can include cache layer characteristics and load layer characteristics. Cache layer characteristics can include cache hit rate (indicating the proportion of requests that actually hit after a policy is applied), cache miss rate, memory usage (the percentage of memory used by the cache), and hot data distribution (changes in the distribution of frequently accessed resources). Load layer characteristics can include request latency (e.g., average response time), data access patterns (changes in request frequency, e.g., image resource access frequency changing from 1000 times / minute to 1500 times / minute), system throughput (e.g., requests processed per second), and database load (write operation frequency, e.g., changing from 50 times / second to 45 times / second, due to reduced cache and decreased database pressure). Second state information. It can also include resource classification granularity indicators (e.g., the actual effect of adjusting the TTL of image resources, where the original TTL is 30 minutes and the new TTL is 36 minutes, at which point the hit rate is improved) and strategy conflict indicators (e.g., the balance between the speed of cold data elimination and the retention rate of hot data after the elimination algorithm is switched).

[0036] In some embodiments, the executor of the agent can be an action execution module that receives first action information and converts it into specific configuration instructions for the backend environment. For example, the action execution module sends the CONFIG SET maxmemory-policy allkeys-lfu command (i.e., the first action information) to the Redis server to change the eviction policy, or executes a script to batch modify the TTL values ​​of specific types of data keys.

[0037] The aforementioned first caching strategy can include multiple configurable dimensions, each dynamically generated based on the real-time status of the backend environment. In one implementation, based on the first action information (eviction algorithm: LFU, image TTL×1.2, preloading: True), the first caching strategy can be configured as follows: eviction algorithm, i.e., enabling LFU (retaining frequently accessed image resources); image resource TTL (e.g., automatically adjusting from 30 minutes to 36 minutes); preloading (i.e., enabling image resources to be cached in advance based on historical access patterns).

[0038] In the above steps, after the action execution module configures the first caching strategy and applies it to the backend environment, the state collection module can continuously collect data (e.g., once per second) to ensure the dynamic process of the coverage strategy taking effect. Furthermore, the state collection module can clean and normalize the raw monitoring data (e.g., Redis keyspace_hits, database innodb_rows_read) to ensure that the structure of the obtained second state information is aligned with the first state information, avoiding interference with the agent's learning.

[0039] Step 103: Determine multiple reward information corresponding to the multiple second state information; wherein, the reward information is used to evaluate the execution effect of the first caching strategy, and each reward information corresponds one-to-one with a second state information.

[0040] It should be noted that after the first action information is configured as the first caching strategy and executed in the backend environment, the backend environment will operate according to the new first caching strategy, forming a new state. Simultaneously, the state collection module collects cache and load-related information in the backend environment at this time, i.e., the second state information. Then, the agent can calculate a scalar "reward" signal, i.e., reward information, based on the second state information. It is understandable that reward information can be a scalar evaluation signal that quantifies the effectiveness of the caching strategy, and can quantify the merits of the first caching strategy (e.g., an increase in cache hit rate or a decrease in system response time). Its core function is to guide the agent to iteratively optimize the caching strategy through reinforcement learning, so that the strategy ultimately tends to maximize the overall performance of the backend system.

[0041] In this embodiment, the intelligent agent can continuously adjust and update the caching strategy through a trial-and-error cycle of observation-action-reward, and can select actions that maximize long-term cumulative rewards for any state information in the backend environment.

[0042] Step 104: Update the reinforcement learning network in the agent based on the first state information, the first action information, the second state information, and the reward information.

[0043] In the above steps, historical interaction data (first state information, first action information, second state information, and reward information) can be used to form a structured training set. The policy value is evaluated through a critic network, comparing the predicted policy effect with the actual reward (e.g., increased hit rate, reduced latency), and parameters are adjusted accordingly. A soft actor network is used to generate policies, maximizing policy value while adaptively updating and utilizing the soft actor-critic network (e.g., avoiding over-adjustment of TTL leading to cache pollution).

[0044] Therefore, in the above embodiments of this application, the agent can achieve online learning through a closed-loop mechanism of observation-action-reward. After the action execution module performs an action, it can receive reward signals based on environmental feedback (e.g., increased cache hit rate or reduced latency), enabling the agent to dynamically update the SAC network without relying on historical labeled data. Simultaneously, the experience replay mechanism allows for the reuse of historical data, improving learning efficiency.

[0045] Step 105: Obtain the third state information of the backend environment, and determine the target action information corresponding to the third state information based on the updated reinforcement learning network, so as to instruct the execution of the corresponding caching strategy in the backend environment.

[0046] In the above steps, when the backend environment generates new real-time state information (i.e., third state information), the agent can directly generate target action information using the soft actor network (Actor) in the updated soft actor-critic network (SAC).

[0047] In one specific embodiment, an e-commerce platform adopts a microservice architecture. Its product catalog service uses a distributed Redis cluster as a cache to store information such as product details, user reviews, and inventory. The access load of this e-commerce platform is highly dynamic and easily affected by factors such as flash sales, marketing promotions, and holidays. The data caching system in this application can be deployed on this e-commerce platform. The state collection module can collect third-party state information by crawling the Prometheus monitoring system to obtain the e-commerce platform load, parsing application logs (to obtain request patterns), and executing Redis commands (e.g., to obtain cache statistics).

[0048] In some embodiments, the agent can use Python and PyTorch libraries, and can run as a standalone microserver. The action execution module configures the corresponding caching strategy by calling the Redis client library and executing commands such as CONFIG SET and TTL.

[0049] Therefore, the embodiments of this application overcome the cache pollution problem of traditional LRU / LFU (e.g., sequential scanning to evict hot data). By configuring the corresponding cache strategy through the multi-dimensional action information determined by the updated SAC network, global optimization is achieved. Furthermore, the feedback closed-loop mechanism in the embodiments of this application enables the agent to respond to load fluctuations in real time, continuously optimize the soft actor-commentator network, adapt to the dynamic changes of the backend environment, overcome the defects of traditional caching algorithms that are susceptible to cache pollution and have poor adaptability under dynamic workloads, provide efficient and adaptive cache management, and improve the performance and resource utilization of data caching devices.

[0050] It should be noted that the SAC network comprises two parts: a soft actor network and a critic network. In some embodiments, the soft actor network can receive first state information. As input, the first action information is output. This soft actor network can output the mean and variance of actions using a probability distribution (e.g., Gaussian distribution) to ensure the exploratory and stable nature of the strategy. For example, when the input state vector shows that the image access frequency is high and the cache miss rate is increasing, the actor network automatically outputs cooperative action information (e.g., switching the LFU algorithm, extending the image TTL, etc.). The critic network can receive second-state information. As input, and combined with the raw reward signals from the backend environment (such as increased hit rate and reduced latency), reward information is calculated. .

[0051] It is worth noting that SAC, as an off-policy algorithm, can reuse historical data through experience replay, resulting in a learning efficiency far exceeding that of on-policy algorithms that require new data for each update. Furthermore, the SAC network is suitable for mixed / continuous action spaces, specifically designed for handling continuous action spaces, and can be naturally extended to the parameterized (mixed) action space defined in this application. In addition, the core of SAC is the Maximum Entropy framework, which maximizes not only the cumulative reward but also the policy entropy (randomness). This enables the agent in the embodiments of this application to effectively avoid the policy prematurely converging to a suboptimal local optimum.

[0052] Optionally, determining the corresponding first action information based on multiple pre-collected first state information includes: Based on the preset actor network in the reinforcement learning network, the plurality of first state information is preprocessed to obtain the plurality of first action information; wherein, the first action information includes a plurality of cache strategy parameters for collaboratively defining cache strategies.

[0053] It should be noted that the Actor network can be the core module responsible for "generating actions" in reinforcement learning. By preprocessing the first state information of the backend environment, a "state-action" mapping relationship is established. The output first action information is not a simple binary choice of "caching / not caching", but contains multiple parameters that collaboratively define caching strategies, so as to realize fine-grained configuration of caching strategies.

[0054] In some embodiments, preprocessing may involve data cleaning, feature extraction, and normalization of pre-collected multi-dimensional backend state data (such as hit rate, latency, load, etc.), where feature extraction may involve selecting core features strongly correlated with the caching strategy. Subsequently, the preprocessed first state information is input into the actor network, and through nonlinear mapping (such as the ReLU activation function), it outputs first action information containing caching strategy parameters, with the parameters collaboratively defining the caching strategy.

[0055] Optionally, determining the multiple reward information corresponding to the multiple second state information includes: Based on the pre-set critic network in the reinforcement learning network, the difference parameter between the second state information and the corresponding first state information is determined, and the difference parameter is quantified into the reward information corresponding to the second state information.

[0056] It should be noted that the critic network can be the core module in reinforcement learning responsible for evaluating the effect of actions. By comparing the backend environment state (first / second state information) before and after the policy execution, it calculates multi-dimensional difference parameters and quantifies them into scalar reward information, which intuitively reflects the execution effect of the first caching policy and provides accurate basis for subsequent network updates.

[0057] In some embodiments, a fully connected neural network is constructed, with the input being the difference features between the first and second state information, and the output being a scalar reward value. Preset weights (such as hit rate weight 0.4, latency weight 0.3, and load weight 0.3) are used to adapt to business priorities. After collecting the second state information after applying the first caching strategy, the values ​​of the first and second state information under the same dimension can be compared to calculate the difference parameters (state change values). Subsequently, the multi-dimensional difference parameters are converted into a single scalar reward value according to preset weights and formulas. Positive changes (increased hit rate / decreased latency / decreased load) correspond to positive rewards, while negative changes correspond to negative rewards.

[0058] The above embodiments, through the actor-critic dual network architecture of reinforcement learning, solve the problems of static rigidity in traditional caching strategies and insufficient state and action modeling in existing simplified RL caching schemes. They enable the generation of multi-dimensional states of the backend environment to fine-grained caching strategy actions, as well as accurate quantitative feedback on the effect of strategy execution, providing a core basis for the update of reinforcement learning networks.

[0059] This embodiment outputs multi-parameter collaborative action information (such as elimination algorithms, TTL coefficients, and preloading thresholds) through an actor network. It can adapt to complex backend environments by addressing the three core aspects of cache eviction, expiration management, and preloading (e.g., simultaneously adjusting three types of parameters for "low hit rate, high latency" states). This solves the shortcomings of static algorithms (LRU / LFU) which have a single strategy and cannot handle dynamic loads. Furthermore, this embodiment can calculate multi-dimensional difference parameters through a critic network and weighted quantize them into scalar rewards. This comprehensively reflects the overall impact of the strategy on the backend environment and clearly provides feedback on the strategy's merits through positive / negative rewards, allowing the reinforcement learning network to iterate and optimize rapidly.

[0060] Optionally, the method further includes: The first action information is parsed to obtain at least one cache strategy parameter, and a cache strategy set is generated based on the cache strategy parameter, the cache strategy set including at least one cache strategy; Based on the first state information, determine the first cache policy in the cache policy set; The first caching policy is sent to the cache control node of the backend environment for execution; The step of generating a cache strategy set based on the cache strategy parameters includes: generating multiple candidate strategy items corresponding to each policy dimension according to the policy dimension indicated by the cache strategy parameters; and combining the multiple candidate strategy items to generate the cache strategy set.

[0061] In some embodiments, a candidate strategy pool is first generated by combining caching strategy parameters in multiple dimensions, and then the optimal strategy is accurately matched based on the first state information. This avoids the limitation of a single strategy in adapting to complex dynamic environments and further enhances the targeting and flexibility of the caching strategy.

[0062] Specifically, the first action information is generated by a reinforcement learning actor network. Essentially, it is a composite vector containing multi-dimensional policy control instructions. Through parsing, parameters defining the core dimensions of the caching policy (i.e., caching policy parameters) can be extracted, providing a foundation for subsequent policy generation. For example, assuming the first action information is "eviction algorithm type = adaptive, TTL adjustment coefficient = 1.0-1.5, preloading threshold = 30-60 times / minute", parsing yields three caching policy parameters, corresponding to policy dimensions such as "eviction rule," "data validity period," and "preloading trigger."

[0063] It should be noted that each caching strategy parameter corresponds to an independent strategy dimension. First, multiple candidate strategy items that meet the parameter indication range are generated for each dimension. Then, by combining across dimensions, a set of caching strategies (i.e., a candidate strategy pool) that covers all possible optimal combinations can be generated.

[0064] For example, policy dimension 1 (elimination algorithm type parameter): candidate policy item = [adaptive LRU, adaptive LFU, ARC] (all conforming to the "adaptive" parameter indication); Strategy Dimension 2 (TTL Adjustment Coefficient Parameter): Candidate strategy items = [1.0x (default), 1.2x (20% extension), 1.5x (50% extension)] (all within the range of 1.0-1.5); Strategy Dimension 3 (Preloaded Threshold Parameters): Candidate strategy items = [30 times / minute, 45 times / minute, 60 times / minute] (all within the range of 30-60 times / minute); Based on the above strategy dimensions, a cache strategy set can be generated: a total of 3×3×3=27 candidate strategies, such as [adaptive LRU + 1.0 times TTL + 30 preloads], [adaptive LFU + 1.5 times TTL + 60 preloads], etc.

[0065] Subsequently, the aforementioned first state information represents the real-time dynamics of the backend environment (such as access patterns and cache load). Based on this state, the most suitable strategy for the current environment is selected from the cache strategy set as the first cache strategy, ensuring the targeted nature of the strategy selection. For example, if the first state information (the current state of the backend environment) includes 60% high-frequency access data, a cache hit rate of 65% (relatively low), and 80% core business data, the selection logic could be as follows: due to the high proportion of high-frequency data, "adaptive LFU" (fitting to access frequency) can be prioritized; due to the large amount of core business data, the TTL needs to be extended, so "1.5 times TTL" is selected (to ensure data validity); due to the concentrated high-frequency data access, the preloading threshold can be appropriately reduced, so "30 times / minute" is selected (pre-caching hot data). Finally, the first cache strategy can be determined as [adaptive LFU + 1.5 times TTL + 30 times / minute preloading].

[0066] Finally, the selected first caching strategy can be converted into execution commands recognizable by the backend environment (such as Redis configuration commands or cache server operation scripts) and sent to the cache control node (such as the Redis cluster control terminal) to complete the strategy implementation. For example, the above first caching strategy can be converted into commands: for example, executing CONFIG SETmaxmemory-policy allkeys-lfu (to enable adaptive LFU); or batch executing EXPIRE key 5400 (extending the TTL of core business data to 1.5 hours, originally the default of 1 hour); or configuring a preloading script to automatically preload related data when the data access volume reaches 30 times / minute, and sending it to the Redis cluster control node for execution.

[0067] Optionally, determining the first caching policy in the caching policy set based on the first state information includes: Based on at least one of the following: the cache load status of the backend environment, the data access mode of the client, the write operation frequency of the database, and the degree of conformity of each cache strategy in the preset applicable scenario, the matching degree between the first status information and each cache strategy in the cache strategy set is determined. The cache strategy with the highest matching degree is selected from the set of cache strategies as the first cache strategy.

[0068] In the above embodiments, by constructing a multi-dimensional and quantifiable matching evaluation system, the core features of the first state information can be accurately matched with the applicability of the caching strategy, and the first caching strategy that is most suitable for the current backend environment can be selected, thus solving the core problems of subjective strategy selection and poor adaptability.

[0069] Specifically, the embodiments of this application have completed two basic operations prior to this: parsing the first action information to extract cache strategy parameters (such as eviction algorithm type, TTL adjustment coefficient, preloading threshold, etc.), and generating a cache strategy set containing multiple candidate cache strategies based on these parameters; collecting first state information, which covers at least one core feature such as the cache load state of the backend environment, client data access mode, database write operation frequency, and business preset applicable scenarios.

[0070] First, at least one of the following can be selected as a matching evaluation dimension: "backend environment cache load status, client data access pattern, database write operation frequency, and the degree of fit between the caching strategy and the preset applicable scenario". Second, based on the technical characteristics of each caching strategy (such as the applicable scenarios of the eviction algorithm, the applicable conditions of the TTL adjustment coefficient, and the matching logic of the preloading threshold), matching judgment rules should be formulated for each selected evaluation dimension to clarify the criteria for judging the adaptability of the caching strategy under different state characteristics. If a comprehensive multi-dimensional evaluation is required, weights can also be configured for each dimension according to business priorities (such as prioritizing hit rate for core business and prioritizing load stability for high-concurrency scenarios) for subsequent comprehensive matching calculation.

[0071] Subsequently, based on the established evaluation rules, each candidate caching strategy in the caching strategy set is compared one by one with the features of the corresponding evaluation dimension in the first state information: if only a single dimension is selected for evaluation, the matching degree of the candidate strategy under that dimension is directly obtained; if multiple dimensions are selected for evaluation, the matching degree score of each candidate strategy under each dimension is calculated first, and then the scores of each dimension are weighted and summed according to the preset weights to obtain the comprehensive matching degree between the candidate strategy and the first state information.

[0072] Finally, after calculating the matching degree of all candidate strategies in the caching strategy set, the candidate caching strategy with the highest comprehensive matching degree is selected as the first caching strategy; if only a single dimension is used for evaluation, the candidate strategy with the best matching degree under that dimension is selected as the first caching strategy. The final determined first caching strategy is the caching strategy with the best adaptability under the current backend environment.

[0073] In the above embodiments, a matching degree system can be constructed through multiple dimensions, weights, and scoring rules, transforming the strategy selection based on experience into a quantifiable and reproducible calculation process. This solves the problems of strong subjectivity and lack of unified standards in traditional strategy selection. By pre-calculating the matching degree to select the optimal strategy, the inefficient strategy in the strategy set is avoided from being directly executed, reducing problems such as decreased cache hit rate and memory waste caused by strategy mismatch, and lowering the trial and error cost of the backend environment.

[0074] Optionally, updating the reinforcement learning network in the agent based on the first state information, the first action information, the second state information, and the reward information includes: Based on the correspondence between the first state information and the first action information, and the correspondence between the second state information and the reward information, at least one first state information, the first action information matched by the first state information, the second state information corresponding to the first action information, and the reward information corresponding to the second state information are associated and bound to obtain at least one training data set. The at least one training data set is input into the reinforcement learning network to update the weights of the soft actor network and the critic network in the reinforcement learning network, thereby obtaining the updated soft actor-critic network.

[0075] In this embodiment, the agent can be trained using an off-policy approach, and all information generated from interactions with the environment forms an experience tuple. Multiple tuples can be stored in a fixed-size ExperienceReplay Buffer. During training, the agent can extract a small batch of experience data from the experience buffer to update the SAC network.

[0076] In some implementations, a Markov decision process (MDP) can be composed of tuples. Define, where: First status information This is a high-dimensional vector that comprehensively describes the state of the backend environment at time step t. Through feature engineering, it integrates information from multiple dimensions to provide the complete context needed for decision-making. For example, This can include workload characteristics (such as request rate, read-write ratio, hot data distribution, access timing patterns, etc.), cache performance characteristics (such as cache hit rate, byte hit rate, memory usage and eviction count, etc.), load characteristics (such as CPU load, memory pressure, response latency, network throughput, etc.), and data characteristics (such as content type distribution, data update frequency and object size distribution, etc.).

[0077] First action information This approach, combining strategy selection and continuous parameter tuning, enhances the relevance of the training data. For example, It can include discrete actions and continuous parameters, where discrete actions are used to select the core elimination algorithm, and continuous parameters can be a real vector for fine-tuning the strategy, such as ttl_multiplier(0.5-2.0) for scaling TTL.

[0078] Reward Information The aim is to balance competing performance metrics and guide agents to learn strategies that are beneficial to the business, rather than simply optimizing a single metric (such as hit rate).

[0079] Second state information After the first caching strategy is applied in the backend environment, the second state information is a core data set collected in real time by the state collection module to dynamically characterize the system's operating status. This second state information is not a static snapshot, but rather real-time feedback after the strategy is executed, directly reflecting the quantitative impact of the caching strategy on system performance.

[0080] Therefore, the above multiple tuples They can form a training dataset, and a training dataset must contain at least one training data group.

[0081] In some embodiments, the agent drives the continuous optimization of the soft actor-critic network by constructing a structured training dataset. The training dataset is fed into the soft actor-critic network, where the soft actor network (responsible for generating cached policy actions) and the critic network (responsible for evaluating policy value) simultaneously update their weights.

[0082] Specifically, the critic network can optimize its parameters by minimizing prediction error by comparing the predicted policy value with the actual reward signal. The soft actor network, on the other hand, can maximize policy value while incorporating adaptive entropy regularization to adjust its output and generate better cooperative policies. This update process allows the agent to learn the global effect of policies from historical experience, rather than optimizing single parameters in isolation, ultimately resulting in an updated network used to generate efficient cached policies in real time.

[0083] In some embodiments, all generated training data sets are input into a reinforcement learning network, and weight updates are performed separately for the two modules of the soft actor = critic network: For the actor network, the first state information in the training data set is used as the input reference. Combined with the corresponding first action information and reward information feedback, the network weights are adjusted and the state-action mapping logic is optimized so that the subsequently generated action information is more in line with the optimization requirements of the backend environment.

[0084] For the commentator network, the first and second state information in the training data set are used as input references. Combined with the corresponding reward information, the network weights are adjusted to optimize the evaluation logic of state difference-reward quantification, so that the quantification results of subsequent reward information can more accurately reflect the policy execution effect.

[0085] Through the collaborative updating of the weights of the two networks, the soft actor-critic network is finally obtained after iterative optimization.

[0086] Therefore, the embodiments of this application significantly improve the adaptability of the agent in dynamic environments and the quality of the strategy, avoiding the cache pollution problem caused by parameter fragmentation in traditional methods.

[0087] In some specific embodiments, Multi-Agent Reinforcement Learning (MARL) can be employed. For large-scale distributed caching systems (such as multiple edge nodes of a Content Delivery Network (CDN), a single agent architecture can be extended into a multi-agent system. Each cache node can deploy an independent agent, enabling agents to cooperate or compete through communication, thereby optimizing local cache performance while maximizing global cache efficiency.

[0088] In other specific embodiments, the current state information or state vector can incorporate future workload prediction information generated by an independent prediction model (such as a Long Short-Term Memory (LSTM) network). This enables the agent to upgrade from reactive decision-making to predictive decision-making, adjusting caching strategies in advance to cope with upcoming load changes.

[0089] In some specific embodiments, automated reward function shaping can be performed. In related technologies, the weights of multi-objective reward functions need to be set manually, while the embodiments of this application can make the weights themselves part of the learning update, enabling the agent to dynamically adjust its learning focus according to higher-level business objectives (e.g., automatically increasing the weight of latency during peak business periods).

[0090] Based on the above, the embodiments of this application can effectively reduce the request pressure on expensive backend databases and computing resources by maximizing cache hit rate, directly reduce server and bandwidth costs, and improve user experience through lower latency and faster response speed.

[0091] Please see Figure 2 This application also provides a data caching method applied to an intelligent agent, the method comprising: Step 201: Obtain multiple first state information, which are used to characterize the state information of the backend environment; Step 202: Based on the soft actor network in the soft actor-critic network, determine multiple first action information corresponding to multiple first state information; Step 203: Based on multiple first action information, configure multiple corresponding first caching strategies, and instruct the execution of the multiple first caching strategies in the backend environment; Step 204: Obtain multiple second state information, which are information collected under the condition that the first caching strategy is applied in the backend environment to characterize the cache state and load state of the backend environment; Step 205: Based on the critic network in the soft actor-critic network, determine multiple reward information corresponding to the multiple second state information; wherein, the reward information is used to evaluate the execution effect of the first caching strategy, and each reward information corresponds one-to-one with a second state information; Step 206: Store multiple first state information, multiple first action information, multiple second state information, and multiple reward information into the experience replay pool; Step 207: Update the reinforcement learning network in the agent based on the first state information, the first action information, the second state information, and the reward information; Step 208: Obtain the third state information of the backend environment, and determine the target action information corresponding to the third state information based on the updated reinforcement learning network, so as to instruct the execution of the corresponding caching strategy in the backend environment.

[0092] It is understood that all implementations of the above embodiments can be found in the foregoing. Figure 1 The relevant descriptions of the data caching method applied to the intelligent agent shown are not repeated in this embodiment to avoid repetition.

[0093] Please see Figure 3 This application also provides a data caching system, specifically including: The status collection module 301 is used to collect status information in the backend environment. The status information is used to characterize the cache status and load status of the backend environment, which is an environment for running the data caching device, database and server. The intelligent agent 302 is used to generate first action information containing cached policy parameters through a built-in reinforcement learning network based on the first state information collected by the state collection module, and update the reinforcement learning network based on the first state information, the first action information, the second state information after the policy corresponding to the first action information is executed, and the reward information corresponding to the second state information, so as to determine the target action information corresponding to the third state information in the backend environment based on the updated reinforcement learning network. The action execution module 303 is used to configure a corresponding caching strategy for action information and instruct the execution of the caching strategy in the backend environment.

[0094] Optionally, the intelligent agent 302 is used for: Based on the preset actor network in the reinforcement learning network, the plurality of first state information is preprocessed to obtain the plurality of first action information; wherein, the first action information includes a plurality of cache strategy parameters for collaboratively defining cache strategies; The determination of the multiple reward information corresponding to the multiple second state information includes: Based on the pre-set critic network in the reinforcement learning network, the difference parameter between the second state information and the corresponding first state information is determined, and the difference parameter is quantified into the reward information corresponding to the second state information.

[0095] Optionally, the intelligent agent 302 is further configured to: The first action information is parsed to obtain at least one cache strategy parameter, and a cache strategy set is generated based on the cache strategy parameter, the cache strategy set including at least one cache strategy; Based on the first state information, determine the first cache policy in the cache policy set; The first caching policy is sent to the cache control node of the backend environment for execution; The step of generating a cache strategy set based on the cache strategy parameters includes: generating multiple candidate strategy items corresponding to each policy dimension according to the policy dimension indicated by the cache strategy parameters; and combining the multiple candidate strategy items to generate the cache strategy set.

[0096] Optionally, the intelligent agent 302 is used for: Based on at least one of the following: the cache load status of the backend environment, the data access mode of the client, the write operation frequency of the database, and the degree of conformity of each cache strategy in the preset applicable scenario, the matching degree between the first status information and each cache strategy in the cache strategy set is determined. The cache strategy with the highest matching degree is selected from the set of cache strategies as the first cache strategy.

[0097] Optionally, the intelligent agent 302 is used for: Based on the correspondence between the first state information and the first action information, and the correspondence between the second state information and the reward information, at least one first state information, the first action information matched by the first state information, the second state information corresponding to the first action information, and the reward information corresponding to the second state information are associated and bound to obtain at least one training data set. The at least one training data set is input into the reinforcement learning network to update the weights of the soft actor network and the critic network in the reinforcement learning network, thereby obtaining the updated soft actor-critic network.

[0098] It is understood that all implementations of the above embodiments can be found in the foregoing. Figure 1 or Figure 2 The relevant descriptions of the data caching method applied to the intelligent agent shown are not repeated in this embodiment to avoid repetition.

[0099] Please see Figure 4 This is a comparative diagram of the data caching system in this application embodiment and the backend application and caching system in the prior art. It can be understood that the backend application and caching system architecture in the prior art includes business logic services, databases and caching modules (such as Redis), which belongs to a passive caching mode. In this mode, user requests trigger the business logic service to read and write the caching module. The caching module relies on fixed static strategies (such as LRU, LFU) to manage the cache, and has no dynamic adjustment capability. It can only passively respond to business read and write operations and cannot adapt to changes in the load and access patterns of the backend environment.

[0100] In this embodiment, an intelligent control closed loop consisting of a "state collection module, an intelligent agent, and an action execution module" is adopted. The state collection module collects state information of the backend environment (cache state, load, etc.), the intelligent agent generates action instructions for a dynamic caching strategy based on the state information, and the action execution module translates the instructions into specific operations, adjusting the strategy of the left-hand caching module. This closed loop achieves proactive adaptation of state collection, intelligent decision-making, and strategy execution, replacing the fixed caching strategy of existing technologies and enabling dynamic responses to changes in the backend environment.

[0101] As can be seen, the existing technology is a passive fixed-strategy caching mode, while this application is an active intelligent adaptive caching system, which solves the defect of the existing technology that cannot adapt to dynamic load.

[0102] Please see Figure 5 This disclosure provides a data caching device for use in an intelligent agent, the device comprising: The first determining module 401 is used to determine a plurality of corresponding first action information based on a plurality of pre-collected first state information, wherein the first state information is used to characterize the state information of the backend environment. The first acquisition module 402 is used to acquire multiple second state information for characterizing the backend environment after applying a first caching strategy configured based on the first action information in the backend environment. The second determining module 403 is used to determine multiple reward information corresponding to the multiple second state information; wherein, the reward information is used to evaluate the execution effect of the first caching strategy, and each reward information corresponds one-to-one with a second state information; Network update module 404 is used to update the reinforcement learning network in the agent according to the first state information, the first action information, the second state information and the reward information; The third determination module 405 is used to obtain the third state information of the backend environment, and determine the target action information corresponding to the third state information based on the updated reinforcement learning network, so as to instruct the execution of the corresponding caching strategy in the backend environment.

[0103] Optionally, the first determining module 401 includes: The first acquisition unit is used to preprocess the plurality of first state information based on the preset actor network in the reinforcement learning network to acquire the plurality of first action information; wherein, the first action information includes a plurality of cache strategy parameters for collaboratively defining cache strategies; The second determining module 403 includes: The first determining unit is configured to determine the difference parameter between the second state information and the corresponding first state information based on a pre-set critic network in the reinforcement learning network, and quantify the difference parameter into the reward information corresponding to the second state information.

[0104] Optionally, the data caching device further includes: The strategy generation module is used to parse the first action information to obtain at least one cache strategy parameter, and to generate a cache strategy set based on the cache strategy parameter, wherein the cache strategy set includes at least one cache strategy; The fourth determining module is used to determine the first caching strategy in the caching strategy set based on the first state information; The policy distribution module is used to distribute the first caching policy to the cache control node of the backend environment for execution; The step of generating a cache strategy set based on the cache strategy parameters includes: generating multiple candidate strategy items corresponding to each policy dimension according to the policy dimension indicated by the cache strategy parameters; and combining the multiple candidate strategy items to generate the cache strategy set.

[0105] Optionally, the fourth determining module 405 includes: The second determining unit is used to determine the matching degree between the first state information and each cache strategy in the cache strategy set based on at least one of the following: the cache load state of the backend environment, the data access mode of the client, the write operation frequency of the database, and the degree of conformity of each cache strategy in a preset applicable scenario. The strategy selection unit is used to select the cache strategy with the highest matching degree from the cache strategy set as the first cache strategy.

[0106] Optionally, the network update module 404 includes: The second acquisition unit is used to associate and bind at least one first state information, the first action information matched by the first state information, the second state information corresponding to the first action information, and the reward information corresponding to the second state information according to the correspondence between the first state information and the first action information and the correspondence between the second state information and the reward information, so as to obtain at least one training data group. The network update unit is used to input the at least one training data set into the reinforcement learning network, update the weights of the soft actor network and the critic network in the reinforcement learning network, and obtain the updated soft actor-critic network.

[0107] The data caching device provided in this disclosure can achieve the above-mentioned... Figure 1 or Figure 2 To avoid repetition, the various processes in the data caching method embodiment shown will not be described again here.

[0108] According to embodiments of this disclosure, this disclosure also provides an electronic device and a readable storage medium.

[0109] Figure 6A schematic block diagram of an example electronic device 500 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0110] like Figure 6 As shown, device 500 includes a computing unit 501, which can perform various appropriate actions and processes based on a computer program stored in read-only memory (ROM) 502 or a computer program loaded from storage unit 508 into random access memory (RAM) 503. RAM 503 may also store various programs and data required for the operation of device 500. The computing unit 501, ROM 502, and RAM 503 are interconnected via bus 504. Input / output (I / O) interface 505 is also connected to bus 504.

[0111] Multiple components in device 500 are connected to I / O interface 505, including: input unit 506, such as keyboard, mouse, etc.; output unit 507, such as various types of monitors, speakers, etc.; storage unit 508, such as disk, optical disk, etc.; and communication unit 509, such as network card, modem, wireless transceiver, etc. Communication unit 509 allows device 500 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0112] The computing unit 501 can be various general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 501 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any suitable processor, controller, microcontroller, etc. The computing unit 501 performs the various methods and processes described above, such as the data caching method. For example, in some embodiments, the data caching method may be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 508. In some embodiments, part or all of the computer program may be loaded and / or installed on device 500 via ROM 502 and / or communication unit 509. When the computer program is loaded into RAM 503 and executed by the computing unit 501, one or more steps of the data caching method described above may be performed. Alternatively, in other embodiments, the computing unit 501 may be configured to perform a data caching method by any other suitable means (e.g., by means of firmware).

[0113] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-chip (SoCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0114] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0115] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0116] As used herein, the term "machine-readable medium" refers to any computer program product, device, and / or apparatus (e.g., disk, optical disk, memory, programmable logic device (PLD)) used to provide machine instructions and / or data to a programmable processor, including machine-readable media that receive machine instructions as machine-readable signals. The term "machine-readable signal" refers to any signal used to provide machine instructions and / or data to a programmable processor.

[0117] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0118] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0119] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact via communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other. Servers can be cloud servers, servers in distributed systems, or servers incorporating blockchain technology.

[0120] This application also provides a computer program product, including computer instructions, which, when executed by a processor, implement the above-described... Figure 1 or Figure 2 The various processes of the method embodiments shown can achieve the same technical effect, and will not be described again here to avoid repetition.

[0121] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this application can be achieved, and this is not limited herein.

[0122] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A data caching method, characterized in that, Applied to intelligent agents, the method includes: Based on multiple pre-collected first state information, multiple corresponding first action information are determined, whereby the first state information is used to characterize the state information of the backend environment. After applying the first caching strategy configured based on the first action information in the backend environment, multiple second state information for characterizing the backend environment are obtained; Determine multiple reward information corresponding to the multiple second state information; wherein, the reward information is used to evaluate the execution effect of the first caching strategy, and each reward information corresponds one-to-one with a second state information; The reinforcement learning network in the agent is updated based on the first state information, the first action information, the second state information, and the reward information. The third state information of the backend environment is obtained, and the target action information corresponding to the third state information is determined based on the updated reinforcement learning network, so as to instruct the execution of the corresponding caching strategy in the backend environment.

2. The method according to claim 1, characterized in that, The determination of corresponding first action information based on multiple pre-collected first state information includes: Based on the preset actor network in the reinforcement learning network, the plurality of first state information is preprocessed to obtain the plurality of first action information; wherein, the first action information includes a plurality of cache strategy parameters for collaboratively defining cache strategies; The determination of the multiple reward information corresponding to the multiple second state information includes: Based on the pre-set critic network in the reinforcement learning network, the difference parameter between the second state information and the corresponding first state information is determined, and the difference parameter is quantified into the reward information corresponding to the second state information.

3. The method according to claim 2, characterized in that, The method further includes: The first action information is parsed to obtain at least one cache strategy parameter, and a cache strategy set is generated based on the cache strategy parameter, the cache strategy set including at least one cache strategy; Based on the first state information, determine the first cache policy in the cache policy set; The first caching policy is sent to the cache control node of the backend environment for execution; The step of generating a cache strategy set based on the cache strategy parameters includes: generating multiple candidate strategy items corresponding to each policy dimension according to the policy dimension indicated by the cache strategy parameters; and combining the multiple candidate strategy items to generate the cache strategy set.

4. The method according to claim 3, characterized in that, Based on the first state information, determining the first cache policy in the cache policy set includes: Based on at least one of the following: the cache load status of the backend environment, the data access mode of the client, the write operation frequency of the database, and the degree of conformity of each cache strategy in the preset applicable scenario, the matching degree between the first status information and each cache strategy in the cache strategy set is determined. The cache strategy with the highest matching degree is selected from the set of cache strategies as the first cache strategy.

5. The method according to claim 1, characterized in that, The step of updating the reinforcement learning network in the agent based on the first state information, the first action information, the second state information, and the reward information includes: Based on the correspondence between the first state information and the first action information, and the correspondence between the second state information and the reward information, at least one first state information, the first action information matched by the first state information, the second state information corresponding to the first action information, and the reward information corresponding to the second state information are associated and bound to obtain at least one training data set. The at least one training data set is input into the reinforcement learning network to update the weights of the soft actor network and the critic network in the reinforcement learning network, thereby obtaining the updated soft actor-critic network.

6. A data caching system, characterized in that, include: The status collection module is used to collect status information in the backend environment. The status information is used to characterize the cache status and load status of the backend environment, which is the environment used to run the data caching device, database and server. An intelligent agent is used to generate first action information containing cached policy parameters through a built-in reinforcement learning network based on the first state information collected by the state collection module, and to update the reinforcement learning network based on the first state information, the first action information, the second state information after the policy corresponding to the first action information is executed, and the reward information corresponding to the second state information, so as to determine the target action information corresponding to the third state information in the backend environment based on the updated reinforcement learning network. The action execution module is used to configure the corresponding caching strategy for action information and instruct the execution of the caching strategy in the backend environment.

7. A data caching device, characterized in that, Applied to intelligent agents, the device includes: The first determining module is used to determine a plurality of corresponding first action information based on a plurality of pre-collected first state information, wherein the first state information is used to characterize the state information of the backend environment. The first acquisition module is used to acquire multiple second state information for characterizing the backend environment after applying a first caching strategy configured based on the first action information in the backend environment. The second determining module is used to determine multiple reward information corresponding to the multiple second state information; wherein, the reward information is used to evaluate the execution effect of the first caching strategy, and each reward information corresponds one-to-one with a second state information; The network update module is used to update the reinforcement learning network in the agent based on the first state information, the first action information, the second state information, and the reward information. The third determination module is used to obtain the third state information of the backend environment, and determine the target action information corresponding to the third state information based on the updated reinforcement learning network, so as to instruct the execution of the corresponding caching strategy in the backend environment.

8. An electronic device, characterized in that, It includes a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the computer program, when executed by the processor, implements the steps of the data caching method as described in any one of claims 1 to 5.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the data caching method as described in any one of claims 1 to 5.

10. A computer program product, characterized in that, Includes computer instructions that, when executed by a processor, implement the steps of the data caching method as described in any one of claims 1 to 5.

Images