Distributed caching method and system fusing reinforcement learning and collaborative learning proof consensus

Abstract

The invention belongs to the field of distributed caching, and discloses a distributed caching method and system fusing reinforcement learning and collaborative learning proof consensus, and the method comprises the steps that a user node initiates a request to an oracle machine node through an application program interface; the oracle machine node broadcasts a preparation message; the oracle machine node trains an indirect cache hit model based on a reinforcement learning network; the oracle machine node broadcasts the trained model; the oracle machine node verifies the trained model; the oracle machine node caches the required content and then responds to a user request, and the cache obtaining required content comprises direct cache hit based on a local cache pool and indirect cache hit based on a reinforcement learning network; according to the method, a centralized training distributed execution normal form is improved, and the training effectiveness is ensured, so that good multi-agent cooperation performance is realized. The distributed cache framework provided by the invention is suitable for various Web3.0 scenes, and the safety and reliability of the distributed cache framework are improved.

Description

Technical Field

[0001] The present invention relates to the field of distributed caching, and in particular to a distributed caching method and system integrating reinforcement learning and collaborative learning to prove consensus. Background Art

[0002] Web3.0 is a decentralized Internet running on blockchain technology. It enables users to build a decentralized universal digital identity system that is available on various platforms, and own their own data on the Internet, realizing cross-platform data use and sharing. In Web3.0, blockchain and decentralized storage play a vital role. With its security and immutability, blockchain provides a solid guarantee for data and funds in Web3.0. In addition, blockchain also provides a secure execution environment for smart contracts; decentralized storage systems provide necessary support for decentralized applications in Web3.0, such as storing user data, smart contract status and other information. Therefore, blockchain and decentralized storage jointly build the infrastructure of the Web3.0 ecosystem, providing users with a more secure, transparent and decentralized network environment. Here, we refer to blockchain and decentralized storage as content providers, and content requesters are mainly mobile terminals and personal computers using Web3.0 applications. Generally, the content requested by the content requester to the content provider is divided into two types: one is various data in the blockchain, such as transaction records, smart contract status, and user identity information; the other is code (application binary interface and bytecode of smart contracts).

[0003] In the Web3.0 cache system, content providers include not only blockchains and decentralized storage with complete content, but also oracles with cache capabilities. An oracle is a decentralized node with storage space and computing power. It uses part of its storage space as a cache pool, specifically for storing those hot content in Web3.0. These hot content mainly refers to files, states, and codes that are frequently accessed and queried. When the oracle receives a content request, it searches for the required content in its own cache space. If the content that matches the received content request has been cached, the oracle will send the required content to the content requester. If not found, the corresponding content will be requested from the blockchain and decentralized storage network.

[0004] Many traditional deterministic cache replacement algorithms have been proposed to maximize cache hit rates, such as the least recently used algorithm, the first-in-first-out algorithm, the least frequently used algorithm, the most recently used algorithm, etc., all of which attempt to implement a reasonable and deterministic strategy under certain assumptions to maximize cache hit rates. However, these algorithms have their own defects and cannot produce significant hit rates in various situations because these traditional algorithms cannot adapt to all request situations.

[0005] To sum up, on the one hand, Web3.0 has not yet fully utilized caching technology, which to a certain extent restricts its performance and improvement; on the other hand, traditional caching strategies can no longer meet Web3.0's requirements for security, privacy, and decentralization; in addition, traditional content distribution networks have centralization risks. Summary of the invention

[0006] In view of this, the purpose of the present invention is to provide a distributed cache framework with better performance and higher security suitable for Web3.0 scenarios; by introducing an efficient cache framework to improve its performance, by redesigning a cache replacement strategy that is more in line with Web3.0 to meet the requirements of security, privacy, and decentralization, and by utilizing a distributed oracle network for data caching to reduce the hidden dangers of single point failure of cache nodes and data security.

[0007] The present invention achieves a technical solution to the purpose of the present invention by a distributed caching method and system that integrates reinforcement learning and collaborative learning to prove consensus.

[0008] In a first aspect of the present invention, the present invention provides a distributed caching method integrating reinforcement learning and collaborative learning proof consensus, comprising the following steps:

[0009] Step 1: The user node initiates a request to the oracle node through the application programming interface;

[0010] Step 2: The oracle node broadcasts a preparation message; the preparation message includes at least the value function and the immediate reward for a specific time period;

[0011] Step 3, the oracle node trains an indirect cache hit model based on a reinforcement learning network; the training of the indirect cache hit model based on a reinforcement learning network includes centralized training and distributed execution, the centralized training is implemented by a decentralized oracle node, the distributed execution is implemented by all oracle nodes, the decentralized oracle node is determined by a consensus mechanism, and the decentralized oracle node minimizes the sum of the reward similarities of the consensus of all oracle nodes in this specific time period; the reward similarity of the consensus of the oracle node in this specific time period is the proof of work in the specific time period calculated by the oracle node through collaborative learning proof;

[0012] Step 4: The oracle node broadcasts the indirect cache hit model based on the reinforcement learning network after training.

[0013] Step 5: The oracle node verifies the indirect cache hit model based on the reinforcement learning network after training.

[0014] Step 6: The oracle node caches the required content and responds to the user request. The cache obtains the required content including direct cache hits based on the local cache pool and indirect cache hits based on the reinforcement learning network.

[0015] In the second aspect of the present invention, the present invention also provides a distributed cache system that integrates reinforcement learning and collaborative learning to prove consensus, the system is used to implement the method described in the first aspect of the present invention, the system includes a decentralized ledger technology layer and an oracle decentralized network; the decentralized ledger technology layer includes a blockchain network and a decentralized storage network; the oracle decentralized network includes multiple oracles; the oracle includes an executor and a cache pool, the executor includes a data management module and a reinforcement learning agent; the cache pool includes a memory and a non-volatile solid-state memory.

[0016] Compared with the prior art, the present invention has the following significant advantages:

[0017] (1) The present invention designs a distributed cache framework based on multi-agent reinforcement learning, and establishes a structure in a modular way, which will not affect the original functions of the blockchain, thereby having the generalization ability for different cache hit scenarios;

[0018] (2) The present invention implements a cache replacement scheme based on multi-agent reinforcement learning. When the oracle node can only obtain partial observations, the cache task is completed by collaboratively training the neural network, and it also has the ability to resist single point failures.

[0019] (3) The present invention improves the centralized training distributed execution paradigm. That is, we propose a distributed training consensus called Proof of Cooperative Learning (PoCL). Based on the original centralized training distributed execution paradigm, decentralized oracle nodes are obtained through node consensus, and decentralized oracle nodes are used to replace centralized nodes for training to update the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] The accompanying drawings are used to provide a further understanding of the present invention and constitute a part of the specification. Together with the following specific embodiments, they are used to explain the present invention but do not constitute a limitation of the present invention. In the accompanying drawings:

[0021] Figure 1 It is a schematic diagram of a distributed cache architecture that integrates reinforcement learning and collaborative learning to prove consensus according to an embodiment of the present invention.

[0022] Figure 2 It is a flow chart of a distributed caching method for integrating reinforcement learning and collaborative learning to prove consensus according to an embodiment of the present invention.

[0023] Figure 3 It is a data structure diagram of the oracle management smart contract in an embodiment of the present invention.

[0024] Figure 4 It is a data structure diagram of the data management smart contract in an embodiment of the present invention.

[0025] Figure 5 It is a schematic diagram of a distributed cache system that integrates reinforcement learning and collaborative learning to prove consensus according to an embodiment of the present invention.

[0026] Figure 6 It is a comparison chart of the cache hit rate in the embodiment of the present invention and the hit rate of the traditional cache replacement solution.

[0027] Figure 7 Graph showing the relationship between the consensus success rate, node failure rate, and number of nodes in an embodiment of the present invention. DETAILED DESCRIPTION

[0028] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0029] In order to better explain the present invention, the important terms involved in the embodiments of the present invention are first explained:

[0030] Reinforcement learning: Reinforcement learning is a method of learning the optimal behavior strategy through the interaction between the agent and the environment based on the reward signal fed back by the environment. The agent performs actions in the environment, and the environment gives rewards or penalties based on the quality of the actions. The agent adjusts its behavior strategy based on these feedbacks to maximize the long-term accumulated rewards; there are many algorithms in reinforcement learning, some of which are typical algorithms: Q-learning, Deep Q-Network (DQN); further, for Multi-Agent Reinforcement Learning (MARL), VDN (Value-Decomposition Networks) algorithm and Qmix algorithm are both important algorithms in MARL, and they each have different characteristics and application scenarios. The VDN algorithm realizes the decomposition of the value function through a simple summation method, which is suitable for scenarios where the interaction between agents is relatively simple; while the Qmix algorithm integrates the local value function of each agent in a more complex nonlinear way to obtain a joint action value function, which is suitable for scenarios that require a higher level of collaboration and complex strategy coordination.

[0031] Consensus algorithm: Each node in the blockchain network confirms a batch of transactions through an algorithm and ensures that all nodes have consistent confirmation results for this batch of data. This algorithm is the consensus algorithm of the blockchain. Currently common consensus algorithms include PBFT, POW, POS, etc.

[0032] Oracle Node: Oracles play an important role in blockchain and distributed systems. They provide a mechanism that enables smart contracts or decentralized applications to interact securely with external data sources. Oracle nodes refer to entities or services that run oracle software and are responsible for introducing real-world data into the blockchain environment.

[0033] See also Figure 1 , an embodiment of the present invention provides a distributed cache framework structure diagram that integrates reinforcement learning and collaborative learning to prove consensus, such as Figure 1 As shown, it includes the request phase, preparation phase, training phase, synchronization phase, verification phase and response phase; in the request phase, the user initiates a request to the oracle node; in the preparation phase, the oracle node broadcasts the message; in the training phase, a decentralized oracle node is selected through a consensus mechanism to perform centralized training and distributed execution of the cache hit model; in the synchronization phase, the trained cache hit model is broadcast to each oracle node; in the verification phase, the cache hit model is verified; in the response phase, the oracle node responds to the user's request content based on the trained cache hit model or the local cache pool.

[0034] See also Figure 2 The present invention provides a distributed caching method integrating reinforcement learning and collaborative learning to prove consensus, comprising the following steps:

[0035] Step 1: The user node initiates a request to the oracle node through the application programming interface;

[0036] In an embodiment of the present invention, when a user generates content needs when using a Web3.0 application, the application will send a content request to the oracle node; the users here may include but are not limited to personal computers, mobile devices, Internet of Things devices, network devices, browsers and applications, etc., and the content of the request may be product prices, weather data, logistics information, etc.

[0037] In the embodiment of the present invention, it is considered that the request contents of user nodes are generally similar in the same geographical location. For example, in the same city recommendation of a certain APP software, users with close distances will have more similar requests. Therefore, the present invention clusters the oracle nodes according to the location information, thereby reducing the cost of repeated or similar calculations. Based on this, the method for constructing the oracle node includes:

[0038] The first node requests to join the oracle network through the oracle management smart contract, obtains a unique ID and provides location information and cache capacity, wherein the unique ID can distinguish different first nodes for subsequent consensus and verification, etc., the location information can be used to cluster similar nodes to reduce computing costs, and the cache capacity can provide a corresponding local cache pool for local cache hits; the first node is an ordinary node that has not joined the oracle network; the first node that joins is clustered according to the location information, the clustered first node is used as the oracle node, and the oracle node constitutes the oracle network, this method can divide the oracle nodes with similar characteristics or similar data requirements into the same cluster. In this way, when processing content requests, unified request content aggregation can be performed for the oracle nodes in each cluster, reducing unnecessary repeated calculations and data transmission, thereby improving the overall efficiency of content aggregation. Through a reasonable clustering strategy, the distribution and number of nodes can also be optimized to avoid waste of resources and further reduce the cost of aggregation.

[0039] In the embodiment of the present invention, Figure 3As shown, the Oracle ManageContract (OMC) is used to manage the joining or exit of oracle nodes and assign unique IDs to oracle nodes. It contains an attribute Oracles, which is used to record the location information of online oracles. In addition, the contract also includes 3 functions: Join(), Quit(), and Clustering(). Among them, ordinary nodes can call the Join() function to become oracle nodes; similarly, oracle nodes can call the Quit() function to change back to ordinary nodes. These methods promote the scalability and flexibility of the blockchain system, and avoid the same oracle node from being a node participating in model training for a long time, causing centralization risks.

[0040] Step 2: The oracle node broadcasts a preparation message; the preparation message includes at least the value function and the immediate reward for a specific time period;

[0041] In this embodiment of the present invention, at this stage, each oracle node will broadcast a prepare message, which contains the value function and instant rewards i ,Specifically, the definition of prepare message is: Among them, each oracle node O i Has a unique ID, t represents the time slot, PK i is the oracle node O i The public key of oracle node O i Use the private key SK i Sign the prepare message. Similarly, at this stage, the oracle node O i It also receives prepare messages from other oracle nodes. These messages are first verified by the public key to ensure their integrity. At the same time, the time slot t is checked to ensure that the message is not from an early time or a future time. i Add the verified prepare message to the message pool In the process, prepare messages that have not passed verification are directly discarded. After the preparation phase is over, the oracle node no longer receives prepare messages, and the message pool is sorted in a dictionary using the corresponding ID of the oracle node as the key value. It should be noted that since the embodiment of the present invention has clustered the oracle nodes in advance, it can be considered that the duration of the preparation phase is much longer than the time for preparing message transmission, and the time for preparing message transmission can be ignored.

[0042] Step 3, the oracle node trains an indirect cache hit model based on a reinforcement learning network; the training of the indirect cache hit model based on a reinforcement learning network includes centralized training and distributed execution, the centralized training is implemented by a decentralized oracle node, the distributed execution is implemented by all oracle nodes, the decentralized oracle node is determined by a consensus mechanism, and the decentralized oracle node minimizes the sum of the reward similarities of the consensus of all oracle nodes in this specific time period; the reward similarity of the consensus of the oracle node in this specific time period is the proof of work in the specific time period calculated by the oracle node through collaborative learning proof;

[0043] In an embodiment of the present invention, the indirect cache hit model based on the reinforcement learning network is constructed by the Qmix network. The QMIX (Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learning) network is a multi-agent reinforcement learning algorithm based on a value function, which is mainly used to solve the joint action optimization problem in the multi-agent decision-making system. The QMIX network uses the framework of centralized training distributed execution (CTDE) to achieve the use of global information optimization strategy in the training phase, while each agent only relies on local information to make decisions in the execution phase. Based on this, the indirect cache hit model based on the reinforcement learning network of the present invention includes:

[0044] Agent Network: Each agent, or oracle node, has an independent network that calculates its Q value based on its local observations. This network usually uses a recurrent neural network (RNN) structure, such as GRU, to capture time dependencies. Each agent has an independent network that represents its value function. This network usually uses a deep recurrent neural network to receive the local observations of the current time step and the action taken in the last step, and outputs the Q value of the agent in the current state.

[0045] Mixing network: A feedforward neural network that uses the output of the proxy network, i.e. the Q value of all agents and the global state as input to calculate the global Q value. The weights and biases of the network are generated by the hypernetworks, and the weights must be non-negative. This is to ensure that the global Q value output by the Mixing Network maintains a monotonic relationship with the Q value of each agent, thereby satisfying the monotonicity constraint.

[0046] Hypernetwork: A neural network used to generate the mixture parameters, whose input is the global state and whose output is the weights and biases of the mixture network.

[0047] In an embodiment of the present invention, the indirect cache hit model based on the reinforcement learning network of the present invention specifically includes:

[0048] The system runs in discrete time t∈{1,2,3…,T}. There are M oracle nodes in the system. In each time slot t, the environment state includes the cache state of all oracle nodes, user request information, and deadline constraints. The environment state space s t It can be described as:

[0049] s t = {Y t ,W t ,d t} (1)

[0050] in, Represents the oracle node O i The cache status, Represents the oracle node O i User request information, Represents the oracle node O i The maximum allowed delay.

[0051] At time slot t, each oracle node O i Get its observed status, which includes the current cache status of the oracle, user request information and deadline constraints. Specifically, each oracle node i observes the following status:

[0052]

[0053] in, is the oracle node O i The cache status, is the oracle node O i Requests for User Content received, is the oracle node O i The maximum allowed delay in serving content.

[0054] At the beginning of each time slot t, the oracle needs to update the local cache. i Cache replacement operation performed at time slot t The definition is as follows:

[0055]

[0056] in, Represents the oracle node Oi Determine cache content f; Represents the oracle node O a Decide not to cache content f; the joint action of the oracle network at time slot t is defined as:

[0057] The reward item is calculated by the actual delay and the maximum allowed delay of the oracle node in a specific time period; the penalty item is calculated according to the cumulative hit count of the cached content of the oracle node in a specific time period; the reward of the oracle node is obtained by the weighted sum of the reward item and the penalty item; therefore, the oracle node O i The reward function is:

[0058]

[0059] in, is the oracle node O i The reward function at time slot t is, is the oracle node O i The reward item at time slot t is as follows: is the oracle node O i The penalty term at time slot t is as shown in formula (6), where λ is the balancing coefficient and should satisfy λ≥0.

[0060]

[0061] Among them, h(t) is the oracle node O i The cumulative hit count of the cache content in time slot t, w is the weight coefficient.

[0062]

[0063] in, is the oracle node O i The actual delay in time slot t, μ, ρ, k, δ are the hyperparameters of the penalty. Hyperparameters refer to the parameters set before starting the learning process, rather than the parameters obtained through training. They should satisfy μ≤0, ρ<0 and k≥1, δ≤0. The specific values ​​can be adjusted accordingly by technicians in this field according to the actual situation.

[0064] The reward function of the system:

[0065]

[0066] By summing up the reward functions of the M oracle nodes at each time slot t, we can get the system’s reward function, which can promote cooperative behavior among intelligent agents.

[0067] In the traditional training phase, the centralized node updates its parameters based on the global state and the actions of all agents in order to better estimate the value of the joint action. At the same time, each distributed node selects an action based on the current observation, and interacts with the environment to obtain new observations, rewards and other information; the interaction data is stored in the experience replay pool. Data is sampled from the experience replay pool, and TD-Error (temporal difference error) is used to update the centralized hybrid network and the network of each agent. In the execution phase, each agent selects an action based only on its local observations, and can execute the corresponding action without global information.

[0068] The present invention improves the centralized training distributed execution paradigm, that is, it proposes a distributed training consensus of Proof of Cooperative Learning (PoCL). On the basis of the original centralized training distributed execution paradigm, decentralized oracle nodes are obtained through node consensus, and decentralized oracle nodes are used to replace centralized nodes for training to update the model.

[0069] In an embodiment of the present invention, the decentralized oracle node makes the sum of the reward similarities of all oracle nodes in the consensus in this specific time period as minimal, including the oracle node receiving a preset number of preparation messages; calculating the reward values ​​of a preset number of oracle nodes based on the preset number of preparation messages; calculating the reward similarity of the current oracle node in the consensus in this specific time period based on the similarity distance between the reward values ​​of the current oracle node and the preset number of oracle nodes; and calculating the decentralized oracle node that minimizes the sum of the reward similarities of all oracle nodes in the consensus in this specific time period based on the reward similarity of the preset number of oracle nodes in the consensus in this specific time period.

[0070] Specifically, each oracle node O i From the message pool Take out the oracle node O j The immediate reward r of (j≠i) j and value function And pass this value function through the test set to get the test reward R j,test With instant rewards j and test reward R j,test Then, record the oracle node O j The reward is R j =(R j,test ,r j ). Oracle Node O i Through the above steps, you can get your own reward R i =(R i,test ,r i ).

[0071] After obtaining the reward data, the similarity between the oracle nodes can be determined by evaluating the similarity between the reward data of the oracle nodes; the commonly used similarity methods are: Minkowski distance, Mahalanobis distance, correlation coefficient and angle cosine, the above methods can measure the similarity of data, among which, the smaller the similarity distance, the greater the similarity of the reward data of the oracle nodes. After selecting the method for evaluating similarity, the condition for using the above method to evaluate data in a distributed environment is that the data has the same distribution. In view of the regional and temporal locality characteristics exhibited by the cache, the present invention believes that within a similar time window, the rewards R obtained by all honest oracles in this network show a high degree of similarity, that is, the data output by the honest oracle obeys the same distribution. In the aforementioned preparation stage, the oracle node O i Total received Prepare messages, so the above similarity calculation process is calculated m-1 times to get the oracle node O i Similarity in this consensus is defined as:

[0072]

[0073] According to the above formula, we can get the similarity set L = {L1, L2, ..., L m}, the similarity set is unique in each consensus and all oracle nodes maintain the same similarity set. Even if an oracle node tampers with local data, it will not affect the results of other oracle nodes.

[0074] By minimizing the sum of the similarities of the rewards of all oracle nodes in this specific time period, decentralized oracle nodes can be obtained through consensus. The minimization function is expressed as:

[0075] O train =argmin i L (9)

[0076] Among them, O train Represents a decentralized oracle node, which is also a training node for centralized training.

[0077] It can be found that the decentralized oracle nodes selected in the above embodiment may have a phenomenon, that is, a certain oracle node may be selected for multiple consecutive rounds, which will lead to a loss of randomness, thereby causing a centralization risk, which is not conducive to the security of the cache system. In view of this, the preferred embodiment of the present invention also calculates the attenuation factor according to the number of times a certain oracle node becomes a decentralized oracle node in succession; according to the attenuation factor, the reward similarity of the oracle node after attenuation in the consensus of this specific time period is calculated; the decentralized oracle node is obtained by minimizing the sum of the reward similarities of all oracle nodes in the consensus of this specific time period.

[0078] The calculation formula used to calculate the reward similarity of the oracle node consensus in this specific time period after attenuation based on the attenuation factor is expressed as:

[0079]

[0080] Among them, L′ i It represents the reward similarity of oracle node i in the consensus of this specific time period after attenuation. α represents the first parameter, α∈(0,1], ∈ represents the second parameter, and h represents the number of consecutive decentralized oracle nodes. For example, if the oracle node is selected as a decentralized training node in the last round of consensus, then h=1, and the values ​​of other oracle nodes are 0; if the oracle node is selected as a decentralized training node in the last two rounds of consensus, then h=2, and the values ​​of other oracle nodes are 0, and so on. constitutes the attenuation factor; L i Indicates the reward similarity of the oracle node i in the consensus of this specific time period.

[0081] According to the above formula, we can get the attenuated similarity set L′={L′1,L′2,…,L′ m}, similarly, the minimization function at this time is expressed as:

[0082] O train =argmin i L′ (11)

[0083] This optimization method avoids the possibility of the same oracle node being selected continuously to a certain extent, allowing the selected decentralized oracle node to update its parameters based on the global state and the actions of all intelligent agents in order to better estimate the value of the joint action.

[0084] Step 4: The oracle node broadcasts the indirect cache hit model based on the reinforcement learning network after training.

[0085] In the embodiment of the present invention, during the synchronization phase, the decentralized oracle node Otrain The value function obtained by training They are sent to the corresponding oracle nodes respectively, and the synchronization message sync is defined as First, use the other party's public key PK i For message M train Encrypt to get M′ train Then use your own private key to encrypt M′ train Sign. The other party receives M′ train Then use the local model to aggregate the O train The public key of train Decrypted to get M train , extract the value function in the message

[0086] In some preferred embodiments, in the event that the oracle node is offline, the training node will send it an empty message for synchronization. The empty message is defined as This completes the synchronization phase.

[0087] Step 5: The oracle node verifies the indirect cache hit model based on the reinforcement learning network after training.

[0088] In the embodiment of the present invention, during the verification phase, the local test set is used to verify Get R′ i,test , if R′ i,test >R i,test It is considered to be improved, so the verification message is broadcast<COMMIT,id,t,true> , otherwise no, so broadcast<COMMIT,id,t,false> .

[0089] When receiving the verification message commit, first verify the message. The conditions for passing the verification are: the signature of the message is correct and the sender is from When the number of verified commit messages is greater than 2f (excluding itself) and<COMMIT,id,t,true> When the number of is greater than f+1 (including itself), the local model is updated, otherwise it is considered that the training node may be a Byzantine node and the update is abandoned.

[0090] Step 6: The oracle node caches the required content and responds to the user request. The cache obtains the required content including direct cache hits based on the local cache pool and indirect cache hits based on the reinforcement learning network.

[0091] In an embodiment of the present invention, the oracle node caches the required content and then responds to the user request, including:

[0092] Step 6.1: After receiving the content request, the oracle node queries the cache pool to see if the corresponding content exists; if it exists, it directly caches the hit based on the local cache pool and directly proceeds to step 6.6; otherwise, proceeds to step 6.2;

[0093] Step 6.2: The data management component of the oracle node queries the decentralized file identifier corresponding to the file hash through the data management smart contract in the blockchain;

[0094] In the embodiment of the present invention, Figure 4 As shown in the figure, the Data Manage Contract (DMC) contains an attribute DataMapping, which is used to record the mapping of file hashes to decentralized identifiers; it also includes two functions: Register() and Mapping(). Among them, the Register() function is used to register decentralized applications to this cache system, and Mapping() is used to map file hashes to decentralized identifiers.

[0095] Step 6.3: The data management smart contract will return the file identifier corresponding to the file hash to the oracle node;

[0096] Step 6.4: The data management component of the oracle node uses the application program interface to download the file from the decentralized storage to the local computer.

[0097] Step 6.5: After the file is downloaded, the oracle node makes a decision based on the indirect cache hit model of the reinforcement learning model: if the cache pool is not full, the new content will be added; if it is full, part of the old content will be replaced;

[0098] Step 6.6: The oracle node responds to the user's request.

[0099] like Figure 5 As shown, the present invention provides a distributed cache system that integrates reinforcement learning and collaborative learning to prove consensus, and the cache system is divided into two layers: the first layer is a decentralized ledger technology layer composed of blockchain and decentralized storage network; the second layer is a decentralized network composed of multiple oracles, the oracle includes an executor and a cache pool, the executor includes data management, reinforcement learning agents, etc.; the cache pool is composed of two types of hardware: memory and non-volatile solid-state memory.

[0100] Example 1

[0101] This embodiment illustrates the performance of the solution proposed in the present invention in combination with simulation results. The simulation conditions are as follows: it is performed on a computer with Windows 11 Pro operating system, 2.1 GHz Intel Core i7-12700 processor, 16 GB DDR4 RAM, and 256 GB NVMe SSD.

[0102] In the simulation, after receiving a request, the oracle uses the results of centralized training to update the network parameters and make action selections (cache replacement). The ratio of the number of request hits to the total number of requests is the hit rate of the oracle cache.

[0103] Figure 6 Comparing the cache hit rate of the present invention with the hit rate of the traditional cache replacement scheme, in this figure simulation, there are three oracle nodes (intelligent agents), the cache size of these nodes is 50, there are 100 types of request files, each training node receives 1000 requests, the distribution of request files conforms to Zipf's law, and a total of 100 rounds of training are performed. Finally, the cache hit rate is tested on the test set using the trained network. As can be seen from the figure, the cache replacement method we provide can achieve a higher cache hit rate than the traditional cache replacement strategy.

[0104] Figure 7 The relationship between the consensus success rate and the node failure rate and the number of nodes is shown. The horizontal axis of the figure is the node failure rate p f , the vertical axis is the consensus success rate, and different curves represent different numbers of nodes m. As can be seen from the figure, each curve has a threshold. f Gradually increase, the consensus success is first maintained at 100%, and then when p f When it is greater than this threshold, the consensus success rate drops steeply until it reaches 0. It can also be found that this threshold is related to the number of nodes. The more nodes there are, the larger the threshold.

[0105] A person skilled in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by instructing the relevant hardware through a program, and the program can be stored in a computer-readable storage medium, which can include: ROM, RAM, disk or CD, etc.

[0106] The technical features of the above embodiments can be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0107] Those skilled in the art will appreciate that the logic and / or steps represented in the flowchart or otherwise described herein, for example, may be considered as an ordered list of executable instructions for implementing logical functions, and may be specifically implemented in any computer-readable storage medium for use by an instruction execution system, apparatus, or device (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, apparatus, or device and execute instructions), or used in conjunction with such instruction execution systems, apparatuses, or devices.

[0108] It should be understood that the various parts of the present invention can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, a plurality of steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or a combination thereof: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0109] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "examples", "specific examples", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. Although embodiments of the present invention have been shown and described, it will be understood by those of ordinary skill in the art that various changes, modifications, substitutions and variations may be made to these embodiments without departing from the principles and spirit of the present invention, and the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A distributed caching method that integrates reinforcement learning and collaborative learning to prove consensus, characterized in that: The following steps are involved: Step 1: The user node initiates a request to the oracle node through the application programming interface; Step 2: The oracle node broadcasts a preparation message; the preparation message includes at least the value function and the immediate reward for a specific time period; Step 3, the oracle node trains an indirect cache hit model based on a reinforcement learning network; the training of the indirect cache hit model based on a reinforcement learning network includes centralized training and distributed execution, the centralized training is implemented by a decentralized oracle node, the distributed execution is implemented by all oracle nodes, the decentralized oracle node is determined by a consensus mechanism, and the decentralized oracle node minimizes the sum of the reward similarities of the consensus of all oracle nodes in this specific time period; the reward similarity of the consensus of the oracle node in this specific time period is the proof of work in the specific time period calculated by the oracle node through collaborative learning proof; Step 4: The oracle node broadcasts the indirect cache hit model based on the reinforcement learning network after training. Step 5: The oracle node verifies the indirect cache hit model based on the reinforcement learning network after training. Step 6: The oracle node caches the required content and responds to the user request. The cache obtains the required content including direct cache hits based on the local cache pool and indirect cache hits based on the reinforcement learning network.

2. According to claim 1, a distributed caching method integrating reinforcement learning and collaborative learning to prove consensus is characterized in that: The construction method of the oracle node includes: The first node requests to join the oracle network through the oracle management smart contract, obtains a unique ID and provides location information and cache capacity; The first nodes that join are clustered according to their location information, the clustered first nodes are used as oracle nodes, and the oracle nodes form an oracle network.

3. According to claim 1, a distributed caching method integrating reinforcement learning and collaborative learning to prove consensus is characterized in that: The decentralized oracle node minimizes the sum of the similarities of the rewards of all oracle nodes in the consensus in this specific time period, including that the oracle node receives a preset number of preparation messages; based on the preset number of preparation messages, the reward values ​​of the preset number of oracle nodes are calculated; based on the similarity distance between the reward values ​​of the current oracle node and the preset number of oracle nodes, the reward similarity of the current oracle node in the consensus in this specific time period is calculated; based on the reward similarity of the preset number of oracle nodes in the consensus in this specific time period, the decentralized oracle node corresponding to the minimum sum of the similarities of the rewards of all oracle nodes in the consensus in this specific time period is calculated.

4. According to claim 3, a distributed caching method integrating reinforcement learning and collaborative learning to prove consensus is characterized in that: The decentralized oracle node minimizes the sum of the similarities of the rewards of all oracle nodes in the consensus in this specific time period, including calculating the attenuation factor according to the number of times a certain oracle node has continuously become a decentralized oracle node; according to the attenuation factor, calculating the reward similarity of the oracle node in the consensus in this specific time period after attenuation; the decentralized oracle node is obtained by minimizing the sum of the similarities of the rewards of all oracle nodes in the consensus in this specific time period.

5. A distributed caching method for integrating reinforcement learning and collaborative learning to prove consensus according to claim 4, characterized in that: The calculation formula used to calculate the reward similarity of the oracle node consensus in this specific time period after attenuation based on the attenuation factor is expressed as: Among them, L ′ i represents the reward similarity of the oracle node i in the consensus of this specific time period after attenuation, α represents the first parameter, ∈ represents the second parameter, and h represents the number of consecutive decentralized oracle nodes; L a Indicates the reward similarity of the oracle node i in the consensus of this specific time period.

6. A distributed caching method for integrating reinforcement learning and collaborative learning to prove consensus according to claim 1, characterized in that: The calculation formula for the reward includes calculating the reward item based on the actual delay of the oracle node in a specific time period and the maximum allowed delay; calculating the penalty item based on the cumulative number of hits on the cached content of the oracle node in a specific time period; and obtaining the reward of the oracle node based on the weighted sum of the reward item and the penalty item.

7. A distributed caching method for integrating reinforcement learning and collaborative learning to prove consensus according to claim 6, characterized in that: The calculation formula of the penalty term is expressed as: in, is the oracle node O i The penalty term at time slot t is, is the oracle node O i The actual delay at time slot t, is the oracle node O i The maximum allowed delay in time slot t, is the oracle node O i The reward term at time slot t, μ, ρ, k, δ are the hyperparameters of the penalty. Hyperparameters refer to the parameters set before starting the learning process, satisfying μ≤0, ρ<0 and k≥1, δ≤0.

8. According to claim 1, a distributed caching method integrating reinforcement learning and collaborative learning to prove consensus is characterized in that: The oracle node caches the required content and responds to the user request including: Step 6.1: After receiving the content request, the oracle node queries the cache pool to see if the corresponding content exists; if it exists, it directly caches the hit based on the local cache pool and directly proceeds to step 6.6; otherwise, proceeds to step 6.2; Step 6.2: The data management component of the oracle node queries the decentralized file identifier corresponding to the file hash through the data management smart contract in the blockchain; Step 6.3: The data management smart contract will return the file identifier corresponding to the file hash to the oracle node; Step 6.4: The data management component of the oracle node uses the application program interface to download the file from the decentralized storage to the local computer. Step 6.5: After the file is downloaded, the oracle node makes a decision based on the indirect cache hit model of the reinforcement learning model: if the cache pool is not full, the new content will be added; if it is full, part of the old content will be replaced; Step 6.6: The oracle node responds to the user's request.

9. A distributed cache system integrating reinforcement learning and collaborative learning to prove consensus, which is applied to a distributed cache method integrating reinforcement learning and collaborative learning to prove consensus as described in any one of claims 1 to 8, characterized in that: It includes a decentralized ledger technology layer and a decentralized oracle network; the decentralized ledger technology layer includes a blockchain network and a decentralized storage network; the decentralized oracle network includes multiple oracles; the oracle includes an executor and a cache pool, the executor includes a data management module and a reinforcement learning agent; the cache pool includes a memory and a non-volatile solid-state memory.

Images