Block chain oracle machine node selection method based on reinforcement learning

Abstract

The invention relates to the technical field of reinforcement learning, and provides a reinforcement learning-based block chain oracle machine node selection method. According to the method, a Bayesian cost-dependent reputation value model is established, and a reinforcement learning Actor-Critic algorithm is used for making a decision; a service user initiates a block chain transaction or a smart contract calls a node selection contract, and the node selection contract calls an AC contract to initialize reputation values and cost values of all oracle machine nodes into a state space set; the AC contract calls an Actor network to select an optimal oracle machine node according to a strategy; and after the decision result is returned to the oracle node group for execution, a result set is returned, a reward value is input into the Critic network to evaluate a strategy value, and the Critic network evaluates an action score to guide the Actor network and update a state space set so as to select an optimal strategy.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of reinforcement learning, and in particular to a blockchain oracle node selection method based on reinforcement learning. BACKGROUND

[0002] Blockchain technology groups data into consecutive blocks and connects them in chronological order into a chain structure, forming a distributed ledger that is tamper-proof and unforgeable. Its integrity and authenticity are guaranteed through cryptographic methods. In a blockchain network, nodes ensure data consistency through consensus algorithms. Ethereum, as a representative of blockchain 2.0, introduces the concept of smart contracts and provides a virtual execution environment. Smart contracts are programs deployed on the blockchain that can automatically execute when certain conditions are met, without the need for third-party intervention to achieve predictable results. These smart contracts have been applied in various industries, including insurance, entertainment, energy, education, and healthcare. To ensure the consistency of smart contract execution across different nodes, direct access to network or real-world data, including current time, is not possible. Smart contracts need to obtain real-world data through an oracle mechanism. For example, sending a transaction to a smart contract address can trigger its execution. These transactions and the resulting state changes are collected by nodes with recording authority and packaged into new blocks at regular intervals. These blocks are connected through hash pointers, forming a chain structure.

[0003] Blockchain technology plays an important role in finance, supply chain management, data sharing, and other fields due to its unique characteristics of decentralization, tamper resistance, and consensus mechanism. The oracle node of the blockchain network connects the on-chain and off-chain blockchain environments through an interface with external data providers, and plays a key role in smart contracts by providing trusted data. Therefore, its credibility is crucial.

[0004] However, connecting blockchain with centralized oracles may undermine the decentralized nature of the system. As a trusted entity, the oracle has the "privilege" to provide unconditionally accepted data, leading to the "oracle problem". That is, for blockchain data processing requests that require real-time information outside the blockchain for combined processing, existing blockchain data processing methods cannot guarantee the authenticity of external data, and thus cannot effectively and timely process blockchain data processing requests. SUMMARY

[0005] In view of the problems in the prior art, the application provides a blockchain oracle node selection scheme based on reinforcement learning, which can effectively obtain external data of a blockchain and improve the timeliness, authenticity and reliability of the obtained data, thereby improving the efficiency, accuracy and reliability of data processing requests of an application relying on real-time information outside the blockchain when accessing the blockchain.

[0006] The application provides a blockchain oracle node selection method based on reinforcement learning, which comprises the following steps:

[0007] Step 1: a service user sends a service request to a node selection contract through a blockchain network;

[0008] Step 2: the node selection contract calls an AC contract according to the service request, the AC contract establishes a reputation value and cost value model based on Bayesian cost dependence for all oracle nodes and inputs an agent as a state space set; wherein the agent comprises an Actor network and a Critic network; the Actor network is used for generating an oracle node selection strategy; the Critic network is used for evaluating the state value of the strategy generated by the Actor network and reversely propagating the state value to the Actor network;

[0009] Step 3: the Actor network obtains the current state space set, calculates the probability distribution of being selected for each oracle node, randomly extracts an oracle node as an optimal node based on the probability distribution, verifies the optimal node according to the reputation value, and submits the optimal node to an Oracle node group after verification;

[0010] Step 4: the Oracle node group assigns a task of the service request to the optimal node, calculates a reward value after task execution and updates the state space set, and returns the reward value and the updated state space set to the agent;

[0011] Step 5: the Critic network calculates the state value according to the state space set before and after updating and the reward value, calculates a TD error based on the state value, takes the square sum of the TD error as a loss function of the Critic network to update the Critic network parameter, and returns the TD error to the Actor network, and the Actor network updates the Actor network parameter according to the TD error;

[0012] Step 6: steps 3 to 5 are repeated until the agent converges to an optimal strategy, and an oracle node is selected according to the optimal strategy to complete the service request.

[0013] Further, in step 2, the AC contract establishes a reputation value and cost value model based on Bayesian cost dependence for all oracle nodes, specifically including:

[0014] The number k of oracle nodes of the Oracle node group is determined, the prior distribution of the k oracle nodes is initialized using Beta(1, 1) distribution, the initial probability of each oracle node is calculated and normalized, and the expectation of the Beta distribution is calculated to represent the reputation value of the oracle node, while the cost value of the oracle node is set.

[0015] Further, the step 3 specifically includes:

[0016] The Actor network calculates the probability distribution of each oracle node being selected according to the current state space set and the Actor network parameters, and randomly selects a node as the best node according to the probability distribution, and calculates the reputation value of the best node; if the reputation value is higher than the threshold, two oracle nodes are additionally randomly selected to participate in the training of the model, otherwise the oracle node with the highest reputation value is directly selected as the best node to complete the service request.

[0017] Further, the step 4 specifically includes:

[0018] The Oracle node group distributes the task after obtaining the three selected nodes, and after distributing the service request to the three nodes, the three nodes simultaneously call an external data source interface to obtain a data set;

[0019] The Oracle node group uses the task completion result and cost value of the oracle node to calculate the reward value: when the oracle node correctly completes the task, the reward value generated by the environment is calculated as , when the task is not completed, the reward value is calculated as r k = -μ·c k , where μ represents a reward factor, c k represents the cost value and c k >1;

[0020] The state space set is updated, and the Oracle node group returns the reward value and the updated state space set to the agent.

[0021] Further, the Oracle node group uses the task completion result and cost value of the oracle node to evaluate the reward value, specifically including:

[0022] The service request task is distributed to three nodes o h , o m and o n ; wherein oh the best node;

[0023] If the task completion result of o h is consistent with the task completion result of o m or o n , the task completion result of o h is output, and the reward values r h , r m and r n are calculated according to the formula

[0024] If the task completion result of o h is inconsistent with the task completion result of o m or o n , and the task completion results returned by the node o m and the node o n are consistent, the consistent task completion result is output, and the reward value r k generated by o k is calculated according to the formula r h = -μ·c h , and the reward values r m and r n are calculated using

[0025] If the task completion results of the three nodes are all inconsistent, the task completion result of o h is output, and the reward values r h , r m and r n are all set to 0.

[0026] Further, in step 5, the state value function formula is where R t is the future return starting from the time step t, and the calculation formula is where γ represents a decay factor;

[0027] The TD error calculation formula is: δ t = r t + γV w (S t+1 ) - V w (S t ), where δ t represents a TD error value, r t is a reward value, V w (S t+1 ) represents a state value under a state space set S t+1 , and V w (S t ) represents a state value under a state space set S t ​​The state value is given by w, which represents the policy of the Critic network parameters, and γ represents the decay factor, where γ∈[0,1], r t+k This represents the reward value at time t+k, where k=0 represents the reward value at the current time, and k>0 represents the predicted reward value.

[0028] The loss function of the Critic network is as follows: Where L represents the loss function.

[0029] Furthermore, in step 5, the gradient of the Critic network parameter w is updated using the gradient descent algorithm. And based on the gradient value Update the Critic network parameters w. Where w t+1 This represents the updated network parameters w, w t Let w and λ represent the network parameters before the update. w The parameter w represents the learning rate.

[0030] Furthermore, in step 5, the Actor network calculates using the policy gradient method. Where θ represents the Actor network parameters, using Update the Actor network parameters to improve the expected return under the current policy, where λ θ π represents the learning rate. θ (a t |s t ) represents the probability distribution of the Actor network output.

[0031] Furthermore, in step 6, the convergence rule is: when the mean square value of the TD error is less than 0.001 for 50 consecutive times, it is determined to be converged.

[0032] The beneficial effects of this invention are as follows:

[0033] This invention operates on a blockchain smart contract, ensuring transparency and immutability. This means that once deployed, the selection and reputation assessment processes for oracle nodes will be fixed on the blockchain, significantly improving the fairness and reliability of the system and preventing external interference and manipulation.

[0034] This invention utilizes a Bayesian reputation model to dynamically update the reputation value of each oracle node based on historical performance and interaction records. This quantification method provides a reliable way to evaluate the trustworthiness of nodes, thereby optimizing node selection and enhancing the accuracy and reliability of data.

[0035] The application adopts the Actor-Critic method, takes the selection of the oracle node as the action, and the observation value of the environment includes the reputation value and the cost value of the current node. The agent can efficiently learn and make decisions in a complex environment. The actor is responsible for policy updating, and the critic transmits the value evaluation to the actor network through back propagation, so that the agent can continuously improve its selection strategy, and the agents can quickly and stably converge, thereby improving the selection efficiency of the oracle node and the overall performance of the system.

[0036] The application takes the cost of the node into account when generating the environment reward, which can optimize resource allocation and ensure the selection of the node with the highest cost performance. This economic consideration not only improves the efficiency of the system, but also reduces the overall operating cost, making the oracle system more sustainable in the long run. By comprehensively evaluating performance and cost, the best node can be selected more intelligently to achieve better economic benefits. BRIEF DESCRIPTION OF DRAWINGS

[0037] Figure 1 A flowchart of a blockchain node selection method based on reinforcement learning provided by an embodiment of the application is provided.

[0038] Figure 2 A timing diagram of a blockchain node selection method based on reinforcement learning provided by an embodiment of the application is provided. DETAILED DESCRIPTION

[0039] To make the purpose, technical solutions and advantages of the application clearer, the technical solutions in the embodiments of the application will be described in detail below with reference to the drawings in the embodiments of the application. Obviously, the described embodiments are part of the embodiments of the application, not all the embodiments. Based on the embodiments in the application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the application.

[0040] As shown in Figure 1 A blockchain node selection method based on reinforcement learning provided by an embodiment of the application uses a Bayesian cost-dependent model, an Actor-Critic reinforcement learning algorithm and a smart contract to efficiently and safely select reliable oracle nodes and obtain off-chain data. The Bayesian cost-dependent reputation model includes the reputation value and the cost value of the oracle node; the Actor-Critic reinforcement learning algorithm belongs to part of the AC contract, which is used to complete the selection and learning of the oracle node.

[0041] In this embodiment, the service user, the node selection contract, the AC contract, the Oracle node group and the external data source are included, the data transmission between the node selection contract, the AC contract and the Oracle node group is completed on the chain, and the tamper resistance of the task result is guaranteed. The node selection contract is used for selecting the service of the node for the service user through the blockchain transaction or the smart contract, the AC contract is used for initializing the Bayesian cost-dependent reputation model for all blockchain oracle nodes, the Oracle node group includes the oracle node, is used for obtaining the information from the external data source and uploading to the blockchain network, and provides the external data for the smart contract. As shown in Figure 2 The method comprises the following steps:

[0042] Step 1: The service user sends a service request to the node selection contract through the blockchain network.

[0043] The service request includes the query requirement for the external data source, and the node selection contract is responsible for managing the requested service of the user.

[0044] Step 2: The node selection contract calls the AC contract according to the service request, and the AC contract establishes the reputation value and cost value model based on the Bayesian cost dependence for all oracle nodes and inputs the agent as the state space set.

[0045] The agent includes an Actor network and a Critic network; the Actor network is used for generating the oracle node selection strategy; the Critic network is used for evaluating the state value of the strategy generated by the Actor network and propagating the state value to the Actor network in the reverse direction. The size of the state space set depends on the number of the Oracle node group.

[0046] Specifically, the node selection contract records and submits the user ID of the service request to the AC contract, and the AC contract needs to collect a certain fee after receiving the calling request of the node selection contract, so as to cover the cost of calling the oracle service.

[0047] Step 3: The Actor network obtains the current state space set, calculates the probability distribution of each oracle node being selected, and randomly extracts an oracle node as the best node based on the probability distribution, verifies according to the reputation value, and submits the best node to the Oracle node group after the verification is passed.

[0048] Specifically, the Actor network obtains the current reputation value and the cost value as the observation value of the agent, and converts the observation value into a state space set, observes the reputation value and the cost value of the node, and calculates the probability distribution of each node being selected. The probability distribution is used as a weight to weight each oracle node, and then a oracle node is randomly selected as the best node, and the result of the best node is verified. If the threshold requirement is not met, the node with the highest reputation value is directly selected as the best node, and the best node is submitted to the Oracle node group.

[0049] Step 4: The Oracle node group allocates the service request task to the best node, calculates the reward value after the task is executed, and updates the state space set. The reward value and the updated state space set are returned to the agent.

[0050] Specifically, the best node returns the state space set and the reward set to the agent after signing them with the private key of the node.

[0051] Step 5: The Critic network calculates the state value according to the updated state space set and the reward value, calculates the TD error based on the state value, takes the square sum of the TD error as the loss function of the Critic network to update the Critic network parameters, and returns the TD error to the Actor network. The Actor network updates the Actor network parameters according to the TD error.

[0052] Specifically, the Critic network returns the result set to the node selection contract after verifying the signature, and completes the service request of the user.

[0053] Step 6: Repeat steps 3 to 5 until the Actor network and the Critic network converge to the optimal strategy, and select the oracle node to complete the service request according to the optimal strategy. The convergence rule is that when the mean square value of the TD error is less than 0.001 for 50 consecutive times, it is determined to be converged.

[0054] The blockchain oracle node selection method provided by the embodiment of the application can efficiently obtain external data of the blockchain, and significantly improve the timeliness, authenticity and reliability of the data. By optimizing the node selection mechanism, the efficiency, accuracy and reliability of data processing requests when an application relying on real-time information outside the blockchain accesses the blockchain again can be greatly improved, thereby providing strong support for stable operation and efficient interaction of related references.

[0055] On the basis of the above embodiment, the embodiment provides specific steps of the AC contract for establishing a reputation value and cost value model based on Bayesian cost dependence for all oracle nodes.

[0056] Determine the number of Oracle node groups k, the prior distribution of k Oracle nodes is initialized using Beta(1, 1) distribution, calculate the initial probability of each Oracle node and normalize the initial probability, calculate the expected value of Beta distribution to represent the reputation value of the Oracle node, and set the cost value of the Oracle node.

[0057] Specifically, assuming that there are k Oracle nodes in the Oracle node group, the prior distribution of k Oracle nodes is initialized using Beta distribution, and the Beta distribution M can be represented as: Where x represents a random variable, a i ,β i represents the parameter of the Beta distribution of the i-th node, where a i represents the number of times that node i correctly completes the task, and β i represents the number of times that node i incorrectly completes the task.

[0058] First, the prior distribution B i of each Oracle node is initialized as Beta(1; 1), where i represents the index of the node, i = 1, 2, …, k, to represent a uniform prior probability, and the initial probability of the node is calculated by the formula , where P i represents the initial probability of the i-th node being selected.

[0059] In order to ensure that the sum of the probabilities of all nodes is 1, it is also necessary to normalize all the initialized probabilities: calculate the total probability of all nodes Then normalize the probability of each node The probability of the node being selected reflects the node selection tendency based on the high reputation and low cost of the node. Since it is uniformly initialized, the initial probability P i of each Oracle node is normalized to The normalized initial probability P i ′ is used as the weight.

[0060] At initialization, the Beta distribution is actually a constant distribution, and the probability density of each x is 1. However, with the feedback of system operation and node performance, these weights P i ′ will be adjusted according to the actual performance of the node, and if the node consistently provides accurate data within a certain period of time, the corresponding weight P i ′ increases; if the node provides inaccurate data, the weight P i ′ decreases. Thus, the Beta distribution reflects the reputation value of each node At the same time, the cost value c of all Oracle nodes is initialized to c i, i = 1, 2, …, k, the reputation value and the cost value of all the oracle nodes are input into the Actor-Critic network as the state space set of the agent. The cost value of the oracle node is determined according to the on-chain transaction fee, data acquisition cost, node operation cost and security protection cost of each oracle node.

[0061] On the basis of the above-mentioned embodiment, the embodiment provides an implementation of step 3, comprising:

[0062] The Actor network calculates the probability distribution of the selection of each oracle node according to the current state space set and the Actor network parameter θ, and randomly selects a node as the best node according to the probability distribution, and calculates the reputation value of the best node; if the reputation value is higher than the threshold value, two oracle nodes are additionally randomly selected to participate in the training of the model, otherwise the oracle node with the highest reputation value is directly selected as the best node to complete the task of the service request.

[0063] Specifically, the Actor network calculates the probability distribution π θ (a t |s t ) of the selection of each oracle node according to the current state space set and the Actor network parameter θ, weights each oracle node by taking the probability distribution as the weight, and randomly selects a node as the best node and then judges the reputation value thereof, if the reputation value is higher than the threshold value, two nodes are additionally randomly selected to participate in the training of the task model, otherwise the node with the highest reputation value is directly selected to complete the task. It can be understood that by weighted random sampling, the pure greedy strategy can be avoided, and the low-probability node still has the opportunity to be selected.

[0064] The Actor network outputs the parameters α i , β i and the weight P i of each Beta distribution through forward propagation, wherein the weight P i represents the probability distribution π θ (a t |s t ) of the selection of the oracle node. Assuming that the best node o h is selected, it is judged whether the reputation value of the node o h is higher than the threshold value (0.7): if the reputation value E(h) is higher than the threshold value, the nodes o m and o n are additionally randomly selected as auxiliary nodes to jointly participate in the completion of the task and the training of the model. If the reputation value E(h) is lower than the threshold value, the node o max with the highest reputation value is directly selected as the best node and submitted to the Oracle node group.

[0065] On the basis of the above-mentioned embodiments, the embodiments provide an implementation of step 4, comprising:

[0066] The Oracle node group distributes the service request task to the selected three nodes, and simultaneously calls the external data source interface to obtain the data set after the three nodes complete the task.

[0067] The Oracle node group uses the task completion result and cost value of the Oracle node to calculate the reward value: when the Oracle node correctly completes the task, the reward value generated by the environment is calculated using ; when the task is not completed, the reward value is calculated using r k = -μ·c k , wherein μ represents a reward factor, c k represents the cost value, and c k > 1.

[0068] Specifically, the service request task is distributed to three nodes o h , o m , and o n .

[0069] If the task completion result of o h is consistent with the task completion result of o m or o n , the task completion result of o h is output, and the reward values r h , r m , and r n are calculated according to the formula .

[0070] If the task completion result of o h is inconsistent with the task completion result of o m or o n , and the task completion results returned by o m and o n are consistent, the consistent task completion result is output, and the reward value r k generated by o k is calculated according to the formula r h = -μ·c h , and the reward values r m and r n are calculated using .

[0071] If the task completion results of the three nodes are all inconsistent, the task completion result of o h is output, and the reward values r h , r m , and r n are all set to 0.

[0072] The state space set is updated, and the Oracle node group returns the reward value and the updated state space set to the agent.

[0073] On the basis of the above-mentioned embodiments, the embodiments provide an implementation of step 5, including:

[0074] The Critic network introduces a state value function V, and the state value function uses the formula where R t is the future return from the time step t, and the specific calculation formula is where γ represents a decay factor, and r t+k represents the reward value at time t+k, k=0 is the current reward value, and k>0 is a predicted reward value, which needs to be estimated through an environmental dynamic model or historical data.

[0075] The Critic network training steps are as follows:

[0076] Determine the state: in each training step, the current state s t is determined.

[0077] Collect rewards: record the reward value r t obtained by taking the action a t in the state s t .

[0078] Calculate the future return: according to the future state V w (S t+1 ) and the reward value r t , the total return in the future is calculated using the decay factor γ.

[0079] Update the value function: update the state value function V w (S t ) using the collected samples, and the loss function of the Critic model is set as the square sum of TD errors, Calculate the TD error of the estimated reward and the immediate reward r t , δ t =r t +γV w (S t+1 )-V w (S t ), where δ t represents the TD error value, r t is the reward value V w (S t+1 ) represents the state value in the state S t+1 , and V w (S t ) represents the state value in the state S tThe state value is updated as follows: V(s) = V(s) + a [R + γV(s') - V(s)], where a is the learning rate, R is the reward, γ is the decay factor, and V(s) is the state value. The TD error reflects the feedback of the Critic to the Actor, and the Actor updates its policy using this error. If the TD error is positive, it means that the current policy performs well in this state, and the Actor will increase the probability of selecting this action. If the TD error is negative, it means that the current policy performs poorly in this state, and the Actor will decrease the probability of selecting this action. The parameters of V(s) are updated by minimizing the loss function. The gradient of the parameters w is calculated using the gradient descent algorithm The policy network parameters w are updated as follows: w = w - a * ∇wL, where a is the learning rate and ∇wL is the gradient of the loss function with respect to the policy network parameters w. The value network parameters w are updated as follows: w = w - a * ∇wL, where a is the learning rate and ∇wL is the gradient of the loss function with respect to the value network parameters w. w

[0080] Finally, the Critic network feeds back the TD error δ to the Actor network, which uses it to guide the policy network to make better node selections. t

[0081] During the training process, the state value function V(s) gradually approaches the true expected return through multiple updates. The Critic network also becomes more accurate in evaluating the value of each state, thereby improving the policy.

[0082] On the basis of the above embodiment, the present embodiment provides an Actor network parameter updating method, specifically comprising: the Actor network uses the policy gradient method to calculate to maximize the expected return, where θ represents the Actor network parameters, and updates the Actor network parameters to improve the expected return under the current policy, where λ θ represents the learning rate, and π θ (a t |s t ) represents the probability of selecting each oracle node output by the Actor network.

[0083] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, and are not limited thereto. Although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that they can modify the technical solutions described in the foregoing embodiments, or make equivalent substitutions for some technical features, without departing from the spirit and scope of the technical solutions of the embodiments of the present application.​​​

Claims

1. A method for selecting a blockchain oracle node based on reinforcement learning, characterized in that, The application relates to a method for selecting a node to complete a service request in a blockchain network, comprising the following steps: Step 1: a service user sends a service request to a node selection contract through a blockchain network; Step 2: the node selection contract calls an AC contract according to the service request, the AC contract establishes a reputation value and cost value model based on Bayesian cost dependence for all oracle nodes and inputs an agent as a state space set; wherein the agent comprises an Actor network and a Critic network; the Actor network is used for generating a prediction node selection strategy; the Critic network is used for evaluating the state value of the strategy generated by the Actor network and back-propagating the state value to the Actor network; Step 3: the Actor network obtains a current state space set, calculates the probability distribution of each prediction node being selected, randomly selects an oracle node as an optimal node based on the probability distribution, verifies the reputation value, and submits the optimal node to an oracle node group after verification; Step 4: the oracle node group allocates a task of the service request to the optimal node, calculates a reward value after task execution and updates the state space set, and returns the reward value and the updated state space set to the agent; Step 5: the Critic network calculates a state value according to the state space set before and after updating and the reward value, calculates a TD error based on the state value, takes the square sum of the TD error as a loss function of the Critic network to update the Critic network parameters, and returns the TD error to the Actor network, and the Actor network updates the Actor network parameters according to the TD error; Step 6: steps 3 to 5 are repeated until the agent converges to an optimal strategy, and an oracle node is selected to complete the service request according to the optimal strategy. 2.The method of claim 1, wherein, In step 2, the AC contract establishes a reputation value and cost value model based on Bayesian cost dependence for all oracle nodes, and the method comprises the following steps: The number k of oracle nodes of the oracle node group is determined, the prior distribution of the k oracle nodes is initialized by using a Beta (1, 1) distribution, the initial probability of each oracle node is calculated and normalized, the expectation of the Beta distribution is calculated to represent the reputation value of the oracle node, and the cost value of the oracle node is set. 3.The method of claim 1, wherein, Step 3 specifically comprises the following steps: The Actor network calculates the probability distribution of each prediction node being selected according to the current state space set and the Actor network parameters, randomly selects an optimal node according to the probability distribution, calculates the reputation value of the optimal node, and randomly selects two prediction nodes to participate in model training if the reputation value is higher than a threshold value, or directly selects the prediction node with the highest reputation value as the optimal node to complete the service request.

4. The method of claim 3, wherein, Step 4 specifically comprises the following steps: The Oracle node group distributes tasks after obtaining three selected nodes, and distributes the service request to the three nodes, and the three nodes simultaneously call an external data source interface to obtain a data set; The Oracle node group uses the task completion result and cost value of the Oracle node to calculate the reward value: when the Oracle node correctly completes the task, the reward value generated by the environment uses ; when the task is not completed, the reward value uses r k = -μ·c k , where μ represents a reward factor, c k represents the cost value, and c k > 1. The Oracle node group uses the task completion result and cost value of the Oracle node to calculate the reward value: when the Oracle node correctly completes the task, the reward value generated by the environment uses ; when the task is not completed, the reward value uses r k = -μ·c k , where μ represents a reward factor, c k represents the cost value, and c k > 1. The state space set is updated, and the Oracle node group returns the reward value and the updated state space set to the agent.

5. The method of claim 4, wherein, The Oracle node group uses the task completion result and the cost value of the Oracle node to evaluate the reward value, and specifically includes: The service request task is distributed to three nodes o h , o m , and o n ; wherein o h is the best node; If the task completion result of o h is consistent with the task completion result of o m or o n , the task completion result of o h is output and the reward values r h , r m and r n are calculated according to the formula . If the task completion result of o h is inconsistent with the task completion result of o m or o n , and the task completion results returned by the node o m and the node o n are consistent, the consistent task completion result is output, and the reward value r k generated by o k is calculated according to the formula r h =-μ·c h , and the reward values r m and r n are calculated using . If the task completion results of the three nodes are all inconsistent, output the task completion result of the node o h , and set the reward values r h , r m , and r n all to 0.

6. The method of claim 1, wherein, In step 5, the state value function is given by where R t is the future return from time step t, given by where γ represents a decay factor, and r t+k represents the reward value at time t+k, with k = 0 for the current time reward value and k > 0 for a predicted reward value. The TD error calculation formula is: t = r t + γV w (S t+1 ) - V w (S t ), wherein δ t represents a TD error value, r t is a reward value, V w (S t+1 ) represents a state value under a state space set S t+1 , V w (S t ) represents a state value under a state space set S t , w is a Critic network parameter representing a policy, and γ represents a decay factor, γ ∈ [0, 1]. The loss function of the Critic network is shown as follows: wherein L represents the loss function.

7. The method of claim 6, wherein, In step 5, the gradient of the Critic network parameter w is updated using a gradient descent algorithm and according to the gradient value update the Critic network parameter w, where w t+1 denotes the updated network parameter w, w t denotes the network parameter w before the update, and λ w denotes the parameter w learning rate. 8.The method of claim 6, wherein, In step 5, the Actor network computes where θ denotes the Actor network parameters, using the Actor network parameters are updated to improve the expected return under the current policy, where λ θ denotes the learning rate, π θ (a t | s t ) denotes the probability distribution output by the Actor network. 9.The method of claim 6, wherein, In step 6, the convergence rule is that when the TD error mean square value is less than 0.001 for 50 consecutive times, it is determined to be converged.

Images