Blockchain-based controllable evolutionary reinforcement learning method and system

Abstract

The application provides a controllable evolution reinforcement learning method and system based on a blockchain, mainly comprising the following steps: obtaining constraint data; storing the constraint data and a current strategy by using the blockchain; converting the constraint data into a constraint boundary of a strategy space by using a constraint strategy optimization method; training by using a historical gradient guided evolution reinforcement learning method and applying the processed strategy space, and using an adaptive scene threshold comparison mechanism after each decision, so that if dangerous behavior occurs, a decision is made again; using the blockchain to trace back the strategy parameters; and storing the constraint data and the current strategy by using the blockchain. The application simplifies the design of a specific reward function for a specific scene each time by effectively converting the constraints of artificial input, so that controllability is achieved.

Description

Technical Field

[0001] This invention relates to a controllable evolutionary reinforcement learning method and system based on blockchain, belonging to the field of reinforcement learning technology. Background Technology

[0002] Reinforcement learning is a learning method in machine learning. It operates based on feedback from the environment, continuously interacting with it, learning through trial and error, and generalizing to acquire complex behaviors. It explores the action space to ultimately achieve a specific goal or maximize overall action rewards. However, reinforcement learning also has significant drawbacks. It requires backpropagation, has poor parallel processing capabilities, and suffers from issues such as confidence assignment, deceptive rewards, and sparse rewards. Furthermore, its algorithms are less robust and prone to premature convergence.

[0003] In the process of reinforcement learning, some actions may result in high reward values, which may cause distress or harm to people. To avoid these actions, a controllable model must be introduced to constrain them before reinforcement learning.

[0004] In view of this, it is indeed necessary to propose a controllable evolutionary reinforcement learning method and system based on blockchain to solve the above problems. Summary of the Invention

[0005] The purpose of this invention is to provide a controllable evolutionary reinforcement learning method and system based on blockchain to enhance its learning ability.

[0006] To achieve the above objectives, this invention provides a blockchain-based controllable evolutionary reinforcement learning method, which mainly includes the following steps:

[0007] Step 1: Obtain constraint data;

[0008] Step 2: Use blockchain to store constraint data and the current policy;

[0009] Step 3: Use the constraint policy optimization method to transform the constraint data into the constraint boundary of the policy space;

[0010] Step 4: Use the historical gradient-guided evolutionary reinforcement learning method to train the policy space processed in Step 3. After each decision is made, use an adaptive scene threshold comparison mechanism to determine whether it is at a low level, i.e., encountering sudden interference or low reward events. If it is at a low level, proceed to Step 5; otherwise, proceed to Step 6. If dangerous behavior occurs, automatically return to the previous route and make a new decision.

[0011] Step 5: Use blockchain to backtrack the strategy parameters;

[0012] Step 6: Use blockchain to store constraint data and current policies.

[0013] As a further improvement of the present invention, the constraint format of the constraint data is specified as [bhv1, bhv2, ..., bhv...]. m Each bhv contains region constraints and behavioral constraints; by default, a two-dimensional region is used to restrict the agent's actions. The format of the region constraint is: [(x1,y1), (x2,y2)], where (x1,y1) and (x2,y2) represent the coordinates of the diagonal of the rectangular region in the two-dimensional coordinate system. By default, (x1,y1) and (x2,y2) are both 0. The format of the behavioral constraint is: Act; through machine learning methods, the mapping relationship between the region constraints and behavioral constraints to the constraint boundary is obtained, that is:

[0014] f(Act,[(x1,y1),(x2,y2)])→bhv.

[0015] As a further improvement of the present invention, in step 2, during the data sharing process, the transaction format sent for the shared data f generated in the device is as follows:

[0016] TX=(H(σ f ), r, D, URL, π, Sig1, Sig2),

[0017] Where H is the hash of the data, r is the new constraint condition input manually, and σ f = {D, URL, H(f)} is the label vector of data f, where D = {d1, d2, ..., dn} π} represents the π characteristic values ​​of data f during the data sharing process, used to represent the index data. The URL is the address of the blockchain node uploaded by the device, and π is the policy parameter of the agent in the device at this moment. H(σ) f H(f) and H(f) represent the hash feature values ​​obtained by hashing the tag vector and the original data, respectively, and Sig1 is the device's hash value for σ. f The digital signature, Sig2, corresponds to the signature of the blockchain node for this transaction.

[0018] As a further improvement of the present invention, step 2 specifically includes:

[0019] Step 21: Data Collection and Upload. Device v collects shared data f through its own sensors and stores the shared data f in the edge storage server K. The sharing process of the shared data f is handled by node U. n Record and analyze the data features D of f, and generate a label vector σ for the shared data f accordingly. fThe obtained tag vector and the original data are hashed, where the URL field points to the address of K. Then, the industrial intelligent mobile device uses its private key to hash σ. f Perform digital signature, along with σ f Upload to a nearby node and wait for the blockchain to process it;

[0020] Step 22: Blockchain transaction encapsulation and consensus. Upon receiving the tag vector sent by v, the node will extract σ. f The D and URL fields in the data are used to encapsulate the transaction into a TX format. The transaction is then broadcast to the blockchain in which it is currently located. When a node in the blockchain collects a sufficient number of TXs, it will package the transaction and broadcast the packaged block to all nodes in the blockchain to complete the subsequent consensus process.

[0021] As a further improvement of the present invention, in step 3, the constraint strategy optimization method is as follows:

[0022]

[0023]

[0024]

[0025] in, Represented as the dominance function, For auxiliary cost function, It is the total variational dispersion among the action distributions at point s. Represented as having parameter θ k The strategy, d i Defined as the input constraint, γ is the discount factor in the range [0, 1), E represents the expectation, k is the number of iterations, δ is the step size and δ > 0, d θ (s) represents the distribution of the discounted future states.

[0026] As a further improvement of the present invention, the discounted future state distribution d θ (s) is

[0027]

[0028] Where P(s) represents the probability of state s occurring under the current policy, and γ is the discount coefficient ranging from [0, 1).

[0029] As a further improvement of the present invention, step 4 specifically includes the following steps:

[0030] Step 41: Initialize the evaluation network Q π and target evaluation network Initialize the replay buffer R, defining its capacity as max_mem; initialize an actor network π with random parameters. μ ; using known hyperparameters: capacity k, adaptive coefficient α, and π μ The number of parameters is initialized for the gradient buffer grad_buffer and noise generator noise_generator required in subsequent steps;

[0031] Step 42: Generate a pop_size number of individuals using the cross-entropy method;

[0032] Step 43: Use the TD3 method to perform reinforcement learning training on the best individuals, using the top 5 individuals ranked by fitness value by default;

[0033] Step 44: Evaluate each individual using the evaluation method and obtain the corresponding fitness value. At the same time, add the experience obtained during the evaluation process to the replay buffer, record the step size, calculate the fitness value ranking, and extract the individual with the best fitness value according to the outstanding strategy.

[0034] Step 45: Determine whether the population is in the warm-up phase by judging the conditions. If it is in the warm-up phase, the search direction is sampled using a normal distribution. Otherwise, the search direction is sampled using a noise_generator. The sampled search direction is added to the elite individuals. All generated individuals are evaluated. The obtained fitness value is compared with the optimal fitness value obtained in the previous round of evaluation multiplied by a threshold of the proportional coefficient. Individuals with a fitness value greater than the threshold will be recruited and enter the elite pool on which the population iteration is based.

[0035] Step 46: Calculate the gradient for this round and put it into grad_buffer. Update the matrix in the cross-entropy method using the parameters of the top 5 individuals in the elite pool.

[0036] Step 47: After completing one round of evolutionary reinforcement learning, an adaptive scene threshold comparison method will be used.

[0037] As a further improvement to the present invention, the hyperparameter α is updated to

[0038]

[0039]

[0040] Wherein, the hyperparameter α represents the scaling factor for using historical gradients. and These represent the two subspaces in the current iteration. and The superiority of M is in The number of times sampled in the middle, PM is the number of times sampled in the middle. The number of times the sample is taken;

[0041]

[0042] Where, α t δ represents the value of α at time t, δ is a scaling factor greater than 1, and k1 and k2 are the lower and upper limits of α.

[0043] To achieve the above objectives, the present invention also provides a blockchain-based controllable evolutionary reinforcement learning system, which applies the blockchain-based controllable evolutionary reinforcement learning method described above.

[0044] As a further improvement of the present invention, the blockchain-based controllable evolutionary reinforcement learning system is applied to industrial intelligent mobile devices.

[0045] The beneficial effects of this invention are: by effectively transforming the constraints of manual input, this invention simplifies the need to design specific reward functions for specific scenarios each time, thereby achieving controllability. Attached Figure Description

[0046] Figure 1 This is a flowchart illustrating the controllable evolutionary reinforcement learning method based on blockchain according to the present invention.

[0047] Figure 2 This is a schematic diagram of the blockchain-based controllable evolutionary reinforcement learning method of the present invention.

[0048] Figure 3 This is a comparative diagram of adaptive scene thresholds in the controllable evolutionary reinforcement learning method based on blockchain of the present invention. Detailed Implementation

[0049] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be described in detail below with reference to the accompanying drawings and specific embodiments.

[0050] It should be noted that, in order to avoid obscuring the present invention with unnecessary details, only the structures and / or processing steps closely related to the present invention are shown in the accompanying drawings, while other details that are not closely related to the present invention are omitted.

[0051] Additionally, it should be noted that the terms “comprising,” “including,” or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus.

[0052] like Figures 1 to 3As shown, this invention discloses a blockchain-based controllable evolutionary reinforcement learning method and system. The blockchain-based controllable evolutionary reinforcement learning system, taking an industrial intelligent mobile device as an example, includes the following steps:

[0053] Step 1: Obtain constraint data. For example: Do not approach moving objects; keep away from water and fire sources; working time should not be too long; the actions of industrial intelligent mobile devices are guided by evolutionary reinforcement learning agents, but can also be prevented by adding constraint actions. Constraint actions can be manually input. The constraint format is defined as [bhv1, bhv2, ..., bhv...]. m Each bhv contains region constraints and behavioral constraints; by default, a two-dimensional region is used to restrict the agent's actions. The format of the region constraint is: [(x1,y1), (x2,y2)], where (x1,y1) and (x2,y2) represent the coordinates of the diagonal of the rectangular region in the two-dimensional coordinate system. By default, (x1,y1) and (x2,y2) are both 0. The format of the behavioral constraint is: Act; through machine learning methods, the mapping relationship between the region constraints and behavioral constraints and the constraint boundaries can be obtained, that is:

[0054] f(Act,[(x1,y1),(x2,y2)])→bhv, (1);

[0055] Step 2: Use blockchain to store constraint data and the current policy;

[0056] Step 3: Use the constraint policy optimization method to transform the constraint data into the constraint boundary of the policy space;

[0057] Step 4: Use the Evolutionary Reinforcement Learning Guided by Historical Gradient (ERLGHG) method to train the policy space processed in Step 3. After each decision is made, use an adaptive scene threshold comparison mechanism to determine whether it is at a low level, i.e., encountering sudden interference or low reward events. If it is, proceed to Step 5; otherwise, proceed to Step 6. If dangerous behaviors such as colliding with pedestrians, touching walls, or being suspended in mid-air occur, automatically return to the previous route and make a new decision.

[0058] Step 5: Use blockchain to backtrack the strategy parameters;

[0059] Step 6: Use blockchain to store constraint data and current policies. The process is the same as step 2. This includes parameters such as the routes and speeds already traveled by intelligent industrial mobile devices.

[0060] The blockchain technology in step 2 will be introduced separately below.

[0061] 1. Format design of shared data.

[0062] During the data sharing process, for the shared data f generated in the device, the transaction format sent is as follows:

[0063] TX=(H(σ f ), r, D, uRL, π, Sig1, Sig2), (2);

[0064] Where H is the hash of the data, r is the new constraint condition input manually, and σ f = {D, URL, H(f)} is the label vector of data f, where D = {d1, d2, ..., d...} rr} represents the π feature values ​​of data f during the data sharing process, used to represent the index data. The URL is the address of the blockchain node uploaded by the device. π is the policy parameter of the agent in the device at this moment. By default, the system stores a parameter matrix, which contains the parameters of each node in the deep network used in historical gradient-guided evolutionary reinforcement learning. H(σ) f H(f) and H(f) represent the hash feature values ​​obtained by hashing the tag vector and the original data, respectively, and Sig1 is the device's hash value for σ. f The digital signature, Sig2, corresponds to the signature of the blockchain node for this transaction.

[0065] 2. Data collection and packaging.

[0066] Due to the complexity of network conditions across different devices, this solution places the training environment locally, with the device's own computing power performing training, data uploading, and recovery according to the set conditions. The device will periodically upload the locally obtained policy parameters, or when new manual constraints are input, it will package the local policy parameters and the manually input constraints and upload them.

[0067] Based on this, blockchain storage is performed.

[0068] Step 21: Data Collection and Upload. The industrial intelligent mobile device v collects shared data f through its own sensors and stores the shared data f in the edge storage server K. The sharing process of the shared data f is handled by node U. n Recording is performed. The industrial intelligent mobile device, based on its own computing power, analyzes the data characteristics D of f and generates a corresponding label vector σ for the data. f The obtained tag vector and the original data are hashed, where the URL field points to the address of K. Then, the industrial intelligent mobile device uses its private key to hash σ. f Perform digital signature, along with σf Upload to a nearby node and wait for the blockchain to process it.

[0069] Step 22: Blockchain Transaction Encapsulation and Consensus. Upon receiving the tag vector sent by v, the node will extract σ. f The `D` and `URL` fields are encapsulated into a transaction format `TX`, and then the transaction is broadcast to the current blockchain. When a node in the blockchain collects a sufficient number of `TX` values, it packages the transaction and broadcasts the packaged block to all nodes in the blockchain to complete the subsequent consensus process. During the consensus process, the transactions in the block are verified. Specifically, a node reads the URL in the blockchain, obtains the hash value H(f) of the data `f`, and then uses its own computing power to perform hash calculations on H(f), `D`, and the URL, comparing it with H(σ) in the transaction. f The hash values ​​are compared, and if they match, the transaction data is considered unaltered and the verification phase is passed; otherwise, the verification fails, and the following judgment is made:

[0070] H(D, URL, H(f)) = H(σ) f ), (3).

[0071] The specific steps for the controllable model in step 3 are described below.

[0072] The classic Constrained Policy Optimization (CPO) approach is introduced. First, three common formulas in reinforcement learning need to be clarified. When optimizing the policy, maximizing formula (6) yields the optimal policy.

[0073] t=0

[0074]

[0075]

[0076]

[0077] Where, d θ (s) represents the discounted future state distribution:

[0078]

[0079] Where P(s) represents the probability of state s occurring under the current policy, γ is the discount coefficient ranging from [0, 1), and ρ(s) is defined as the unnormalized discounted frequency of future state visits.

[0080] Generally, in policy optimization, to ensure stability during the optimization process, J(θ′) is not directly optimized. Instead, an iterative trust region optimization method is used, as shown below:

[0081]

[0082] Where D(θ,θ) k ) is a kind of distance metric, and δ>0 is the step size.

[0083] In general, the restricted policy space is defined as follows:

[0084]

[0085] in, These are auxiliary cost functions.

[0086] Therefore, optimization problem (8) can be modified to the following form:

[0087]

[0088] D(θ,θ k )≤δ, (9),

[0089] Optimization problems (9) in continuous high-dimensional spaces are difficult to update. Generally, a surrogate function is updated: that is, a lower bound is derived, and then the lower bound is raised during optimization, thereby achieving the goal of optimizing the original problem. So, the problem now focuses on: how to obtain a lower bound and then optimize this lower bound.

[0090] The following section presents three inferences.

[0091] Inference 1:

[0092]

[0093]

[0094] J(θ′)-J(θ), (12),

[0095] Among them, formula (11) is an approximation of formula (12).

[0096] Inference 2:

[0097]

[0098] (13), where C i It is a cost function.

[0099] Inference 3:

[0100] Based on existing conclusions in information theory

[0101]

[0102] D in Corollary 1 and Corollary 2 TV Rewritten as D KL The two corollaries still hold true (the upper and lower bounds still hold). Therefore, the final agent optimization objective of CPO can be obtained.

[0103] For the objective function, it is only necessary to raise the lower bound of formula (10). For the policy constraint, it is only necessary to ensure that the new policy still satisfies the constraint conditions. Therefore, formulas (10) and (13) are substituted into formula (11). Thus, through corollaries 1, 2, and 3, for an appropriate coefficient a k , Updated to:

[0104]

[0105]

[0106] D(θ,θ k )≤δ, (15),

[0107] Equation (15) is a strategy that guarantees the generation of monotonically non-decreasing returns that satisfy the original constraints (note that the constraints here are...). (Multiplied by the upper bound of (15)). The non-policy evaluation problem is alleviated because both the objective and constraints involve the distribution of states. The expectation is assumed to have a sample. Because the boundary is compact, the problem is always feasible (as long as π). θ (It is feasible). However, for discount factors close to 1, the penalty for policy divergence is quite severe, so the steps taken in this update may be small.

[0108] Next, the penalty term in the standard function is used instead of the distance metric. Inspired by the trust region method, a CPO method is produced, which uses the trust region instead of the penalty for policy divergence to achieve a larger step size. Therefore, the optimization problem (15) is rewritten as:

[0109]

[0110]

[0111] in, Represented as the dominance function, For auxiliary cost function, It is the total variational dispersion among the action distributions at point s. Represented as having parameter θk The strategy, d i Defined as the input constraint, γ is the discount coefficient for the range [0, 1), E represents the expectation, k is the number of iterations, and δ is the step size and δ>0.

[0112] Based on the above derivation, a strategy that guarantees the generation of monotonically non-decreasing returns that satisfy the original constraints is obtained, which is the Constrained Policy Optimization (CPO).

[0113] For d in the policy space constrained by formula (8) j , representing all m constraints that the model needs to satisfy during training, i.e., bhv in step 1. i , i = 1, 2, ..., m; each constraint corresponds to the input of the constraint format therein.

[0114] Step 31: According to the above CPO method, i.e. formula (16), it is used as the agent optimization target of evolutionary reinforcement learning, which is equivalent to satisfying the constraints of human input. This simplifies the need to design a specific reward function for a specific scenario each time, thereby avoiding decisions with high rewards but which may cause trouble or harm to people.

[0115] The specific steps for the evolutionary reinforcement learning method in step 4 are as follows.

[0116] Before we begin evolutionary reinforcement learning, we need to define several key parameters.

[0117] max_steps: The maximum step size of the algorithm, i.e. the range of steps experienced during the training phase.

[0118] enlist_pop: The number of elite individuals retained during the evolutionary process.

[0119] α: Adaptive coefficients for using historical gradients.

[0120] k: Size of the historical gradient container.

[0121] pop_size: The number of individuals generated in each iteration.

[0122] Ω: Threshold for determining whether the reward parameter W is at a low level.

[0123] W: Reward parameter value.

[0124] Step 41: Initialize the evaluation network Q π and target evaluation network Initialize the replay buffer R, defining its capacity as max_mem; initialize an actor network π with random parameters.μ Its network parameters can be regarded as an individual in the population; through known hyperparameters: capacity k, adaptive coefficient α, and π μ The number of parameters is initialized for the gradient buffer grad_buffer and noise generator noise_generator required in subsequent steps.

[0125] Step 42: Generate pop_size individuals using the cross-entropy method (CEM), each of which can be a parameter of an actor network.

[0126] Step 43: Use the TD3 method to perform reinforcement learning training on the first 5 individuals to optimize the policy distribution.

[0127] Step 44: Evaluate each individual using the evaluation method and obtain the corresponding fitness value. At the same time, add the experience obtained during the evaluation process to the replay buffer, record the step size, calculate the fitness ranking, and extract the individual with the best fitness value according to the outstanding strategy.

[0128] Step 45: Determine if the system is in the warm-up phase. If it is, sample search directions using a normal distribution; otherwise, use a noise generator to sample search directions. Add the sampled search directions to the elite individuals and evaluate all generated individuals. Compare the resulting fitness value with the result of multiplying the fitness value of the best individual from the previous evaluation by a proportional coefficient (threshold). Individuals with a fitness value greater than the threshold will be recruited and added to the elite pool used for population iteration.

[0129] Step 46: Calculate the gradient for this round and put it into the grad_buffer. Update the value of α according to formula (17). Update the matrix in CEM using the parameters of the top 5 individuals in the elite pool. The hyperparameter α is updated as follows:

[0130]

[0131]

[0132] Wherein, the hyperparameter α represents the scaling factor for using historical gradients. and These represent the two subspaces in the current iteration. and The superiority of M is in The number of times sampled in the middle, PM is the number of times sampled in the middle. The number of times the sample is taken;

[0133]

[0134] Where, α t δ represents the value of α at time t, δ is a scaling factor greater than 1, and k1 and k2 are the lower and upper limits of α.

[0135] Step 47: After completing one round of evolutionary reinforcement learning, an adaptive scene threshold comparison method will be applied. The original evaluation value is the reward parameter W, and the tracking value is the average of the reward parameter over the previous P periods. As the number of periods increases, at a certain moment when the tracking value is greater than the original evaluation value, the average of the original evaluation value and the tracking value at that moment is used as the threshold Ω. This method makes it easy to determine whether an abnormal state is in progress. Furthermore, since the threshold Ω is the average of the original evaluation value and the tracking value, the condition is only triggered when the next abnormal value falls below the threshold. Therefore, this mechanism has a certain degree of redundancy, allowing for a limited range of abnormal occurrences.

[0136] The specific steps for using blockchain to trace back in step 5 are as follows.

[0137] By leveraging the backup function of the blockchain, we find the neighboring nodes that store the three most recent policy parameters and retrieve the three most recent policy parameters locally. We then average these three policy parameters. If the resulting policy parameters are still poor, we can continue backtracking until we obtain better parameters, which are then set as the parameters for the evolutionary reinforcement learning method.

[0138] This enables intelligent industrial mobile devices to avoid performing actions that users do not want to do, while ensuring the efficient completion of their tasks.

[0139] This invention addresses the shortcomings of reinforcement learning in terms of environmental exploration capabilities and robustness, making it susceptible to deceptive gradients caused by fraudulent rewards. Evolutionary algorithms, on the other hand, generally possess advantages such as better global search capabilities, robustness, and parallelism. Therefore, this invention proposes an evolutionary reinforcement learning algorithm. Reinforcement learning incorporating evolutionary algorithms is less prone to getting trapped in local optima.

[0140] To address the issue that policies trained using evolutionary reinforcement learning often lead to optimal solutions, which are not necessarily safe for humans, thus causing distress and harm, a controllable model is proposed. This model transforms manually input data into constraints, simplifying the design of specific reward functions for each scenario. Furthermore, with these constraints, the policy will not attempt any actions solely for high rewards, unlike in reinforcement learning.

[0141] Traditional storage technologies cannot guarantee the security of data, so blockchain technology has been creatively incorporated. By introducing blockchain technology, historical strategies and manually entered data can be securely saved and viewed at any time.

[0142] For evolutionary reinforcement learning, external and internal factors can cause rewards to remain at a low level for extended periods, leading to low algorithm efficiency. Blockchain technology can address this issue. By periodically uploading training decisions to the blockchain, when rewards remain low for an extended period, a designed mechanism can be used to backtrack the current policy using the better decisions on the blockchain.

[0143] In summary, this invention proposes an improved evolutionary reinforcement learning method that combines blockchain technology and a controllable model. This method overcomes the limitations of traditional evolutionary reinforcement learning in backtracking and the ambiguity of constraints, allowing the algorithm to remain within a safe policy search space during decision-making, thus avoiding high-reward but high-risk actions. Furthermore, this invention utilizes blockchain technology to effectively preserve the decision-making process and store periodic policy parameters, simplifying policy management and backtracking for administrators.

[0144] The above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention.

Claims

1. A controllable evolutionary reinforcement learning method based on blockchain, characterized in that, The main steps include: Step 1: Obtain constraint data; Step 2: Use blockchain to store constraint data and the current policy; In step 2, during the data sharing process, the transaction format sent for the shared data f generated in the device is as follows: TX = (H(σ f ), r, D, URL, π, Sig1, Sig2), Where H is the hash of the data, r is the new constraint condition input manually, and σ f = {D, URL, H(f)} is the label vector of data f, where D = {d1, d2, ..., d...} π } represents the π characteristic values ​​of data f during the data sharing process, used to represent the index data. The URL is the address of the blockchain node uploaded by the device, and π is the policy parameter of the agent in the device at this moment. H(σ) f H(f) and H(f) represent the hash feature values ​​obtained by hashing the tag vector and the original data, respectively, and Sig1 is the device's hash value for σ. f The digital signature, Sig2, corresponds to the signature of the blockchain node for this transaction; Step 2 specifically includes: Step 21, data collection and upload, the device v collects shared data f through its own sensor and stores the shared data f in the edge storage server K, the sharing process of the shared data f is performed by the node U n Records, analyzes the data characteristics D of f, and generates a label vector σ for the shared data f accordingly f , Hash operation is performed on the obtained label vector and the original data, wherein the URL field points to the address of K, and then the industrial intelligent mobile device uses its own private key to perform digital signature on σ f , and σ f is uploaded to the adjacent node and waits for the processing process of the blockchain; Step 22, blockchain transaction packaging and consensus, the node receiving the label vector v sent will extract D, URL field in σ f , package it into the format of transaction TX, and then broadcast the transaction to the blockchain where the node itself is currently located. When a node in the blockchain collects a sufficient number of TX, it will package the transaction and broadcast the packaged block to all nodes in the blockchain to complete the subsequent consensus process; Step 3: Use the constraint policy optimization method to transform the constraint data into the constraint boundary of the policy space; Step 4: Use the historical gradient-guided evolutionary reinforcement learning method to train the policy space processed in Step 3. After each decision is made, use an adaptive scene threshold comparison mechanism to determine whether it is at a low level, i.e., encountering sudden interference or low reward events. If it is at a low level, proceed to Step 5; if it is not at a low level, proceed to Step 6. If dangerous behavior occurs, make a new decision. Step 5: Use blockchain to backtrack the strategy parameters; Step 6: Use blockchain to store constraint data and current policies. 2.The controllable evolutionary reinforcement learning method based on blockchain according to claim 1, wherein: The constraint format of the constraint data is defined as [bhv1,bhv2,…,bhv]. m Each bhv contains region constraints and behavioral constraints; by default, a two-dimensional region is used to restrict the agent's actions. The format of the region constraint is: [(x1,y1),(x2,y2)], where (x1,y1) and (x2,y2) represent the coordinates of the diagonal of the rectangular region in the two-dimensional coordinate system. By default, (x1,y1) and (x2,y2) are both 0. The format of the behavioral constraint is: Act; through machine learning methods, the mapping relationship between the region constraints and behavioral constraints to the constraint boundaries is obtained, that is: f(Act,[(x1,y1),(x2,y2)])→bhv. 3.The blockchain-based controllable evolutionary reinforcement learning method of claim 1, wherein: In step 3, the constraint strategy optimization method is as follows: in, Represented as the dominance function, For auxiliary cost function, It is the total variational dispersion among the action distributions at point s. Represented as having parameter θ k The strategy, d i Defined as the input constraint, γ is the discount factor in the range [0,1), E represents the expectation, k is the number of iterations, δ is the step size and δ>0, d θ (s) represents the distribution of the discounted future states. 4.The controllable evolutionary reinforcement learning method based on blockchain according to claim 3, characterized in that: The discounted future state distribution d θ (s) is Where P(s) represents the probability of state s occurring under the current policy, and γ is the discount coefficient ranging from [0,1). 5.The controllable evolutionary reinforcement learning method based on blockchain according to claim 3, wherein: Step 4 specifically includes the following steps: Step 41, initialize the evaluation network Q π and the target evaluation network Initialize the replay buffer R, define its capacity size as max_mem; initialize an actor network π with random parameters μ ; initialize the gradient buffer grad_buffer and the noise generator noise_generator needed in the subsequent steps by the known hyperparameters: the capacity k, the adaptive coefficient α and the number of parameters of π μ ; Step 42: Generate a pop_size number of individuals using the cross-entropy method; Step 43: Use the TD3 method to perform reinforcement learning training on the best individuals, using the top 5 individuals ranked by fitness value by default; Step 44: Evaluate each individual using the evaluation method and obtain the corresponding fitness value. At the same time, add the experience obtained during the evaluation process to the replay buffer, record the step size, calculate the fitness value ranking, and extract the individual with the best fitness value according to the outstanding strategy. Step 45: Determine whether the population is in the warm-up phase by judging the conditions. If it is in the warm-up phase, the search direction is sampled using a normal distribution. Otherwise, the search direction is sampled using a noise_generator. The sampled search direction is added to the elite individuals. All generated individuals are evaluated. The obtained fitness value is compared with the optimal fitness value obtained in the previous round of evaluation multiplied by a threshold of the proportional coefficient. Individuals with a fitness value greater than the threshold will be recruited and enter the elite pool on which the population iteration is based. Step 46: Calculate the gradient for this round and put it into grad_buffer. Update the matrix in the cross-entropy method using the parameters of the top 5 individuals in the elite pool. Step 47: After completing one round of evolutionary reinforcement learning, an adaptive scene threshold comparison method will be used.

6. The blockchain-based controllable evolutionary reinforcement learning method according to claim 5, characterized in that: The hyperparameter α is updated to wherein the super parameter a represents a proportionality coefficient of using historical gradient, and respectively represent the goodness of two subspaces and in the current iteration, M is the number of sampling in , and P-M is the number of sampling in . where α t represents the value of a at time t, δ is a scaling factor greater than 1, and k1 and k2 are lower and upper limits for a.

7. A controllable evolutionary reinforcement learning system based on blockchain, characterized in that: The controllable evolutionary reinforcement learning method based on blockchain, as described in any one of claims 1-6, is applied. 8.The blockchain-based controllable evolutionary reinforcement learning system of claim 7, wherein: The blockchain-based controllable evolutionary reinforcement learning system is applied to intelligent industrial mobile devices.

Images