A continuous control method based on value reinforcement learning

Abstract

The application provides a continuous control method based on value reinforcement learning, comprising: constructing a value function of a multi-layer neural network model; searching an optimal action for the value function; based on the optimal action, performing GPU accelerated batch action optimization on sample data to obtain a batch optimal action; based on the batch optimal action, constructing a candidate action set; performing an action according to the candidate action set and updating the value function; and iteratively processing the updated value function to obtain an optimal strategy of the value function. The application provides a learning-based, data-driven and effective control method for a continuous control task in a complex environment.

Description

Technical Field

[0001] This invention belongs to the field of continuous control technology, and particularly relates to a continuous control method based on value reinforcement learning. Background Technology

[0002] Intelligent control methods for complex environments are an important topic in the field of control. Complex environments include those with unknown models, uncertainties, open environments, and incomplete information, and are widely found in autonomous unmanned control systems such as autonomous driving and robotics. How to solve autonomous control problems in complex environments using artificial intelligence methods is a cutting-edge research direction.

[0003] Standard control engineering has limitations in complex environments, specifically including: 1) Constructing dynamic systems in time-varying, uncertain, and open environments is difficult (e.g., whether the ground can support the weight of a multi-legged robot after it performs a walking motion is uncertain); 2) Even in relatively stable environments, experts need to construct dynamic systems tailored to specific needs, resulting in low adaptability of control strategies across different machines (e.g., a multi-legged robot with the same function needs a redesigned dynamic system after the leg length changes); 3) Solving for optimal actions is slow in control systems with many variables (e.g., the computational cost of solving dynamic systems constructed for multi-legged robots is higher than that of single-joint robotic arms). Therefore, adaptive, data-driven, and learning-based control methods are needed, such as reinforcement learning.

[0004] Reinforcement learning is a data-driven artificial intelligence method and also an important intelligent control method. It achieves optimal control oriented towards a goal (maximizing expected reward) through a cyclical learning process of "decision-interaction with the environment-updating decision indicators." Thanks to deep neural networks, reinforcement learning expands its application prospects in unknown and uncertain environments by combining with sensing technologies such as computer vision. For example, it has applications in game-playing scenarios like Go, and in continuous control scenarios such as multi-legged robots and robotic arms.

[0005] Currently, the key constraint on the practical application of reinforcement learning is insufficient sample efficiency, resulting in persistently high training times. Policy-based reinforcement learning, due to its inherent mechanism, allows for smooth policy changes with relatively small fluctuations in the short term; value-based reinforcement learning, on the other hand, allows for large policy changes and holds promise for reducing training costs.

[0006] The existing technologies are divided into two parts: 1. Application of value reinforcement learning in continuous control problems and 2. Application of swarm intelligence optimization in reinforcement learning.

[0007] 1. The application of value-based reinforcement learning in continuous control problems can be roughly divided into the following three categories: 1) Simplifying the value function, which sacrifices the strong expressive power of the neural network to obtain a more standardized value function (e.g., a convex function) to achieve action optimization; 2) Specializing the action distribution, which uses prior assumptions about the action distribution model (e.g., a Gaussian distribution) and then learns to adjust the key parameters in the distribution model to achieve action optimization; 3) Discretizing the action space, which divides the continuous space into multiple discrete spaces and then uses reinforcement learning methods to solve the transformed discrete control problem. All three lead to a decrease in the expressive power of the value function.

[0008] 2. The application of swarm intelligence optimization in reinforcement learning can be divided into two aspects: 1) Value function model optimization, that is, using swarm intelligence optimization methods to search for the optimal value function structure, parameters, etc., so as to obtain the optimal value function based on existing samples; 2) Policy optimization, that is, by replacing the behavior policy in reinforcement learning with individuals in swarm intelligence optimization, under the condition of multiple individuals searching in parallel, the best decision sequence (optimal experience) is obtained, thereby improving learning efficiency.

[0009] Neither of them achieved action optimization in a continuous space within the framework of standard reinforcement learning. Summary of the Invention

[0010] To address the aforementioned technical problems, this invention proposes a continuous control method based on value reinforcement learning, providing a learning-based, data-driven, and effective control method for continuous control tasks in complex environments.

[0011] To achieve the above objectives, this invention provides a continuous control method based on value reinforcement learning, comprising:

[0012] Construct a value function for a multilayer neural network model that takes the state as input;

[0013] A swarm intelligence optimization algorithm is used to search for the optimal action of the value function in any state;

[0014] Based on the aforementioned swarm intelligence optimization algorithm, GPU acceleration is used to optimize actions in batch states to obtain the optimal actions in the batch.

[0015] Based on the batch optimal actions, construct a candidate action set;

[0016] Based on the candidate action set, select an action to execute and update the value function;

[0017] The updated value function is iterated through a loop of state input-action search-action execution-state input to obtain the optimal strategy for the value function.

[0018] Optionally, the multi-layer neural network model includes: an input layer, a hidden layer, and an output layer;

[0019] The input layer is:

[0020] out0 = (s t ,a t )

[0021] Among them, s t For state, a t For the action, out0 is the feature vector corresponding to the state-action pair;

[0022] The hidden layer is:

[0023]

[0024] Where ReLU is the activation function. Let θ be the output vector of the (i-1)th layer. i Let the weights be those of the i-th layer.

[0025] The output layer is:

[0026]

[0027] Where out4 is the output vector of the output layer. θ is the output vector of the hidden layer, and θ4 is the weight of the output layer.

[0028] Optionally, searching for the optimal action of the value function in any state includes:

[0029] Based on the value function and the input state, the target to be optimized is obtained;

[0030] Based on the swarm intelligence optimization algorithm, the target to be optimized is subjected to action optimization processing to obtain the optimal action.

[0031] Optionally, obtaining the optimal batch action includes:

[0032] Based on batch input states, construct a large population related to the number of input states;

[0033] The large population is input into the value function to obtain the locally optimal individual and the globally optimal individual.

[0034] Construct a matrix-form particle swarm optimization model;

[0035] Based on the particle swarm optimization model, the individual positions of the locally optimal individual and the globally optimal individual are updated to obtain the batch optimal action.

[0036] Optionally, the particle swarm optimization model is:

[0037] V = wV + r1c1(Lb-P) + r2c2(x) g -P)

[0038] P = P + V

[0039] Where V is the particle velocity matrix, w is the inertia weight, r1 is a random number in the interval [0,1], c1 is the local learning factor, Lb is the local optimal individual, P is the particle position matrix, r2 is a random number in the interval [0,1], c2 is the global learning factor, and x g It is the globally optimal individual.

[0040] Optionally, obtaining the locally optimal individual and the globally optimal individual includes:

[0041] The large population is input into the value function to obtain the fitness matrix of each individual.

[0042] Perform a matrix transformation of a preset dimension on the fitness matrix;

[0043] The transformed fitness matrix is ​​processed by the argmax function to obtain the local optimal individual and the global optimal individual.

[0044] Optionally, the method for selecting the action to be executed based on the candidate action set is: random sampling using the softmax operator;

[0045] The softmax operator is randomly sampled as follows:

[0046]

[0047] Among them, Pr(a i |s t ) is in state s t Choose action a i The probability, f(s) t ,a i ) represents state s t Next action a i The values ​​are: t is the time step (integer value), and i and j are indices (integer values).

[0048] Optionally, updating the value function includes:

[0049] The optimal value function is obtained by using the square of the time difference error as the loss function and optimizing the weights of each node in the network through backpropagation of the neural network.

[0050] The time difference error is:

[0051]

[0052] Where γ is the discount factor. Treat it as a number.

[0053] Optionally, the optimal strategy is:

[0054]

[0055] Wherein, π(s) t ) is the strategy, f(s) t a) represents state s t The value of the next action 'a'.

[0056] Compared with the prior art, the present invention has the following advantages and technical effects:

[0057] This invention addresses the challenge of optimal action sampling in value-based reinforcement learning by starting with a data-driven intelligent control method. Specifically, it integrates swarm intelligence optimization algorithms to optimize actions under complex value functions; utilizes GPU platforms to improve the efficiency of optimal action search in batch training scenarios; and provides theoretical guidance for setting key parameters in the action optimization stage. This effectively solves the problem of optimal action sampling in value-based reinforcement learning when solving continuous control problems. This invention provides a learning-based, data-driven, and effective control method for continuous control tasks in complex environments. Attached Figure Description

[0058] The accompanying drawings, which form part of this application, are used to provide a further understanding of this application. The illustrative embodiments and descriptions of this application are used to explain this application and do not constitute an undue limitation of this application. In the drawings:

[0059] Figure 1 This is a schematic flowchart of the continuous control method according to an embodiment of the present invention;

[0060] Figure 2 This is a schematic diagram of a bipedal robot according to an embodiment of the present invention. Detailed Implementation

[0061] It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other. This application will now be described in detail with reference to the accompanying drawings and embodiments.

[0062] It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases the steps shown or described may be executed in a different order than that shown here.

[0063] This invention proposes a continuous control method based on value reinforcement learning, comprising:

[0064] Construct a value function for a multilayer neural network model that takes the state as input;

[0065] A swarm intelligence optimization algorithm is used to search for the optimal action of the value function in any state;

[0066] Based on the aforementioned swarm intelligence optimization algorithm, GPU acceleration is used to optimize actions in batch states to obtain the optimal actions in the batch.

[0067] Based on the batch optimal actions, construct a candidate action set;

[0068] Based on the candidate action set, select an action to execute and update the value function;

[0069] The updated value function is iterated through a loop of state input-action search-action execution-state input to obtain the optimal strategy for the value function.

[0070] Furthermore, the multi-layer neural network model includes an input layer, a hidden layer, and an output layer.

[0071] Furthermore, searching for the optimal action of the value function in any state includes:

[0072] Based on the value function and the input state, the target to be optimized is obtained;

[0073] Based on the swarm intelligence optimization algorithm, the target to be optimized is subjected to action optimization processing to obtain the optimal action.

[0074] Furthermore, obtaining the optimal batch action includes:

[0075] Based on batch input states, construct a large population related to the number of input states;

[0076] The large population is input into the value function to obtain the locally optimal individual and the globally optimal individual.

[0077] Construct a matrix-form particle swarm optimization model;

[0078] Based on the particle swarm optimization model, the individual positions of the locally optimal individual and the globally optimal individual are updated to obtain the batch optimal action.

[0079] Furthermore, obtaining the locally optimal individual and the globally optimal individual includes:

[0080] The large population is input into the value function to obtain the fitness matrix of each individual.

[0081] Perform a matrix transformation of a preset dimension on the fitness matrix;

[0082] The transformed fitness matrix is ​​processed by the argmax function to obtain the local optimal individual and the global optimal individual.

[0083] Furthermore, based on the candidate action set, the method for selecting the action to be executed is: random sampling using the softmax operator.

[0084] Furthermore, updating the value function includes:

[0085] The optimal value function is obtained by using the square of the time difference error as the loss function and optimizing the weights of each node in the network through backpropagation of the neural network.

[0086] Example

[0087] This embodiment provides a continuous control method based on value reinforcement learning. Taking a bipedal robot as an example, such as... Figure 2 As shown, from top to bottom, the five white dots represent the leg joint, knee joint (left and right), and ankle joint (left and right). Each joint is connected using controllable hinges. Data related to the reinforcement learning algorithm includes:

[0088] State: A vector consisting of 12 numbers, including the opening and closing angles of each joint hinge and their corresponding angular velocities, plus the current height and position of the leg joint. The state representation vector used varies in different simulation environments.

[0089] Action: The torque applied to each joint is used to adjust the walking posture of the bipedal robot.

[0090] Reward: The cost of moving horizontal distances and adjusting the posture of each joint with the bipedal robot (e.g., the sum of squares of the movements).

[0091] The above-mentioned feature construction is unrelated to this invention. One of the advantages of reinforcement learning is that it can achieve intelligent control through data-driven learning without the need for professional knowledge.

[0092] In the algorithm flow of this invention, it is assumed that the above-mentioned environment ENV, namely a bipedal robot walking simulation system, has been constructed. When the executable actions are continuously changing, this environment is called a continuous control task.

[0093] The learning process corresponds to Figure 1 The iterative learning process is described as follows: At each moment (corresponding to a fixed time interval in real time, such as 0.01 seconds, denoted as Δt), the algorithm obtains the state from the environment ENV; after calculation, the algorithm outputs the action and controls the bipedal robot to change its posture accordingly; after the change, the new state and the reward obtained after the action are fed back to the algorithm, which are collectively referred to as the (trajectory), and are used to learn and improve the control strategy.

[0094] In summary, the input to the algorithm is the data that needs to be collected, including the state and reward of the bipedal robot; the output of the algorithm is the final result, which is the action that the bipedal robot is about to perform.

[0095] The overall flow of this embodiment is as follows: Figure 1 As shown, the following is a detailed explanation of each flowchart.

[0096] This embodiment employs a neural network to fit value functions of arbitrary forms, ensuring sufficient expressive power to handle the current continuous control task. A swarm intelligence optimization algorithm is used to search for the optimal action in the current state. GPU-accelerated batch processing mode is used for multi-function search to reduce computational complexity. Individuals from multiple populations are selected as candidate actions, and a priority exploration strategy is used to balance exploration and utilization to complete the decision (action) formulation. Through interaction with the environment using the selected action, rewards and state transitions are obtained, and the neural network is updated using mean squared error. This process is iterative, ultimately achieving the goal of solving continuous control problems using value reinforcement learning. The specific implementation steps are as follows:

[0097] 1. Constructing value functions

[0098] To ensure generality, this invention uses a multi-layer neural network to fit the value function, with the key component being the network structure.

[0099] (1) Input layer

[0100] The number of nodes in the input layer is the same as the length of the input feature. This embodiment addresses a continuous control problem, where it is impossible to enumerate all executable actions. Therefore, the input layer consists of states s. t and action a t A multi-head input network consisting of two parts.

[0101] out0 = (s t ,a t )

[0102] Among them, s t For state, a t For the action, out0 is the feature vector corresponding to the state-action pair;

[0103] (2) Hidden layer

[0104] The hidden layer consists of three fully connected layers with multiple nodes, using ReLU activation. Let the input of the i-th layer be out. i-1 Then this layer can be formally described as:

[0105]

[0106] in, ReLU is an activation function. Let θ be the output vector of the (i-1)th layer. i Let the weights be those of the i-th layer.

[0107] In this embodiment, the output is a real number vector with the same length as the number of hidden layer nodes.

[0108] (3) Output layer

[0109] The output layer contains the values ​​of state-action pairs; therefore, it has only one node. Furthermore, since the range of action values ​​is unknown, the output layer has no activation function.

[0110]

[0111] Where out4 is the output vector of the output layer (the value of the action, a one-dimensional real number). θ is the output vector of the hidden layer, and θ4 is the weight of the output layer.

[0112] In this embodiment, the output is the value of the action, a single real number.

[0113] 2. Search for the optimal action

[0114] Let the value function based on the neural network be f(s) t ,a t Since the value function's expression is unknown, an analytical solution cannot be constructed. Furthermore, the value function may be non-convex, limiting the effectiveness of traditional numerical methods. Therefore, this embodiment uses a swarm intelligence optimization algorithm to achieve action optimization. Without loss of generality, this embodiment uses particle swarm optimization as an example.

[0115] (1) Particle Swarm Optimization (PSO)

[0116] The specific steps of particle swarm optimization are as follows:

[0117] 1) Determine the target to be optimized

[0118] 2) Initialize the population, which includes the positions x of N individuals. i and speed v i .

[0119] 3) Update the position and velocity of each body according to the following formula:

[0120] v i =wv i +r1c1(x i,l -x i )+r2c2(x g -x i )

[0121] x i =xi +v i

[0122] Where w is the inertia factor, c {1,2} It is the learning factor, r {1,2} ∈[0,1] is a random number, x i,l and Let represent the local optimal solution for the i-th known individual and the global optimal solution known for the population, respectively.

[0123] 4) Repeat step 3) until the maximum number of iterations is reached.

[0124] 5) Output the optimal individual x g .

[0125] In this embodiment, the output of this step is the optimal individual. Since the optimization is performed on the action, the optimal individual is the action, which is a real number vector.

[0126] (2) Action optimization

[0127] 1) Determine the target to be optimized f(s) t ,a t ), due to s t Therefore, the value function based on the neural network can be further rewritten as g(a t )=f(s t (a) t ).

[0128] 2) Input the target to be optimized into the particle swarm optimization algorithm and wait for the output:

[0129] a t+1 =PSO(g)

[0130] In this embodiment, this step involves selecting an action, so the output is still a real number vector.

[0131] 3. GPU-accelerated batch action optimization

[0132] Using a neural network as the value function ensures the value function has strong expressive power while increasing the number of samples required for training (the amount of data needed for training, corresponding to <states s> in the bipedal robot example). t -Action a t -Rewards t+1 -states t+1The tuple is composed of tuples, where the subscript t indicates time t. However, gradient descent on all samples and stochastic gradient descent on a single sample both have certain limitations. Therefore, batch training is usually used when training neural networks. Batch training greatly hinders step 2, searching for the optimal action. This embodiment improves batch action optimization by accelerating it with GPU, based on changing the particle swarm update method.

[0133] Batch processing takes a set of independent training samples as input and outputs multiple actions, the number of which depends on the batch size. For example, with a batch size of 20 and an action length of 5, the output is a 20×5 real matrix. Each row represents an action.

[0134] (1) Matrix acceleration

[0135] The specific steps for accelerating batch action optimization through matrix transformation are as follows:

[0136] 1) Matrix-form particle swarm optimization algorithm

[0137] Let P = {x} i} N V = {v i} N Lb={x i,l} N Then the particle swarm update rule can be adjusted as follows:

[0138] V = wV + r1c1(Lb-P) + r2c2(x) g -P)

[0139] P = P + V

[0140] Where V is the particle velocity matrix, w is the inertia weight, r1 is a random number in the interval [0,1], c1 is the local learning factor, Lb is the local optimal individual, P is the particle position matrix, r2 is a random number in the interval [0,1], c2 is the global learning factor, and x g It is the globally optimal individual.

[0141] 2) Constructing large populations

[0142] If the batch size is m, then with the value function f unchanged, we need to input m states and find m actions simultaneously. That is, we need to construct m populations at the same time.

[0143] In this embodiment, m populations are merged into a single population of size (N×M)×(|s). t |+|a t The matrix is ​​a series of rows, each representing the position of an individual. The velocity matrices are constructed in the same way, denoted as Pos and Vel, respectively.

[0144] 3) Record the local and global optimal individuals

[0145] By inputting the large population position Pos into the value function f, the fitness matrix of each individual can be obtained, denoted as . The one-dimensional matrix Fit is transformed into an N×M two-dimensional matrix using the reshape function. Each column represents the fitness of an individual in the population. The global and local optima can then be obtained using the argmax function, denoted as X. g And Pbest.

[0146] 4) Update individual location

[0147] In batch training scenarios, the objective function to be optimized is unique, namely the value function f. To distinguish different states, the state needs to be encoded into the individual's position. Simultaneously, the state does not participate in updates; therefore, the velocity component corresponding to the state is 0. Let there exist a mask matrix. Among them, M i = (0,0,…,0,1,…,1), the first |s t All elements are 0, and all other elements are 1. In this case, the individual update can be formally described as:

[0148] Vel=wVel+r1c1(Pbest-Pos)+r2c2(X g -Pos)

[0149] Pos = Pos + M·Vel

[0150] (2) Theoretical guarantee

[0151] The theoretical proof clarifies the relationship between the number of individuals, the number of iterations, and the difference between the final output individual and the optimal individual. The theoretical conclusions are described as follows:

[0152] If the known optimal solution obtained by a swarm intelligence optimization algorithm (e.g., PSO) in each iteration is no worse than the optimal solution obtained by random search, then there exists Δ∈(0,1] such that the known optimal solution x g and the true optimal solution x * The difference between |x g -x * | Satisfies:

[0153]

[0154] Where R, ρ, N, L and n represent the radius of the search space, the space compression ratio (i.e., the ratio of the search space after each iteration to the original search space), the number of individuals in the population, the number of iterations, and the dimension of the search space, respectively.

[0155] Theoretical conclusions show that increasing the number of individuals and increasing the number of iterations both reduce the error, but increasing the number of iterations cannot reduce the algorithm's complexity through parallelization. Therefore, increasing the number of individuals to reduce the error is the technical approach chosen in this invention.

[0156] 4. Individual-based candidate actions

[0157] Another problem in continuous control tasks is that the continuous action space makes it difficult to balance exploration and development. This invention addresses this by randomly merging multiple individuals from different periods to form a candidate action set, while preserving the optimal individual.

[0158] Let the set of candidate actions based on the individual be... The total number of iterations is L. Therefore, That is, in In the next iteration, an individual is randomly selected from the population to add to the candidate action set.

[0159] Based on the action candidate set The softmax operator is used to randomly sample and execute actions. That is:

[0160]

[0161] 5. Update value function

[0162] This embodiment uses the square of the time difference error as the loss function, optimizes the weights of each node in the network through backpropagation of a neural network, and finally obtains the optimal value function. Let the reward obtained from the interaction with the environment and the state after the transition be r, respectively. t+1 and s t+1 The time difference error can then be denoted as:

[0163]

[0164] Where γ is the discount factor, Treat them as numbers and do not include them in gradient calculation.

[0165] In this embodiment, this step involves updating the node weights (parameters) within the neural network.

[0166] 6. Iterative learning

[0167] The state s to be acquired at time t+1 t+1 As input to the neural network, steps 2-6 are executed sequentially until the maximum number of iterations is reached. Let the neural network be f(s). t ,a t If the optimal strategy can be determined from the current value function, then:

[0168]

[0169] Wherein, π(s) t ) is the strategy, f(s) t a) represents state s t The value of the next action 'a'.

[0170] Output the optimal control strategy, that is, a function that obtains the optimal action based on the current state.

[0171] The technical effect of this embodiment is as follows:

[0172] Reinforcement learning is a data-driven, learning-based control method. Therefore, it is domain-independent (only related to the data), reducing expert involvement and lowering labor costs. Furthermore, reinforcement learning handles uncertainty well, enabling it to address control tasks in complex, time-varying, uncertain, and open environments. Moreover, after outputting the optimal policy, reinforcement learning can use the current state as input to quickly obtain actions through a neural network, with the state dimension having a minimal impact on computation time. Therefore, reinforcement learning provides new ideas and approaches for solving continuous control problems.

[0173] This embodiment addresses the challenge of optimal action sampling in value-based reinforcement learning by starting with a data-driven intelligent control approach. Specifically, it incorporates swarm intelligence optimization algorithms to optimize actions under complex value functions; utilizes a GPU platform to improve the efficiency of optimal action search in batch training scenarios; and provides theoretical guidance for setting key parameters in the action optimization stage. This effectively solves the problem of optimal action sampling in value-based reinforcement learning when solving continuous control problems.

[0174] This embodiment avoids methods such as 1) value function simplification; 2) action space discretization; and 3) action distribution priors used to obtain the optimal action, and achieves action sampling within a finite time while ensuring the expressive power of the value function.

[0175] This embodiment, when solving continuous control problems, achieves similar reward returns to other algorithms while using only 1 / 10 of the interaction count. Furthermore, this embodiment is an algorithmic framework with good scalability, and can be combined with other algorithms besides the examples (Q-learning and particle swarm optimization) to further improve learning efficiency. In other words, the time cost required for learning is reduced.

[0176] In summary, this invention provides a learning-based, data-driven, and effective control method for continuous control tasks in complex environments.

[0177] The above are merely preferred embodiments of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A continuous control method based on value reinforcement learning, characterized in that, This method is applied to two groups of robots, which, from top to bottom, consist of: leg joints, left and right knee joints, and left and right ankle joints. Each joint is connected by hinges that can be controlled to open and close. Data related to reinforcement learning algorithms includes: Status: The opening and closing angles of each joint hinge and the corresponding angular velocities, plus the current height and position of the leg joint; Action: The torque applied to each joint is used to adjust the walking posture of the bipedal robot; Reward: The cost of moving horizontal distances and adjusting the posture of each joint with the bipedal robot, including the sum of squares of the movements; The continuous control method includes: Construct a value function for a multilayer neural network model that takes the state as input; A swarm intelligence optimization algorithm is used to search for the optimal action of the value function in any state; Based on the aforementioned swarm intelligence optimization algorithm, GPU acceleration is used to optimize actions in batch states to obtain the optimal actions in the batch. Based on the batch optimal actions, construct a candidate action set; Based on the candidate action set, select an action to execute and update the value function; The function to update the value includes: The optimal value function is obtained by using the square of the time difference error as the loss function and optimizing the weights of each node in the network through backpropagation of the neural network. The time difference error is: in, As a discount factor, Treat it as a number; The updated value function is iterated through a loop of state input-action search-action execution-state input to obtain the optimal strategy for the value function.

2. The continuous control method based on value reinforcement learning according to claim 1, characterized in that, The multi-layer neural network model includes: an input layer, a hidden layer, and an output layer; The input layer is: in, For state, For action, The feature vector corresponding to the state-action pair; The hidden layer is: in, For activation function, For the first The output vector of the layer, For the first Layer weights; The output layer is: in, The output vector of the output layer. The output vector of the hidden layer. These are the output layer weights.

3. The continuous control method based on value reinforcement learning according to claim 1, characterized in that, Searching for the optimal action of the value function in any state includes: Based on the value function and the input state, the target to be optimized is obtained; Based on the swarm intelligence optimization algorithm, the target to be optimized is subjected to action optimization processing to obtain the optimal action.

4. The continuous control method based on value reinforcement learning according to claim 1, characterized in that, Obtaining the optimal batch action includes: Based on batch input states, construct a large population related to the number of input states; The large population is input into the value function to obtain the locally optimal individual and the globally optimal individual. Construct a matrix-form particle swarm optimization model; Based on the particle swarm optimization model, the individual positions of the locally optimal individual and the globally optimal individual are updated to obtain the batch optimal action.

5. The continuous control method based on value reinforcement learning according to claim 4, characterized in that, The particle swarm optimization model is as follows: in, The velocity matrix of the particles. For inertial weights, for Random numbers within the interval For local learning factors, For a locally optimal individual, Let be the position matrix of the particles. for Random numbers within the interval As a global learning factor, It is the globally optimal individual.

6. The continuous control method based on value reinforcement learning according to claim 4, characterized in that, Obtaining the locally optimal individual and the globally optimal individual includes: The large population is input into the value function to obtain the fitness matrix of each individual. Perform a matrix transformation of a preset dimension on the fitness matrix; The transformed fitness matrix is ​​processed by the argmax function to obtain the local optimal individual and the global optimal individual.

7. The continuous control method based on value reinforcement learning according to claim 1, characterized in that, Based on the candidate action set, the method for selecting the action to be executed is: random sampling using the softmax operator; The softmax operator is randomly sampled as follows: in, In the state Select action The probability, For state Next action The value, It is a time step. and All are indexes.

8. The continuous control method based on value reinforcement learning according to claim 1, characterized in that, The optimal strategy is: in, As a strategy, For state Next action The value of .

Images