Hierarchical controller based on symbolic language and reinforcement learning

Abstract

The application discloses a hierarchical controller based on symbolic language and reinforcement learning, which comprises an environment discretization preprocessing module, a pre-training model generation module, a training model generation module and a control execution module. The pre-training model generation module generates a training model before the training model generation module, and the neural network parameters of each lower layer strategy in the pre-training model generation module are fixed. When the pre-training model generation module is trained, the training execution of the training model generation module is entered. The application can achieve a higher success rate in basic control of a mechanical arm task, has better generalization, and the success rate of control execution can reach about 98%.

Description

Technical Field

[0001] This invention relates to the fields of computer science and artificial intelligence, and more specifically, to a hierarchical controller based on symbolic language and reinforcement learning. Background Technology

[0002] Currently, the best approach in computer science for controlling simulated robotic arms is to break down tasks into sub-tasks using human logic. This involves a planner at the upper level of the control model using human logic to break down the task into a series of detailed actions, while other controllers at the lower level of the control model are used to complete these detailed actions.

[0003] The lower-level controller of the control model can be a controller based on classical mechanical algorithms or a controller based on machine learning algorithms. The main advantage of machine learning algorithm control is that it can adaptively learn the control task, which can save human resources in the design and operation of algorithms, and requires less mathematical and mechanical knowledge.

[0004] However, in the aforementioned existing control methods, the upper-level planner of the control model still primarily relies on human guidance to break down the entire task of controlling the robotic arm. Therefore, the following problems exist:

[0005] On the one hand, the upper-level planner's breakdown of the entire robotic arm control task places high demands on the user's understanding of the task, requiring the user to have a complete understanding of the entire task.

[0006] On the other hand, such a design approach requires redesigning the model for each specific task, lacking generalization ability. Summary of the Invention

[0007] The purpose of this invention is to provide a hierarchical controller method based on symbolic language and reinforcement learning, which can systematically solve the problem of continuous control of robotic arms and has generalization ability.

[0008] To achieve the above objectives, the technical solution of the present invention is as follows:

[0009] A hierarchical controller based on symbolic language and reinforcement learning, comprising:

[0010] Taking the task of controlling a robotic arm as an example, the technical effects of this invention are mainly reflected in the following aspects:

[0011] ① Compared with previous research results, this invention achieves a higher success rate in basic robotic arm control tasks. Previous research results showed a success rate of around 92% in object-grabbing experiments, while this invention achieves a final success rate of 98%.

[0012] ② The present invention has better generalizability

[0013] When performing similar tasks, such as gripping objects whose size or shape changes, the algorithm of this invention does not require complete retraining. The upper-level logic machine model of this invention does not need to be changed, and the machine learning algorithm for fine-tuning the lower-level control can complete the task again. Attached Figure Description

[0014] Figure 1 The diagram shown is a schematic representation of the hierarchical controller based on symbolic language and reinforcement learning in an embodiment of the present invention. Detailed Implementation

[0015] The following is in conjunction with the appendix Figure 1 The specific embodiments of the present invention will be further described in detail below.

[0016] In the following detailed description of the embodiments of the present invention, in order to clearly illustrate the structure of the present invention for ease of explanation, the structures in the accompanying drawings are not drawn to scale and have been partially enlarged, deformed and simplified. Therefore, it should be avoided to interpret this as a limitation of the present invention.

[0017] Please see Figure 1 , Figure 1 The diagram shown is a schematic representation of the hierarchical controller based on symbolic language and reinforcement learning in an embodiment of the present invention. Figure 1 As shown, the hierarchical controller includes an environment discretization preprocessing module, a pre-trained model generation module, a trained model generation module, and a control execution module.

[0018] Both the pre-trained model generation module and the trained model generation module include the following:

[0019] Upper-level planner: Neural Logic Machine

[0020] Evaluation Network: Composed of neural networks, it receives input from the environment and the policy output by the neural logic machine. It scores the choices made by the neural logic machine and optimizes the neural logic machine accordingly.

[0021] Normalization layer: The normalization layer uses the softmax function to normalize the output of the neural logic machine, making it a probability distribution for selecting the next policy layer.

[0022] Lower-level policy set: The lower-level policy set contains a series of policies, assuming there are n policies. During each execution, the upper-level controller selects one of them to execute.

[0023] The pre-trained model generation module accepts input from the environment, performs logical assertions to judge useful environmental information, and then inputs the true values ​​of the judgments into the upper-level planner in the form of a vector.

[0024] The pre-trained model generation module generates the training model before the training model generation module. In the pre-trained model generation module, the neural network parameters of each lower-level strategy are fixed. After the pre-trained model generation module completes its training, the training is then executed in the training model generation module.

[0025] In an embodiment of the present invention, the environment discretization preprocessing module accepts the state vector input of environmental information, performs logical assertion judgment on the useful environmental information, and then outputs the judgment truth value in the form of a first state vector; wherein, the dimension of the first state vector is associated with the number of input judgment truth value variables.

[0026] A pre-trained model generation module includes an upper-layer planner and a lower-layer controller. The upper-layer planner determines its current state vector and selects a lower-layer policy based on the true values ​​corresponding to the state vector. It includes a Neural Logic Machine (NLM) model and a normalization layer. The NLM model receives the true values ​​as a vector and outputs a second state vector. The second state vector is input to the normalization layer, which uses a softmax function to normalize the output of the NLM model, generating a probability distribution for selecting the lower-layer policy. The dimension of the second state vector is the same as the number of lower-layer policies; the dimension of the second state vector is less than or equal to the dimension of the first state vector. The layer controller comprises a combination of N lower-level strategies, each with fixed neural network parameters. During each execution, the upper-level planner selects one of the lower-level strategies based on its probability distribution. Each lower-level strategy corresponds to a simple subtask based on the state vector of the environment information. The neural network of the lower-level strategy outputs a control action to the environment for execution, and provides a reward value indicating whether the task is completed based on the execution result. In other words, the lower-level controller uses the reward obtained after the last step of the lower-level strategy is executed as feedback to the neural logic machine model (NLM) to indicate whether the selected strategy is appropriate, thereby optimizing the NLM model.

[0027] The training model generation module also includes an upper-layer planner and a lower-layer controller, but the neural network parameters of each lower-layer policy are no longer fixed. The upper-layer planner is used to determine its current state vector and select a lower-layer policy based on the true value corresponding to the state vector. It includes an evaluation network, a neural logic machine model (NLM), and a normalization layer. The evaluation network receives the true value output as a vector and feeds it back to the NLM, scoring the selected lower-layer policy and optimizing its neural network parameters. The NLM receives the true value output as a vector and outputs the second state vector. The second state vector is input to the normalization layer, which uses a softmax function to normalize the output of the NLM, generating a probability distribution for selecting the lower-layer policy. The dimension of the second state vector is the same as the number of lower-layer policies; the dimension of the second state vector is less than or equal to the dimension of the first state vector. The lower-level controller comprises a combination of N lower-level strategies. During each execution, the upper-level planner selects one of the lower-level strategies to execute based on its probability distribution. Each lower-level strategy corresponds to a simple subtask based on the state vector of the environment information. A control action is output from the neural network of the lower-level strategy to the environment for execution, and a reward value indicating whether the task is completed is fed back based on the execution result of the environment. That is, the lower-level controller uses the reward obtained after the last step of the lower-level strategy is completed as feedback to the neural logic machine model (NLM) on whether the selected strategy is appropriate, so as to optimize the model of the neural logic machine model (NLM).

[0028] After the above model training is completed, the control execution module uses the neural logic machine model (NLM) optimized by the training model generation module and the optimized neural network model of the lower-level policy, inputs the state vector of the environmental information, and outputs the lower-level policy.

[0029] The following is a detailed introduction to the specific application of the model and algorithm in the robotic arm environment:

[0030] The robotic arm environment is a simulation environment. Environmental information is output as vectors with 25 dimensions, including the position and velocity of the object, the position and velocity of the robotic arm, the distance between the robotic arm's grippers, and the target position. Control information for the robotic arm needs to be input as four-dimensional vectors, representing the target position the robotic arm needs to achieve and the required distance between the two grippers.

[0031] The preprocessing section uses feature extraction algorithms to extract key features from the environment and present them in a symbolic language as input to the upper-level logic machine planner. Specifically, the environmental information is input as a vector, which in the robotic arm environment is a 25-dimensional vector. The most crucial information is in the first 11 dimensions, including the 3D coordinates of the robotic arm gripper, the 3D coordinates of the object to be gripped, the 3D coordinates of the target point to be reached, and the distance between the two grippers. After obtaining this raw data, a series of conditional statements are used to evaluate these distance values, returning a true value. For example, assuming S is the environmental vector information and s1 is the first component of S, if the value of s1 is 0.4, then 0.5 > s1 > 0.3 will return true, and s1 < 0.2 will return false. In the robotic arm task, the specific range is set to [0.0, 0.3, 0.5, 0.6, 0.7, 0.8, 0.9, 1]. All ranges are evaluated, and all information is recorded using a large input vector. For all true logical expressions, the corresponding position in the vector has a value of 1; otherwise, it has a value of 0. For the case mentioned above where s1 = 0.4, the resulting vector is [0, 1, 0, 0, 0, 0, 0]. This operation is performed on all s1-s11, merging the resulting vectors into a single large truth vector. The merged truth vector is then passed to the upper-level planner.

[0032] The upper-level logic planner is a neural network-like structure that selects an appropriate policy based on the input truth vector. During training, assuming the input truth vector to the upper-level planner is T, the output of the upper-level planner is a vector of dimension n. The value of the i-th dimension of this vector corresponds to the probability of selecting the lower-level policy i. In the robotic arm task, the selectable sub-policies are: approach the block, grasp the block, lift the block to a certain height, and move the block to the target position. During training, after the lower-level policy i executes a series of steps, if the environmental task is completed, it will provide feedback to the upper-level controller with a value of 1; otherwise, the feedback to the upper-level controller will be 0. The upper-level planner continuously optimizes based on the environmental feedback, using the Proximal Policy Optimization (PPO) algorithm, with the goal of finding a series of policies that result in a feedback value of 1. Ultimately, this achieves the effect of selecting the appropriate sub-policy at the appropriate task stage: initially, the upper-level planner will choose the "approach the block" policy. Then, after the input is determined to be a proximity action, the upper layer will select "clamp the block," and finally, once the block is clamped, it will select the strategy of "moving the block to the target position."

[0033] After the upper-level planner selects a lower-level strategy, the selected strategy i interacts directly with the environment, obtaining the environment's state vector S. It then outputs a specific control action 'a' through a neural network. In the robotic arm environment, 'a' is a 4-dimensional vector, where the first three dimensions represent the target position the robotic arm needs to reach, and the fourth dimension is the distance between the two grippers. The robotic arm then reaches the target position and the target gripper width based on the output 'a'.

[0034] The training of the above model is accomplished through continuous interaction with the environment. In the pre-training phase, the parameters of the lower-level policies are fixed, and only the upper-level policies are trained. That is, the upper-level planner selects a lower-level policy and executes it for several steps each time, then acquires new environmental information, performs preprocessing, inputs it into the upper-level planner, and then the upper-level planner selects the next policy to continue execution. This allows the upper-level policy to roughly understand the logical relationships between the subtasks of the entire task. For example, in a robotic arm control task, the upper-level policy can roughly learn that it first needs to perform a "approach" action, then a "grip the block" action, and finally a "move the block to the target position" operation.

[0035] After pre-training, co-training is required to further improve the success rate of task completion. During this stage, the parameters of both the upper-level planner and the lower-level policy are adjustable. At each training step, the environment is executed from scratch, acquiring information about the environment and the policies executed by both the upper and lower layers at each step, and then both the upper-level planner and the lower-level policy are optimized simultaneously. The co-training algorithm uses a reward function to obtain feedback on the lower-level policy actions. This reward function consists of two parts: one part comes directly from the feedback from the environment, and the other part is provided by the upper-level controller. When the upper-level controller determines that a subtask is completed, it gives the lower-level policy an additional reward. This allows the neural network to learn appropriate actions.

[0036] Furthermore, the lower-level controller also needs to adaptively adjust when task details change. For example, when the size of the block changes or the target position to be moved changes, it needs to fine-tune through certain steps to adapt to the new target. Therefore, in these cases, the lower-level controller also needs to be trained together, and the co-training algorithm mentioned above can be used again. Thus, the model can achieve adaptive adjustment when task details change, solving similar control problems of the same type. Moreover, this adaptive adjustment is faster and more efficient than directly training a new strategy from scratch.

[0037] In an embodiment of the present invention, a control method employing the above-described hierarchical controller based on symbolic language and reinforcement learning is also included, comprising a pre-trained model generation step S1, a trained model generation step S2, and a control execution step S3.

[0038] Step S1 may specifically include:

[0039] Step S11: Fix the neural network parameters of each lower-level policy; the Neural Logic Machine (NLM) model accepts the true values ​​of the judgments output in vector form and outputs the second state vector; the second state vector is input to the normalization layer, which uses the softmax function to normalize the output of the NLM model to generate a probability distribution for selecting the lower-level policy; wherein, the dimension of the second state vector is the same as the number of lower-level policies; the dimension of the second state vector is less than or equal to the dimension of the first state vector;

[0040] Step S12: The evaluation network receives the true judgment values ​​output as vectors and feeds them back to the neural logic machine model (NLM), scoring the selected lower-level policy and optimizing the neural network parameters of the lower-level policy; the NLM receives the true judgment values ​​output as vectors and outputs the second state vector; the second state vector is input to the normalization layer, which uses the softmax function to normalize the output of the NLM, generating a probability distribution for selecting the lower-level policy; wherein, the dimension of the second state vector is the same as the number of lower-level policies; the dimension of the second state vector is less than or equal to the dimension of the first state vector;

[0041] The training model generation step S2 specifically includes:

[0042] Step S21: The evaluation network receives the true judgment values ​​output as vectors and feeds them back to the neural logic machine model (NLM), scoring the selected lower-level policy and optimizing the neural network parameters of the lower-level policy; the NLM receives the true judgment values ​​output as vectors and outputs the second state vector; the second state vector is input to the normalization layer, which uses the softmax function to normalize the output of the NLM and generate a probability distribution for selecting the lower-level policy; wherein, the dimension of the second state vector is the same as the number of lower-level policies; the dimension of the second state vector is less than or equal to the dimension of the first state vector;

[0043] Step S22: During each execution, the upper-level planner selects one of the lower-level strategies to execute based on the probability distribution of the lower-level strategies. Each lower-level strategy corresponds to a simple subtask based on the state vector of the environmental information. The neural network of the lower-level strategy outputs a control action to the environment for execution, and feeds back a reward value indicating whether the task is completed based on the execution result of the environment. That is, the lower-level controller uses the reward obtained after the last step of the lower-level strategy is completed as feedback to the neural logic machine model (NLM) on whether the strategy selected by the NLM is appropriate, so as to optimize the model of the NLM.

[0044] Step S3 involves using the optimized Neural Logic Machine (NLM) model generated by the training model generation module and the optimized neural network model of the lower-level policy, inputting the state vector of the environmental information, and outputting the lower-level policy.

[0045] One specific implementation of this invention is in a simulation simulator for a robotic arm grasping a block. The robotic arm simulation environment has four control dimensions. The first three dimensions output the coordinates of the robotic arm gripper in three-dimensional space, and the fourth dimension controls the distance between the two claws of the robotic arm gripper. In the robotic arm control simulation environment, the model receives input from the environment, senses the information in the environment, and outputs the control parameters for each of these four dimensions for each step. Guided by this, the robotic arm first approaches the block, then clamps the block, and finally clamps the block to the designated position.

[0046] The upper-level planning controller, mentioned in the previous section, determines the current state of the entire task, or in other words, whether the subtask to be completed is to approach the block, grasp the block, or move the block to a designated position. Once the upper-level planner has determined the task, it selects a specified strategy from the lower-level strategy set to complete the task. The strategy in the lower-level strategy set will then provide specific control over the task, outputting control information in four dimensions. Thus, the upper and lower layers collaborate to complete the entire task of grasping the block.

[0047] The results show that the present invention can achieve a higher success rate in basic control robotic arm tasks, has better generalization ability, and the success rate of control execution can reach about 98%.

[0048] The above description is merely a preferred embodiment of the present invention. The embodiments are not intended to limit the scope of patent protection of the present invention. Therefore, any equivalent structural changes made based on the description and drawings of the present invention should also be included within the scope of protection of the present invention.

Claims

1. A hierarchical controller based on symbolic language and reinforcement learning, characterized in that, include: The environment discretization preprocessing module accepts the state vector input of environmental information, performs logical assertion judgment on the useful environmental information, and then outputs the judgment truth value in the form of a first state vector; wherein, the dimension of the first state vector is related to the number of input judgment truth value variables; The pre-trained model generation module includes: An upper-level planner is used to determine its current state vector and select a lower-level strategy based on the true values ​​corresponding to the state vector. It includes a Neural Logic Machine (NLM) model and a normalization layer. The NLM model receives the true values ​​as a vector and outputs a second state vector. The second state vector is input to the normalization layer, which uses a softmax function to normalize the output of the NLM model, generating a probability distribution for selecting the lower-level strategy. The dimension of the second state vector is the same as the number of lower-level strategies; the dimension of the second state vector is less than or equal to the dimension of the first state vector. The lower-level controller comprises a combination of N lower-level strategies, each with fixed neural network parameters. During each execution, the upper-level planner selects one of the lower-level strategies based on its probability distribution. Each lower-level strategy corresponds to a simple subtask based on the state vector of the environment information. The neural network of the lower-level strategy outputs a control action to the environment for execution, and provides a reward value indicating whether the task is completed based on the execution result. In other words, the lower-level controller uses the reward obtained after the final step of the lower-level strategy as feedback on whether the strategy selected by the neural logic machine model (NLM) is appropriate, thereby optimizing the NLM model. The training model generation module includes: An upper-level planner is used to determine its current state vector and select a lower-level strategy based on the true values ​​corresponding to the state vector. It includes an evaluation network, a neural logic machine (NLM) model, and a normalization layer. The evaluation network receives the true values ​​output as a vector and feeds them back to the lower-level controller, scoring the selected lower-level strategy and optimizing the neural network parameters of the lower-level strategy. The NLM receives the true values ​​output as a vector and outputs a second state vector. The second state vector is input to the normalization layer, which uses a softmax function to normalize the output of the NLM model, generating a probability distribution for selecting the lower-level strategy. The dimension of the second state vector is the same as the number of lower-level strategies; the dimension of the second state vector is less than or equal to the dimension of the first state vector. The lower-level controller comprises a combination of N lower-level strategies. During each execution, the upper-level planner selects one of the lower-level strategies to execute based on its probability distribution. Each lower-level strategy corresponds to a simple subtask based on the state vector of the environment information. A control action is output from the neural network of the lower-level strategy to the environment for execution, and a reward value indicating whether the task is completed is fed back based on the execution result of the environment. That is, the lower-level controller uses the reward obtained after the last step of the lower-level strategy is executed as feedback to the neural logic machine model (NLM) on whether the strategy selected by the NLM is appropriate, so as to optimize the model of the NLM. The control execution module uses the neural logic machine model (NLM) optimized by the training model generation module and the optimized neural network model of the lower-level policy, inputs the state vector of the environmental information, and outputs the lower-level policy.

2. A control method employing the hierarchical controller based on symbolic language and reinforcement learning as described in claim 1, characterized in that, This includes the pre-trained model generation step S1 and the trained model generation step S2; Step S1 specifically includes: Step S11: Fix the neural network parameters of each lower-level policy; the Neural Logic Machine (NLM) model accepts the true values ​​of the judgments output in vector form and outputs the second state vector; the second state vector is input to the normalization layer, which uses the softmax function to normalize the output of the NLM model to generate a probability distribution for selecting the lower-level policy; wherein, the dimension of the second state vector is the same as the number of lower-level policies; the dimension of the second state vector is less than or equal to the dimension of the first state vector; Step S12: During each execution, the upper-level planner selects one of the lower-level strategies to execute based on the probability distribution of the lower-level strategies. Each lower-level strategy corresponds to a simple subtask based on the state vector of the environmental information. The neural network of the lower-level strategy outputs a control action to the environment for execution, and feeds back a reward value indicating whether the task is completed based on the execution result of the environment. That is, the lower-level controller uses the reward obtained after the last step of the lower-level strategy is completed as feedback to the neural logic machine model (NLM) on whether the strategy selected by the NLM is appropriate, so as to optimize the model of the NLM. The training model generation step S2 specifically includes: Step S21: The evaluation network receives the true judgment values ​​output in vector form and feeds them back to the lower-level controller to score the selected lower-level policy and optimize the neural network parameters of the lower-level policy; the neural logic machine model (NLM) receives the true judgment values ​​output in vector form and outputs the second state vector; the second state vector is input to the normalization layer, which uses the softmax function to normalize the output of the NLM and generate a probability distribution for selecting the lower-level policy; wherein, the dimension of the second state vector is the same as the number of lower-level policies; the dimension of the second state vector is less than or equal to the dimension of the first state vector; Step S22: During each execution, the upper-level planner selects one of the lower-level strategies to execute based on the probability distribution of the lower-level strategies. Each lower-level strategy corresponds to a simple subtask based on the state vector of the environmental information. The neural network of the lower-level strategy outputs a control action to the environment for execution, and feeds back a reward value indicating whether the task is completed based on the execution result of the environment. That is, the lower-level controller uses the reward obtained after the last step of the lower-level strategy is completed as feedback to the neural logic machine model (NLM) on whether the strategy selected by the NLM is appropriate, so as to optimize the model of the NLM.

3. The control method for a hierarchical controller based on symbolic language and reinforcement learning according to claim 2, characterized in that, The process includes a control execution step S3, which uses the neural logic machine model (NLM) optimized by the training model generation module and the optimized neural network model of the lower-level policy, inputs the state vector of the environmental information, and outputs the lower-level policy.

Images