A task planning method based on symbolic options and self-learning of action models

Abstract

The present invention discloses a task planning method based on self-learning of symbolic options and action models. The method includes: based on a symbolic state mapping module, mapping high-dimensional image data into symbolic states according to prior knowledge and digital image processing; based on an option set module, constructing an option set according to random actions and symbolic states; based on an action model learning module, learning the action model and updating the option set according to the change of the symbolic state to obtain a new action model and planning target; based on a planner module, solving the planning path according to the new action model and planning target, and exploring whether there is a new action model based on the planning result. By using the present invention, it is possible to automatically learn the action model and symbolic options and the corresponding relationship between them, perform planning and training exploration, and obtain an action strategy for solving the problem. As a task planning method based on self-learning of symbolic options and action models, the present invention can be widely used in the field of symbolic planning.

Description

Technical Field

[0001] The present invention relates to the field of symbolic planning, and in particular to a task planning method based on symbolic options and action model self-learning. Background Art

[0002] Symbolic planning, also known as intelligent planning, is a key area of ​​artificial intelligence. Its key idea is to abstract environmental states and actions into a symbolic representation based on human prior knowledge. Using these symbolic representations of the initial state, target state, and action model, a symbolic planner solves a feasible sequence of actions from the initial state to the target state. This sequence of actions is also called a plan. There are many symbolic planning languages, such as PDDL or action language, and each has its own corresponding planner, such as FASTDOWNWARD and CLINGO. Symbolic planning is highly interpretable and does not require extensive interaction with the environment. However, because it requires human prior knowledge to abstract the environment and actions, it struggles to adapt to complex and changing high-dimensional environments. Summary of the Invention

[0003] In order to solve the above technical problems, the purpose of the present invention is to provide a task planning method based on self-learning of symbolic options and action models. According to the exploration trajectory of the intelligent agent in the environment, the action model and symbolic options as well as the correspondence between them are automatically learned, and the learned action model is used for planning, so as to select symbolic options for training and exploration, and obtain the action strategy for solving the problem.

[0004] The first technical solution adopted by the present invention is: a task planning method based on symbol selection and action model self-learning, comprising the following steps:

[0005] S1, based on the symbolic state mapping module, maps high-dimensional image data into symbolic states according to prior knowledge and digital image processing;

[0006] S2, based on the option set module, constructs an option set based on random actions and symbolic states;

[0007] S3, based on the action model learning module, learns the action model and updates the option set according to the change of the symbol state to obtain the new action model and planning target;

[0008] S4. Based on the planner module, solve the planning path according to the new motion model and planning goal, and explore whether there is a new motion model based on the planning results.

[0009] Furthermore, the step of mapping the high-dimensional image data into symbolic states based on the symbolic state mapping module according to prior knowledge and digital image processing specifically includes:

[0010] Determine the intelligent agent based on prior knowledge and digital image processing, locate the position of the intelligent agent, and obtain the symbolic proposition of the position of the intelligent agent;

[0011] According to the symbolic proposition of the agent's position, a high-level state is mapped to a symbolic state consisting of two propositions.

[0012] Furthermore, the option set consists of multiple symbol state options and a random option.

[0013] Furthermore, the symbolic state option and the random option are both represented by triples, the symbolic state option includes a precondition, an internal strategy and an effect, and the random option includes an initial condition, a strategy and a termination condition.

[0014] Furthermore, the step of constructing an option set based on the option set module according to the random action and the symbol state specifically includes:

[0015] Options make decisions based on the state of the environment and choose the actions to perform;

[0016] After the agent performs an action, the environment returns a new high-dimensional state and reward to the option;

[0017] The option adjusts the strategy based on the returned reward and state, and continues to the next interaction until the termination condition of the option is reached;

[0018] Loop through and return steps to get a set of options.

[0019] Furthermore, the step of learning the action model and updating the option set based on the change of the symbol state based on the action model learning module to obtain a new action model and planning target specifically includes:

[0020] Calculate its preconditions and effects based on the symbol-state pairs;

[0021] Determine the corresponding action model based on the calculated prerequisites and effects, and calculate the average value of the corresponding cumulative rewards;

[0022] Calculate the reward corresponding to the action model and obtain a new action model;

[0023] Update the option set and determine the new planning target based on the new action model, and pass the new action model and new planning target to the planner.

[0024] Furthermore, the step of solving the planning path based on the new motion model and the planning goal based on the planner module and exploring whether a new motion model exists based on the planning result specifically includes:

[0025] Solve the planning path based on the action model and planning goals;

[0026] Determine the option selected from the option set based on the planning results;

[0027] After determining that all planning options have been completed, select the random option to explore whether there is a new action model.

[0028] The second technical solution adopted by the present invention is: a task planning system based on symbol selection and action model self-learning, comprising:

[0029] A symbolic state mapping module, for mapping high-dimensional image data into symbolic states based on prior knowledge and digital image processing;

[0030] Option Set module, used to construct option sets based on random actions and symbolic states;

[0031] The action model learning module is used to learn the action model and update the option set according to the change of the symbol state to obtain the new action model and planning target;

[0032] The planner module is used to solve the planning path according to the new motion model and planning goal, and explore whether there is a new motion model based on the planning results.

[0033] The third technical solution adopted by the present invention is: a task planning device based on self-learning of symbol options and action models, comprising:

[0034] at least one processor;

[0035] at least one memory for storing at least one program;

[0036] When the at least one program is executed by the at least one processor, the at least one processor implements the task planning method based on self-learning of symbolic options and action models as described above.

[0037] The fourth technical solution adopted by the present invention is: a storage medium, which stores processor-executable instructions, characterized in that: the processor-executable instructions are used to implement the above-mentioned task planning method based on self-learning of symbolic options and action models when executed by the processor.

[0038] The method of the present invention has the following beneficial effects: it automatically learns action models and symbolic options from data obtained through reinforcement learning exploration. This automatically learned action model is more adaptable to the environment. The action model and planner generate a planning sequence, improving the interpretability of the algorithm and guiding the reinforcement learning exploration, enabling exploration while planning, thereby improving exploration efficiency. Furthermore, the portability of symbolic options allows multiple action models with the same effect to use the same symbolic options, avoiding repeated training and improving the data efficiency of the entire algorithm. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 It is a flowchart of the steps of a task planning method based on symbol selection and action model self-learning of the present invention;

[0040] Figure 2 It is a structural block diagram of a task planning system based on symbol selection and action model self-learning of the present invention;

[0041] Figure 3 is a schematic diagram of mapping from a high-dimensional state to a symbolic state according to a specific embodiment of the present invention;

[0042] Figure 4 It is a flowchart of discovering new options according to a specific embodiment of the present invention. DETAILED DESCRIPTION

[0043] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. The step numbers in the following embodiments are provided for ease of description only and do not limit the order of the steps. The order of execution of the steps in the embodiments can be adaptively adjusted based on the understanding of those skilled in the art.

[0044] Reference Figure 1 and Figure 2 The present invention provides a task planning method based on symbol selection and action model self-learning, the method comprising the following steps:

[0045] Based on the symbolic state mapping module, high-dimensional image data is mapped into symbolic states according to prior knowledge and digital image processing;

[0046] Specifically, the symbolic state mapping module is based on the digital image algorithm of entity detection to achieve the mapping from unstructured data to symbolic states composed of propositions. In the game of Montezuma's Revenge, the high-dimensional state (a picture) is composed of Indicates that, using human prior knowledge and digital image processing technology, the five positions of the right door (RightDoor), middle ladder (MiddleLadder), and key (Key) are identified, and then human prior knowledge is embedded to obtain the corresponding symbolic state composed of propositions, which is represented by symbol s. Figure 3As shown, the agent is located on the middle ladder through digital image technology, and the key also exists, so it can be mapped as follows Figure 3 The symbol state shown. The entire symbol state mapping module can be abstracted into a state mapping function At each time step, whether the option is completed can be determined by judging whether the symbolic state corresponding to the high-dimensional state meets the termination condition of the option.

[0047] Based on the option set module, an option set is constructed based on random actions and symbolic states;

[0048] Specifically, the option set module is a key underlying module, primarily consisting of multiple symbolic options and one random option. An option is represented by a triple o = (I, π, β), where I is the initial condition, π is the policy, and β is the termination condition. The option makes a decision based on the environment state and selects an action to perform. After the agent performs an action, the environment returns a new high-dimensional state and a reward to the option. The option then adjusts its policy based on the returned reward and state, continuing with the next interaction until the option's termination condition is reached.

[0049] The symbolic option is represented by a triple o = (pre,π,eff), where pre is the union of the preconditions, and + and pre - It is composed of, which represents the prerequisite for the execution of a symbolic option. Only when the high-dimensional state of formula (1) is True, the meta-controller can select the symbolic option. π represents an internal strategy, which is implemented by a single-layer reinforcement learning algorithm, and the input is the current high-dimensional state π will generate corresponding actions Strategy functions can be implemented through networks or tables eff by eff + and eff - This is also the effect of the symbol option. When the high-dimensional state satisfies formula (2) and is True, the option is executed successfully. After the option is executed, the high-dimensional state will be mapped to the abstract state through the abstract state mapping module to form an abstract state pair (s t ,s t+1 ), and record the cumulative rewards given by the environment and pass them to the top layer.

[0050]

[0051]

[0052] Random options are also represented by triples ro = (I, π, β), for all high-dimensional states It is always true, indicating that the random option can be executed in any state. Indicates that when a random option is taken, an action is randomly selected and executed each time. It means that the random option is executed until the abstract state corresponding to the high-dimensional state changes.

[0053] Step 1: The option set module selects the corresponding option o = (pre, π, eff) according to the meta-controller

[0054] Step 2: Get the initial high-dimensional state According to formula (1), the initial state condition of the current state option is determined. If it is not satisfied, the current option is replaced with a random option and restarted from step S1. If it is satisfied, continue to the next step

[0055] Step 3: Get the action performed by the agent in the current state according to the strategy π of the option

[0056] Step 4: The agent performs the action and obtains the next state of the environment feedback and reward value The experience Add option o to the experience pool and train its strategy.

[0057] Step 5: Determine the current high-dimensional state according to formula (2) Whether the termination condition of option o is met, if so, the symbol state is changed to And the cumulative reward value obtained during the entire interaction process Passed to the action model learning module.

[0058] Based on the action model learning module, the action model is learned and the option set is updated according to the change of the symbol state to obtain the new action model and planning target;

[0059] The action model set is represented by the symbol A. Each action model is represented by a triple a=(name, pre, eff), where name is the name of a, pre is the prerequisite of a, and eff is the effect of a. Both pre and eff are sets composed of propositions. Each symbol-state pair and action model have corresponding meanings. For a given symbol-state pair (s t , s t+1 ), the corresponding action model a, pre = s t , eff=s t+1 \s t ∪(increase qualityρ), where ρ is the reward corresponding to action model a ρ = r a +r e . r a is a symbol-state pair (s t , s t+1 )The average value of the cumulative rewards obtained r e For exploration rewards, r e =(1-sr)c, where sr is the success rate of the action model execution and c is a fixed constant. The planning objective is a linear function in the form of (quality>G), where G is the maximum reward obtained during the entire interaction process.

[0060] Symbol options and action models do not correspond one-to-one. Multiple action models with the same effect may correspond to the same symbol option. In other words, if a new action model is found and its corresponding symbol option already exists, the prerequisites for the symbol option are the union of the prerequisites of the original symbol option and the new action model. If it corresponds to a new symbol option, its prerequisites and effects are the same as those of the action model, and the new symbol option is added to the option set.

[0061] Step 1: According to the symbol state pair (s t , s t+1 ) calculates its prerequisite pre and effect eff.

[0062] Step 2: Determine the corresponding action model based on the calculated prerequisites and effects, and then calculate the average value of the corresponding cumulative rewards.

[0063] Step 3: Calculate the reward corresponding to the action model to obtain the complete action model, update the option set based on the new action model, determine the new planning target, and pass the action model and planning target to the planner.

[0064] Based on the planner module, the planning path is solved according to the new motion model and planning goal, and whether there is a new motion model is explored based on the planning results.

[0065] Specifically, the planner solves the planning path based on the motion model and planning goal given by the motion model learning module, and then determines the options selected from the option set based on the planning results. If all planning options have been completed, a random option is selected to explore whether there is a new motion model.

[0066] Figure 4 The figure shows how new options are discovered. After the planner creates a sequence of plans, it executes the options corresponding to each planned action in sequence. After executing the options corresponding to the sequence, if the environment has not yet ended, a random option is selected for execution. During the interaction between the random options and the environment, a high-dimensional state is input, and the random option determines the corresponding action. After the agent takes an action, a new high-dimensional state is obtained. Each high-dimensional state is mapped to a symbolic state. If a new symbol-state pair is obtained, a new action model is obtained after learning the action model.

[0067] Given a high-dimensional environment, an agent can obtain the full state of the environment. Human experts abstract the environment and express it in propositions. A given state mapping function F maps the high-dimensional state to a symbolic state.

[0068] Hierarchical reinforcement learning based on symbolic options, the specific implementation steps are as follows:

[0069] Step S1. The planner performs planning based on the action model set and planning goals to obtain a planned action model sequence Π.

[0070] Step S2. The meta-controller selects the option corresponding to the action model according to the planned action model sequence Π. If every action model in Π has been executed successfully, a random option is selected.

[0071] Step S3: In the option set module, the selected option is trained. If it is a random option, random exploration is performed. After training or exploration, the symbol-state pair and the accumulated reward are passed to the action model learning module.

[0072] Step S4. The action model learning module learns the action model according to the symbol-state pairs and cumulative rewards transmitted by the option set module.

[0073] Step S5: Update the option set according to the new action model set and determine a new planning target.

[0074] Step S6. Repeat steps S1, S2, S3, S4, and S5 until the reward obtained by the agent converges.

[0075] like Figure 2 As shown, a task planning system based on self-learning of symbolic options and action models includes:

[0076] A symbolic state mapping module, for mapping high-dimensional image data into symbolic states based on prior knowledge and digital image processing;

[0077] Option Set module, used to construct option sets based on random actions and symbolic states;

[0078] The action model learning module is used to learn the action model and update the option set according to the change of the symbol state to obtain the new action model and planning target;

[0079] The planner module is used to solve the planning path according to the new motion model and planning goal, and explore whether there is a new motion model based on the planning results.

[0080] The contents of the above method embodiments are all applicable to the present system embodiments. The functions specifically implemented by the present system embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0081] A task planning device based on self-learning of symbolic options and action models:

[0082] at least one processor;

[0083] at least one memory for storing at least one program;

[0084] When the at least one program is executed by the at least one processor, the at least one processor implements the task planning method based on self-learning of symbolic options and action models as described above.

[0085] The contents of the above method embodiments are all applicable to the present device embodiments. The functions specifically implemented by the present device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0086] A storage medium storing processor-executable instructions, characterized in that the processor-executable instructions, when executed by the processor, are used to implement the above-mentioned task planning method based on self-learning of symbolic options and action models.

[0087] The contents of the above method embodiments are all applicable to the present storage medium embodiment. The functions specifically implemented by the present storage medium embodiment are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0088] The above is a specific description of the preferred implementation of the present invention, but the invention is not limited to the embodiments. Those skilled in the art can make various equivalent modifications or substitutions without violating the spirit of the present invention. These equivalent modifications or substitutions are all included in the scope defined by the claims of this application.

Claims

1. A task planning method based on symbolic options and action model self-learning, characterized in that: The following steps are involved: Based on the symbolic state mapping module, high-dimensional image data is mapped into symbolic states according to prior knowledge and digital image processing; Based on the option set module, an option set is constructed based on random actions and symbolic states; Based on the action model learning module, the action model is learned and the option set is updated according to the change of the symbol state to obtain the new action model and planning target; Based on the planner module, the planning path is solved according to the new motion model and planning goal, and the existence of new motion models is explored based on the planning results; The step of learning the action model and updating the option set based on the change of the symbol state based on the action model learning module to obtain the new action model and planning target specifically includes: Calculate its preconditions and effects based on the symbol-state pairs; Determine the corresponding action model based on the calculated prerequisites and effects, and calculate the average value of the corresponding cumulative rewards; Calculate the reward corresponding to the action model and obtain a new action model; Update the option set and determine the new planning target based on the new action model, and pass the new action model and new planning target to the planner; The planner module solves the planning path according to the new motion model and the planning goal, and explores whether there is a new motion model based on the planning result. Specifically, the step includes: Solve the planning path based on the action model and planning goals; Determine the option selected from the option set based on the planning results; After determining that all planning options have been completed, select the random option to explore whether there is a new action model.

2. The task planning method based on symbol selection and action model self-learning according to claim 1, characterized in that: The step of mapping the high-dimensional image data into symbol states based on the symbol state mapping module according to prior knowledge and digital image processing specifically includes: Determine the intelligent agent based on prior knowledge and digital image processing, locate the position of the intelligent agent, and obtain the symbolic proposition of the position of the intelligent agent; According to the symbolic proposition of the agent's position, a high-level state is mapped to a symbolic state consisting of two propositions.

3. The task planning method based on symbol selection and action model self-learning according to claim 2, characterized in that: The option set consists of multiple symbol state options and a random option.

4. The task planning method based on symbol selection and action model self-learning according to claim 3, characterized in that: The symbolic state option and the random option are both represented by triples. The symbolic state option includes a precondition, an internal strategy, and an effect, and the random option includes an initial condition, a strategy, and a termination condition.

5. The task planning method based on symbol selection and action model self-learning according to claim 4, characterized in that: The step of constructing an option set based on the option set module according to the random action and the symbol state specifically includes: Options make decisions based on the state of the environment and choose the actions to perform; After the agent performs an action, the environment returns a new high-dimensional state and reward to the option; The option adjusts the strategy based on the returned reward and state, and continues to the next interaction until the termination condition of the option is reached; Loop through and return steps to get a set of options.

Images