Method for planning actions based on basic attribute space

Abstract

To overcome limitations of narrow and specialized domains of available robotic assistance systems.SOLUTION: Method for operating an assistance system based on an attribute space comprises: obtaining task information defining at least one target state to be achieved; obtaining initial state information defining an initial state of an autonomous device or user and of at least one object involved in achieving the target state; defining a planning problem based on the task information, the initial state information and attribute information related to the object; selecting an action or a sequence of actions to solve the planning problem; and outputting a control signal to a human-machine interface HMI or the autonomous device for achieving the target state; determining whether the action or sequence of actions achieves the target state, and, if it is not achieved, decomposing the planning problem based on the attribute information to generate a decomposed planning problem; solving the decomposed planning problem; and outputting the control signal to the HMI or the autonomous device.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application claims the benefit of priority to U.S. Application No. 63 / 449,056, filed March 1, 2023. The entirety of the aforementioned patent application is incorporated herein by reference and made a part hereof.

[0002] The present disclosure relates to the general fields of robotics and assistance systems. In particular, a method and corresponding computer program are proposed for improving the action planning process for operating assistance systems. [Background technology]

[0003] The environment of an assistance system or robotic system can be represented by a set of attributes. The term attribute refers to a variable that can take on different values. Assigning an actual value to each attribute defines a specific state of the environment. In general, a task for a robotic system can be formulated to reach a given goal by starting from an initial state and performing an action or series of actions to reach a target state of the environment. Correspondingly, a task for an assistance system can be formulated to guide the robotic system, or a human user via a human-machine interface, to realize a given target state of the environment, starting from an initial state. The assistance system commands a set of actions that define what needs to be done to change from one state (initial state) to one or several next states (target state(s)). The domain knowledge of an assistance system includes the attributes and actions available to the domain of the assistance system.

[0004] In the planning process, the assistance system selects actions, possibly combining them in a certain order, to reach a desired goal state according to the problem description. The planning problem defines the problem description and provides the input to the planning process.

[0005] The assistance system uses attributes to represent the state of the environment. For example, in an assistance system operating a kitchen robot, an example of an autonomous device, the kitchen robot may have behaviors or skills to navigate the kitchen environment and reach specific locations. The kitchen robot may have skills to move its arms and end effectors, e.g., hands, located at the ends of its arms, and to interact with objects, e.g., bread, and devices, e.g., knives and toasters, in the kitchen environment. Thus, the assistance system operating the kitchen robot has detailed attributes that describe what devices are in the kitchen, where each device is currently located, which parts of each device—e.g., which switches, handles, and levers of each device are operable, and which openings or doors provide access to the interior of a device, e.g., a cupboard. This allows the assistance system to use a planning module to find a plan for the kitchen robot on how to reach and operate each device, which is an assembly of modeled parts or skills. This is true even if the robot has not previously experienced some of the devices in an action.

[0006] Nevertheless, using attributes as abstractions results in a loss of detail.

[0007] For example, many aspects are typically simplified and reduced in attribute space. A kitchen robot might be given attributes that represent the application of a device to a particular object, such as "was_in_toaster bread_1," "was_in_egg_cooker egg_1," or "was_in_pan meat_3." Attributes might also define the overall effect of an action, such as "toasted bread_1," "cooked egg_2," or "cooked meat_3." Such attributes are composite attributes, which provide a high-level abstraction of which of the more fundamental physical properties of an object changed. In the example kitchen environment, the composite attribute "toasted bread" represents an abstraction corresponding to the combination of the basic attributes "hot, brown, crispy." Using composite attributes, an assistance system can efficiently plan how to prepare a recipe simply by following a learned sequence of known types of devices.

[0008] However, the use of composite attributes in particular can result in limitations.

[0009] State-of-the-art planning problems are typically defined in narrow, problem-specific domains, such as a kitchen robot operating with food and kitchen appliances in a kitchen environment. Expert users define state and action spaces explicitly designed for a particular problem. Thus, planning problems are applicable to narrow, specific domains, where current planners achieve superior results. Nevertheless, the narrow approaches currently pursued reach their limits when considering broader problem-solving capabilities beyond the predefined problem domain.

[0010] The focus of existing assistance systems and planning modules emphasizes constrained services in small, closed, and well-defined domains. However, extending the capabilities of assistance systems to handle the entire kitchen environment—the entire home, or even the world, as a single domain—would provide an expanded set of services and applications. Furthermore, planning problems, or problems during plan execution that arise from the unavailability of a required device, such as a missing toaster, would also be problematic with the current specific and limited domain definition. Summary of the Invention [Problem to be solved by the invention]

[0011] It is desirable to overcome the narrow, specialized domain limitations of available robotic assistance systems. [Means for solving the problem]

[0012] A method for operating an assistance system based on an attribute space according to a first aspect and a computer program according to a second aspect address this problem.

[0013] The dependent claims define further advantageous embodiments.

[0014] A method for operating an assistance system based on an attribute space according to a first aspect includes acquiring task information defining at least one goal state to be achieved by the assistance system, and acquiring initial state information defining an initial state of at least one of an autonomous device or an assistanceee and at least one object involved in achieving the goal state, or one of the goal states if multiple goal states are defined. Hereinafter, to improve the readability of this disclosure, it should be noted that the term "goal state" will always refer to a single goal state if only one goal state is defined, or one of the multiple goal states if multiple goal states are defined. The method then defines a planning problem based on the acquired task information, the initial state information, and further based on attribute information related to at least one object stored in a database. A planning module then selects an action or set of actions to solve the generated planning problem. The method then generates a control signal and outputs the control signal to at least one of a human-machine interface to assist a user in achieving the goal state or to an autonomous device to achieve the goal state. The method further determines whether the selected action or set of actions achieves the goal state. If it is determined that the selected action or series of actions will not achieve the goal state, the method continues by decomposing the generated planning problem based on the attribute information stored in the database to generate decomposed planning problems and providing the decomposed planning problems to a planning module. The planning module solves the decomposed planning problems. The method then generates control signals based on the solved decomposed planning problems and outputs the control signals to at least one of a human-machine interface to assist a user in achieving the at least one goal state or to an autonomous device to achieve the at least one goal state.

[0015] A computer program product according to a second aspect includes a non-transitory computer-readable medium having a computer-readable program embedded therein, which, when executed on a computing device, causes the computing device to perform a method for operating an assistance system based on an attribute space. The method includes acquiring task information defining at least one goal state to be achieved by the assistance system and acquiring initial state information defining an initial state of at least one of an autonomous device or an assisted person and at least one object involved in achieving the goal state (one of the goal states if multiple goal states are defined). The method includes defining a planning problem based on the acquired task information, initial state information, and further based on attribute information related to the at least one object stored in a database, and selecting, by a planning module, an action or set of actions for solving the generated planning problem. The method includes generating a control signal and outputting the control signal to at least one of a human-machine interface (HMI) to assist a user in achieving the goal state or to an autonomous device to achieve the goal state. The method further includes determining whether the selected action or series of actions achieves a goal state (one of the goal states), and if it is determined that the selected action or series of actions does not achieve the respective goal state, decomposing the generated planning problem based on the attribute information stored in the database to generate decomposed planning problems and providing the decomposed planning problems to a planning module. The method then includes solving the decomposed planning problems, generating control signals based on the solved decomposed planning problems, and outputting the control signals to at least one of a human-machine interface to assist a user in achieving the goal state or to an autonomous device to achieve the goal state.

[0016] The description of the embodiments will refer to the accompanying drawings, in which: [Brief explanation of the drawings]

[0017] [Figure 1]FIG. 2 is a schematic diagram of a general planning loop of a method according to an embodiment. [Figure 2] 1 is a flowchart illustrating selected steps of decomposing a planning problem, according to one embodiment. [Figure 3] 10 is a diagram of an exemplary data structure of attribute information for mapping a composite attribute in a composite attribute space to a set of base attributes in a base attribute space. [Figure 4] 1 is a diagram of an example data structure of attribute information representing the state of an object for mapping a composite attribute in a composite attribute space to a set of base attributes in a base attribute space. [Figure 5] 1 is a diagram of an exemplary data structure of attribute information showing the correspondence between objects, means for realizing effects, and effects on objects. [Figure 6] FIG. 10 is a diagram of an exemplary data structure of attribute information showing the correspondence between means for achieving an effect, objects, and expressions of the means in a basic attribute space. [Figure 7] Specific application scenario of a method for operating an assistance system including autonomous devices in a kitchen environment. DETAILED DESCRIPTION OF THE INVENTION

[0018] In the figures, corresponding elements in different figures have the same reference numerals. In the description of the figures, where possible, the description of the same reference numerals in different figures will be omitted without negatively affecting the understandability.

[0019] The proposed iterative planning system first attempts to solve the original planning problem, which typically involves complex attributes. If the planner is unable to solve the original planning problem, the proposed method innovatively exploits the potential similarities of objects, devices, and effects in the assistance system's environment.

[0020] Solving the original planning problem may fail in the absence of a preferred device. Assuming this is the case, the planning module decomposes the planning problem into decomposed or basic planning problems in a more general physical attribute space, called the decomposed attribute space or sometimes the base attribute space. In the decomposed attribute space, instead of representing actions and effects in a somewhat abstract composite attribute space for the task predicted in each specific domain, the decomposition process decomposes the composite attributes into a single, more universal set of lower-level, grounded attributes that capture the underlying physical processes in more detail than the composite attributes can. The shared underlying representation in the base attribute space facilitates transition between different tasks and different environments by treating the world as a single domain. The decomposition step expands the solution space for a given problem, which improves the likelihood of solving the problem.

[0021] Some basic attributes, unlike composite attributes, may be measurable by the assistance system, so that the assistance system may infer new behaviors based on observations of the task environment and even explore the task environment independently.

[0022] Regarding the example of an assistance system in a kitchen environment, an assistance system that originally uses composite attributes can plan the preparation of a given recipe simply by following a learned sequence of applying known types of devices. In general, the use of composite attributes creates limitations that are overcome by the proposed method for operating an assistance system that includes an attribute decomposition step.

[0023] Known assistance systems cannot plan to replace commonly used devices because they have no information about how different composite attributes relate to each other. This may be related to composite attributes such as "was_in_toaster" and "was_in_egg_cooker" because known assistance systems have no theoretical basis for how "toasted" relates to "cooked". Basic attributes capture relationships between different actions and states that composite attributes do not describe due to their higher level of abstraction compared to basic attributes.

[0024] Complex attributes can have different meanings, each of which is relevant in different situations or conditions, which can lead to incorrect plans for solving planning problems defined in complex attribute spaces. This is the case for example with "being cooked", which can have different meanings for different objects, e.g., "egg" and "meat", "egg cooker" and "pot", etc.

[0025] Assistance systems cannot measure composite attributes. This prevents them from accessing the current state of the environment. The current state of the environment, commonly referred to as the initial state, needs to be known before planning and possibly also during the execution of a found plan to verify the success of the action. In the example of an assistance system for a kitchen environment, it is useful to know if the eggs are already cooked, but this requires a measurable notion of what it means for an egg to be cooked. In a further example, the assistance system needs to determine if the toaster is not working properly, which requires a measurable notion of what it means to be toasted.

[0026] Decomposing composite attributes into basic attributes overcomes current limitations in the level of detail at which users can express their preferences and feedback, such as a preference for more burnt or less burnt toast in the composite attribute space compared to a preference for crispier but less hot toast in the basic attribute space.

[0027] In a method for operating an assistance system according to one embodiment, the stored attribute information includes composite attributes and basic attributes that define at least a state of at least one object, an initial state of a planning problem, and an effect that a device has on the at least one object under at least one predetermined condition.

[0028] According to one embodiment of the method for operating the assistance system, decomposing the generated planning problem includes mapping at least one composite attribute included in the generated planning problem to a set of basic attributes based on the stored attribute information.

[0029] The stored attribute information, in one embodiment of the method, includes a correspondence table that stores composite attributes associated with a set of base attributes to perform this mapping.

[0030] The correspondence table may be a pre-stored look-up table.

[0031] According to one embodiment of the method for operating the assistance system, the mapping is performed by mapping at least one goal state g, an initial state s in the composite attribute space. i , and at least one of the planning domains Σ is divided into at least one respective basic goal state g′, basic initial state s i ', and includes mapping it to the master planning domain Σ'.

[0032] A method for operating an assistance system according to one embodiment includes predicting a new mapping of a composite attribute to a set of base attributes based on a similarity between the new mapping and a predetermined mapping in terms of a similarity between at least one object and another object.

[0033] In one embodiment, the prediction includes predicting that the at least one object will have a similar effect as the other object if the at least one object and the other object are exposed to the same means.

[0034] The term "means" refers to a physical process: a device exposes an object to a physical process that ultimately produces a particular effect on the object.

[0035] A method for operating an assistance system according to one embodiment includes predicting a new mapping of a composite attribute to a set of base attributes based on similarity, where predicting the new mapping includes generalizing a prediction that a given object will have an effect on an object category, and predicting the new mapping includes predicting that another given object of the same object category will expose the object to the same measures and provide the same effect as the given object.

[0036] According to one embodiment of the method for operating an assistance system, predicting a new mapping of composite attributes to a set of basic attributes is based on at least one of predefined relationship information provided during a design phase of the assistance system, relationship information generated by experts, and relationship information generated online during an interaction process with a user of the assistance system.

[0037] A method for operating an assistance system according to any one of the preceding claims may comprise iteratively decomposing the generated planning problem based on the stored attribute information to generate decomposed planning problems until predetermined criteria are met.

[0038] Iterative decomposition means that the method repeatedly performs the steps of decomposing the generated planning problem until a predetermined criterion is met.

[0039] According to one embodiment of a method for operating an assistance system, the method determines that a predetermined criterion is met if a selected action or sequence of actions that solves the decomposed planning problem achieves at least one goal state.

[0040] The method may determine that the predetermined criteria is met when no further mapping of the composite attribute to a set of corresponding basic attributes is possible based on the stored attribute information.

[0041] According to one embodiment of the method, the method is performed online during a planning process to achieve at least one target state.

[0042] The assistance system may be included in a robotic assistance system that includes an autonomous device.

[0043] The assistance system may be implemented in a controller of the autonomous device.

[0044] The assistance system may be implemented as software running on a remote server connected to the autonomous device over a network.

[0045] According to one embodiment of a method for operating an assistance system, the assistance system guides a user in achieving a goal state via a human-machine interface. The human-machine interface may include a virtual reality device configured to visualize a real-world task environment for the user. Additionally or alternatively, the human-machine interface may include an augmented reality device configured to present information for guiding the user as an overlay on the real-world task environment for and perceived by the user. Additionally or alternatively, the human-machine interface may include a virtual reality device configured to visualize a generated virtual environment for the user.

[0046] According to a second aspect, a computer program comprising program code means is configured to cause a computer or signal processor to perform a method according to any of the embodiments of the method for operating an assistance system when the program is run on the computer or signal processor. The proposed method and system relate to planning tools, in particular in the field of operating autonomous devices, for example in the field of robotics, for planning actions and helping users to cope with given tasks and achieve set goals. The terminology used in the description of the following embodiments comes from these application fields and in particular includes:

[0047] The term attribute means a named variable that can take on different values.

[0048] The values ​​may be categorical, for example Boolean, or continuous.

[0049] A composite attribute is a single attribute that is a placeholder or abstraction for a set of more specific attributes. A composite attribute may, for example, represent a property of an object.

[0050] Basic attributes are more physical than composite attributes. By themselves, they represent concrete, often measurable, properties of an object. Because it is not possible to decompose a basic attribute into shared parts, basic attributes usually exhibit strong independence from other basic attributes.

[0051] An attribute space refers to the set of attributes that a system uses to model its environment.

[0052] A composite attribute space is an attribute space that contains several composite attributes or at least one composite attribute.

[0053] A basic attribute space is an attribute space that, at least ideally, does not contain any composite attributes.

[0054] A state is the actual assignment of values ​​to each attribute in the attribute space.

[0055] The initial state is the current state of the environment as observed or modeled by the system.

[0056] The term goal refers to the definition of one or more states that the system should achieve. Goals are usually incomplete, for example, required values ​​are given for only some attributes. Goals can be inferred from the description given by the user. In the description, the term target state is used interchangeably with the term goal.

[0057] Behavior refers to a rule that describes under what conditions a state changes or transitions to a next state (post-change state).

[0058] An effect is a change in the value of an attribute that an action causes.

[0059] A plan is an ordered sequence of actions that can transition an initial state to a state that satisfies a goal or at least one target state. A planner or planning module may find a plan by searching, e.g., by iteratively trying actions. A planning module is a structural element, e.g., a combination of software and at least one data processing hardware device, that performs planning by applying a planning algorithm.

[0060] A domain comprises a set of behaviors and an attribute space that the system uses to represent and solve a particular planning problem.

[0061] A device is a specific structural object or tool designed for a particular purpose. A device may be used on an object to change some attribute of that object. Thus, applying a device to an object has an effect on the object.

[0062] The term means refers to the physical process to which a device exposes an object, and exposing the object to the means ultimately produces an effect on the object.

[0063] 1 is a schematic diagram of a general planning loop of a method according to one embodiment. The description of this embodiment uses the example of an assistance system 1 in a kitchen environment performing a specific task such as preparing food. One example involves the assistance system 1 controlling an autonomous device 2, a kitchen robot, in the kitchen environment to prepare toasted bread.

[0064] The description of an embodiment of a method for operating the assistance system 1 will, for illustrative purposes, refer to the robotic assistance system 1 in a kitchen environment to illustrate specific aspects of a possible implementation. The described aspects may be transferred to the robotic assistance system 1 in a different task environment, such as a manufacturing environment, a logistics environment, or a vehicle workshop. The assistance system 1 may operate in a real-world environment, such as the described kitchen environment, or may assist a user in achieving goals in a virtual environment, such as a training environment or even a gaming environment for entertainment purposes.

[0065] 1 begins at step S1, which involves obtaining a task from a user. The obtained task may have the form of an initial request or goal formulated by the user, which may include composite attributes, for example, the goal "I want toasted bread" verbally provided by the user and which includes the composite attribute "toasted bread."

[0066] The obtained goal defines at least one target state that the user requests the assistance system 1 to achieve in the environment.

[0067] In step S2, the assistance system 1 interprets the obtained goal using information from the knowledge base available to the assistance system 1. The assistance system 1 interprets the obtained goal as "made toast" being the result of the action "use a toaster" because the user may have previously instructed the assistance system 1 to do so.

[0068] Step S2 includes acquiring initial state information that defines an initial state of at least one of the autonomous device 2 or the assisted person and at least one object involved in achieving the at least one goal state. The initial state may include a current location of the kitchen robot, a location of bread in the kitchen as an example of an object, and a current location and operation status of a toaster as an example of a device.

[0069] The knowledge base may have the form of a database stored locally, for example in the memory of the autonomous device 2. Alternatively or additionally, the database may be stored in at least one server located locally in the kitchen or at a remote location and connected to the assistance system 1 via a network N.

[0070] In step S2, the assistance system 1 defines a planning problem based on the acquired task information, initial state information, and attribute information related to at least one object stored in the database. In detail, the database may provide information on actions and initial states to define the planning problem as an initial planning problem in step S2.

[0071] In step S3, the assistance system 1 provides the generated initial planning problem to a planning module of the assistance system 1 in order to solve the initial planning problem. The assistance system 1 finds a plan that represents a solution in the composite attribute space only under specific, and therefore narrow, conditions, for example, when a preferred device for preparing toasted bread, i.e., a toaster, is present and ready to use in the kitchen.

[0072] The planning module, in step S3, selects an action, or more likely a sequence of actions, for solving the generated initial planning problem formulated in the composite attribute space. The planning module may arrange the selected actions in a specific sequence to transition from the initial state, possibly via intermediate states, to the goal state / one of the defined goal states. The planning module, in step S3, outputs a plan for solving the initial planning problem.

[0073] Step S3 is followed by step S4, in which the assistance system 1 determines whether the planned action or series of actions will achieve the goal state.

[0074] If the assistance system 1 determines in step S4 that it can find a plan to address the initial planning problem, in particular that the plan is determined to be feasible and to achieve the target state, the assistance system 1 may then, in step S7, execute the plan, in particular the action or series of actions contained in the plan.

[0075] If it is determined that the selected action or sequence of actions according to the plan provided by the planning module does not achieve (one of) the goal states, or even if the planning module fails to find a plan for solving the obtained initial planning problem, the method proceeds from step S4 to step S5.

[0076] In step S5, the assistance system 1 then, in a planning loop, decomposes some or all of the compound attributes of the initial problem into elementary attributes using the stored knowledge in the knowledge base, as will be explained in more detail with reference to FIG. 3.

[0077] In particular, the database provides attribute information (decomposition information) that enables the assistance system 1 to decompose at least one composite attribute into a set of decomposed attributes, preferably into a set of basic attributes.

[0078] Step S5 includes decomposing the generated planning problem, here the initial planning problem, based on the attribute information stored in the database to generate decomposed planning problems, and providing the decomposed planning problems to the planning module. The decomposed planning problems may be formulated in a basic attribute space if the decomposed planning problem does not include any composite attributes.

[0079] Alternatively, the decomposed planning problem, which still contains at least one composite attribute, is represented in the basic attribute space, but at least one composite attribute of the planning problem input to the decomposition step S4 has been replaced by a set of basic attributes.

[0080] Step S4 may generate a modified planning problem (new planning problem) with a larger solution space, which the assistance system 1 provides to the planning module for finding a solution to the modified planning problem in the decomposed attribute space or the basic attribute space, respectively. Preferably, the planning module is the same planning module used in step S3 to solve the initial planning problem in the composite attribute space.

[0081] The assistance system 1 may implement the planning module using a standard planner commonly known in the field of planning systems.

[0082] If the assistance system 1 determines in step S6 following step S5 that the decomposition in step S5 was unable to decompose the planning problem input to the decomposition step S5, the process ends (NO). In this case, the assistance system 1 failed to devise a workable solution to the planning problem and the obtained goal.

[0083] If the assistance system 1 determines in step S6 that the decomposition step S5 has succeeded in decomposing the planning problem in the composite attribute space input to the decomposition step S5, the process provides the decomposed planning problem in the basic attribute space or the decomposed composite attribute space to the planning module. Thus, the process resumes from step S3, which involves the planning module attempting to find a plan that solves the decomposed planning problem instead of solving the initial planning problem.

[0084] The assistance system may repeat the processing loop of steps S3-S4-S5-S6 in an iterative planning process until it determines that predetermined criteria are met.

[0085] The iterative planning process may terminate when the planner finds a plan in steps S4 and S7 that proves feasible, or when no further decomposition of the planning problem input to decomposition step S5 is possible.

[0086] If it is determined in step S4 that the initial planning problem or the decomposed planning problem can be solved by the plan devised by the planning module in step S3, the assistance system 1 executes the plan in step S7. Step S7 may specifically include generating control signals based on the plan, and therefore based on the solved decomposed planning problem, and outputting them to the human-machine interface of the assistance system 1 to assist the user in achieving at least one target state or goal provided to the assistance system 1 in step S1. The assistance system 1 may guide the user (the person being assisted) in achieving the at least one target state via perceptible signals output by the human-machine interface.

[0087] Alternatively or additionally, step S7 may include generating control signals based on the solved decomposed planning problem and outputting them to the autonomous device 2, e.g., a kitchen robot, to realize the target state or goal obtained in step S1 by the assistance system 1. The autonomous device 2 may autonomously take action based on the control signals to realize the target state.

[0088] Alternatively or additionally, step S7 may include generating and outputting control signals to the autonomous device 2 based on the solved decomposed planning problem in order to realize the target state by the autonomous device 2 in cooperation with the user under the guidance of the assistance system 1.

[0089] A typical planning loop, as described with reference to FIG. 1, involves generating an initial planning problem from a given goal and based on the current state of the task environment and available knowledge about actions and devices with which the assistance system 1 already has experience. Typically, the initial planning problem uses a high level of abstraction, which provides the advantage of efficient planning and plan execution processes. Thus, the initial planning problem is necessarily located in a more abstract, complex attribute space. If the assistance system 1 fails to find a plan, it attempts to decompose the original planning problem representation into more basic planning problem representations (decomposed planning problems) and subsequently solve the decomposed planning problems as new planning problems.

[0090] The decomposition process summarized in step S7 may be an iterative decomposition process, for example, decomposing the planning problem may involve several levels of decomposition of the planning problem, starting from an initial planning problem and building sequentially on each other.

[0091] FIG. 2 provides a flowchart illustrating selected steps for decomposing a planning problem according to one embodiment.

[0092] The assistance system 1 has a set of goal or target states g and a set of initial states s i , and a composite attribute space P = (Σ,s i ,g), the processing step S4 is performed on the planning problem. At least one or more of the attributes included in the planning problem are composite attributes. Therefore, the planning problem is expressed as a composite attribute space P=(Σ,s i ,g) will be formulated.

[0093] The decomposition step S4 may include successfully executing at least one of steps S41, S42, and S43 shown in FIG. 2 to successfully decompose the input planning problem P and generate and output a decomposed planning problem P′. In the specific example of FIG. 2, the decomposed planning problem P′ is a decomposition of the basic planning problem P′=(Σ′,s′ i ,g'), which is a composite attribute Σ,s i,g does not include it, but instead includes a set of basic goals (basic goal states) g' and a set of basic initial states s'. i , and contains only basic attributes including the basic planning domain Σ′.

[0094] The step S4 of decomposing the planning problem in a composite attribute space P based on attribute information stored in the database to generate a decomposed planning problem P′ may include at least one of sub-steps S41, S42, and S43.

[0095] Steps S41, S42, and S43 may each be performed during one cycle of decomposition step S4. Alternatively, steps S41, S42, and S43, or a combination of these steps, may be performed individually during one cycle of decomposition step S4, each during an iterative approach that implements a planning loop in one embodiment.

[0096] Step S41 involves determining a base goal state g' and replacing the composite goal state g with the corresponding base goal state g'.

[0097] Step S42 is to set the basic initial state s' i Determine the composite initial state s i The corresponding basic initial state s' i This includes replacing

[0098] Step S43 determines the basic planning domain Σ' and converts the composite planning domain Σ into the corresponding basic planning domain Σ' i This includes replacing

[0099] 2 may be interpreted as showing a general process flow for inferring and solving a planning problem in a basic attribute space P', starting from an initial planning problem in a composite attribute space P. The support system 1 infers and solves a planning problem in the basic attribute space P' by inferring a basic goal g', a basic initial state s', i , and infer a basic planning domain Σ′, which essentially contains basic actions.

[0100] Basic goal g', basic initial state s' i , and inferring the basic planning domain Σ' is done by using the goal g in the composite attribute space P, the initial state s i , and from the definition of the planning domain Σ, the corresponding goal g' in the basic attribute space P', the initial state s i ', and a mapping to the planning domain Σ'.

[0101] In detail, the support system 1 determines a goal g in a composite attribute space P, an initial state s i , and from the definition of the planning domain Σ, the corresponding goal g' in the basic attribute space P', the initial state s i ', and the mapping to the planning domain Σ' may be performed based on attribute information (decomposition information) obtained from a database. The decomposition information may include one correspondence table (lookup table) or multiple correspondence tables. The data structures related to the decomposition information and used for mapping in the decomposition step S5 are the subject of a more detailed discussion with reference to Figures 3, 4, 5 and 6.

[0102] Basic attribute space P'=(Σ', s' i , g') form new inputs to the planning module in order to solve the decomposed planning problem in the basic attribute space P', in particular to find a plan that provides a feasible solution to the obtained task.

[0103] Figure 3 provides an exemplary data structure of attribute information for mapping composite attributes in the composite attribute space P to a set of basic attributes in the basic attribute space P'. Figure 3 shows the composite attribute space P and the basic attribute space P' for a specific application area, the kitchen environment.

[0104] The mapping of composite attributes to a set of basic attributes in the decomposition process of step S5 may include mapping at least one composite attribute to a corresponding set of basic attributes, which may include mapping each composite attribute included in the planning problem representation input to the decomposition step S5 to a set of basic attributes.

[0105] Figure 3 shows a group of composite attributes in the composite attribute space P at the top of Figure 3. The composite attributes include "fresh bread" and "toasted bread".

[0106] The bottom part of Figure 3 lists groups of basic attributes in the basic attribute space P'. The basic attributes shown in Figure 3 include instances "cold" and "hot" for the basic attribute "temperature". Further basic attributes include instances "hard" and "soft" for the basic attribute "hardness". Further basic attributes include instances "white" and "brown" for the basic attribute "color". Further basic attributes include instances "moist" and "dry" for the basic attribute "moisture". Further basic attributes include instances "light color" and "dark color" for the basic attribute "color brightness".

[0107] 3 shows the attribute information in the form of correspondence relationships shown as arrows. The composite attribute "toasted bread" is mapped to a set of basic attributes ["hot", "hard", "brown", "dark"]. Using the attribute information, the decomposition process of step S5 may replace the composite feature "toasted bread" with a set of basic attributes for the object "bread", including the basic attributes "hot", "hard", "brown", and "dark", based on the attribute information contained in FIG. 3.

[0108] The mapping performed based on the attribute information shown in FIG. 3 infers at least one goal state g' in the basic attribute space P' in step S41 of FIG. 2, and at least one initial state s' in the basic attribute space P' in step S42 of FIG. 2. i may be used to infer

[0109] The decomposition in step S5 may involve the assistance system 1 internally referencing the correspondence table storing the attribute information in FIG. 3 and mapping the composite attribute to a set of basic attributes based on the stored correspondence table. The initial planning problem generated based on the acquired task defining that the user desires toasted bread includes the composite attribute “toasted bread” in the composite attribute space P as part of the goal state. In the decomposition step S5, the assistance system 1 may refer to the attribute information to determine a set of basic attributes corresponding to the composite attribute of “toasted bread.” According to the attribute information shown in FIG. 3, the corresponding set of basic attributes is [“hot,” “hard,” “brown,” “dark”]. The assistance system 1 can then attempt to solve this new decomposed planning problem using the set of basic attributes [“hot,” “hard,” “brown,” “dark”] in the decomposed planning problem in the basic attribute space P′.

[0110] Furthermore, in the decomposition step S5, the assistance system 1 may refer to the attribute information to determine a set of basic attributes corresponding to the composite attribute of "fresh bread." According to the attribute information shown in FIG. 3, the corresponding set of basic attributes is ["cold," "soft," "white," "light-colored"]. The assistance system 1 uses this set of basic attributes ["cold," "soft," "white," "light-colored"] in the decomposed planning problem as a new decomposed initial state s i ' can be used as

[0111] FIG. 4 displays a second exemplary data structure of attribute information representing the state of an object for mapping composite attributes in composite attribute space P to a set of basic attributes in basic attribute space P′.

[0112] In detail, Figure 4 displays the attributes that define the state of an object before and after using a device in the composite attribute space P, and the corresponding representation in terms of a set of basic attributes in the basic attribute space P'. The attributes shown in Figure 4 define the state of the object "bread" before and after using the devices "toaster", "hair dryer", and "Nutella".

[0113] As a first example, for the composite attribute “in front of the toaster” in the composite attribute space P, the corresponding set of basic attributes in the basic attribute space P′ includes the basic attributes [“cold”, “soft”, “white”, “light-colored”].

[0114] As a second example, for the composite attribute “after toaster” in the composite attribute space P, the corresponding set of basic attributes in the basic attribute space P′ includes the basic attributes [“hot”, “hard”, “brown”, “dark”].

[0115] As a third example, for the composite attribute "before the hair dryer" in the composite attribute space P, the corresponding set of basic attributes in the basic attribute space P' includes the basic attribute ["cold"].

[0116] As a fourth example, for the composite attribute "before the hair dryer" in the composite attribute space P, the corresponding set of basic attributes in the basic attribute space P' includes the basic attribute ["hot"].

[0117] As a fifth example, for the composite attribute "before Nutella" in the composite attribute space P, the corresponding set of basic attributes in the basic attribute space P' includes the basic attributes ["white", "light-colored"].

[0118] As a sixth example, for the composite attribute “after Nutella” in the composite attribute space P, the corresponding set of basic attributes in the basic attribute space P′ includes the basic attributes [“brown”, “dark color”].

[0119] Now, the assistance system 1 can infer for each device "toaster", "hair dryer", and "Nutella" the effect it may have on the object "bread" under certain conditions and use this as a set of new actions to plan in the decomposed planning domain Σ' of the basic attribute space P'.

[0120] 3 and 4 show the attribute information in the decomposition step S5 to generate a new decomposed planning problem in the basic attribute space P'. A standard planner implementing the planning module then uses the decomposed planning problem in the fully defined basic attribute space P' to find a solution in the second approach.

[0121] FIG. 5 provides a further example of a data structure of attribute information, showing the correspondence between objects, means for realizing effects, and effects on objects.

[0122] The devices shown in Figure 5 include examples typically found in a kitchen environment and task environments specific to the kitchen environment. The devices shown include a "refrigerator," a "soldering bar," a "toaster," a "hair dryer," a "dishwasher," a "washing machine," and a "microwave."

[0123] The attribute information shown in FIG. 5 includes means. The attribute information may define specific means that may have an effect on an object exposed to the specific means. A device does not directly affect an object, but rather exposes the object to specific means that may have an effect on the object. For example, a device "dishwasher" exposes an object to means including "detergent," "hot water," and "hot air."

[0124] The means in FIG. 5 include "heat sink," "heat source," "substance," and "force." The listed means may include further differentiation as shown in FIG. 5. For example, the means "heat source" allows differentiation between "hot metal," "infrared," "hot air," "hot water," and "microwave." The means "substance" groups together "cold air," "hot air," "hot water," and "detergent." The means "force" encompasses "rotation" and "weight." The means "substance" includes "cold air," "hot air," "hot water," and "detergent." The means "force" differentiates between "rotation" and "weight." The means "heat sink" may correspond to the means of "hot air."

[0125] Including the device "toaster" in the original planning problem defined in the composite attribute space may therefore result in a decomposed planning problem in the basic attribute space that includes the means "infrared" and "heat source." Providing the decomposed planning problem to the planning module may expand the space for searching for a solution to the planning problem because, in addition to the device "toaster," additional devices that expose the object to the means "heat source" and "infrared" may be considered.

[0126] The assistance system 1 may be configured to observe, determine, or measure the effect of a device on exposing objects to only a few objects that are common in the task environment and in combination with the device. The direct mapping shown in Figure 4 may still result in an incomplete decomposition.

[0127] By planning based on the decomposed planning problem in the basic attribute space P', refined mappings may still be advantageous or even required to achieve innovative planning results. Based on the decomposition information shown in Fig. 5, the assistance system 1 may predict as yet unknown mappings from composite attributes in the composite attribute space P to decomposed attributes in the decomposed attribute space P', leveraging knowledge of similarities with already known or already observed mappings stored in a database for the devices and objects in question.

[0128] FIG. 6 provides an exemplary data structure of attribute information showing the correspondence between the representation of means, objects, and means for realizing an effect and the basic attributes of the basic attribute space P′.

[0129] The top part of Figure 6 shows different means, including "microwave radiation," "infrared radiation," and "detergent." These different means each have a different effect on a particular object in the basic attribute space P'. In the example of Figure 6, exposing the object "knife" to "infrared radiation" has the effect that the object "knife" then has the basic attribute "hot" as temperature in the basic attribute space P'. Exposing the object "bread" to the means "infrared radiation" has a more complex effect on the object "bread." After being exposed to the means "infrared radiation," the object "bread" then has a set of basic attributes in the basic attribute space P' that include ["hot," "hard," "dark," and "brown"].

[0130] Correspondingly, the assistance system 1 may obtain from the database which effect in the basic attribute space P' is realized on an object when the object is exposed to a certain means, as shown in Figure 6. Based on the obtained means, the assistance system 1 can predict that when two devices expose the object to the same means, an unobserved device included in the task environment may have a similar effect on the object as the observed device has on the same object. For example, the decomposition in step S5 may utilize that exposing the object "bread" to a toaster has the same means "infrared radiation" as an unobserved device "baking oven."

[0131] The knowledge reflected in the attribute information regarding the means-to-effect correspondence shown in FIG. 6 may involve a generalized object category, such as “food” for “bread,” and each instance of “food,” such that the effect may be realized in the attribute information as generalizing from a particular instance, e.g., “bread,” to all objects in that category, e.g., “food,” even though it has never been observed in most of the other specific instances in the category “food.” Thus, the decomposed planning problem in the basic attribute space P′, including the category “food” and other instances of the category “food,” may further expand the space for searching for potential solutions in the planning process performed by the planning module. Thus, the decomposed planning problem in the basic attribute space P′ helps find plans for planning problems defined in the basic attribute space P′ that do not have solutions when defined in the composite attribute space P.

[0132] Embodiments of the method are particularly useful for planning applications relating to robotic kitchen assistance systems, as described with reference to a robotic kitchen assistance system.

[0133] In general, robotic assistance systems can more accurately and consistently represent the properties of objects and devices in a task environment, along with the processes they undergo. Planning based on a decomposed planning problem in the basic attribute space P' can yield more innovative, and therefore more compelling, planning solutions than previously available when the planning problem is viewed in the composite attribute space P, despite using essentially the same planning modules.

[0134] The planning-based robotic assistance system determines whether it is necessary to transition the abstract planning problem to a more physical representation of the objects in the task environment and the processes and actions they undergo. Specifically, the method uses a more physical attribute space called the decomposition attribute space P' or basic attribute space P'. The method decomposes non-basic attributes, particularly composite attributes, into a set of basic attributes. The assistance system 1 can access the attribute information as decomposition information in a database to perform the decomposition step S5. The method may also include a process for inferring additional missing basic attributes, thereby further expanding the basic attribute space P' to include additional basic attributes, which adds unobserved effects and states to the attribute information stored in the database. By building and maintaining a basic attribute representation of the task environment, the method plans using the expanded basic attribute space P', which includes objects, devices, states, and actions, and thus can find different, and often more compelling and efficient, solutions to the planning problem that are not currently available for the planning problem defined in the composite attribute space P.

[0135] FIG. 7 illustrates a particular application scenario of a method for operating an assistance system 1 including an autonomous device 2 in a kitchen environment.

[0136] An autonomous device 2 or agent is an artifact whose configuration of sensors, actuators, and embedded control systems provides a significant level of flexible, independent, and autonomous skill. The term autonomous refers to the degree to which the autonomous device 2 is capable of sensing its environment, planning behavior based on the sensed environment, and acting by performing a course of actions or skills based on the sensed environment with the intent of reaching a goal without external control. A user may provide a goal to the autonomous device 2 through a human-machine interface.

[0137] Alternatively, the autonomous device 2 may create the goal itself.

[0138] The robotic assistance system 1 of Figure 7 comprises an autonomous device 2, shown as a stationary dual-arm robot, having two effectors 8 and an object tracking device 6, e.g., a camera-based or marker-based object tracking device 6. The robotic assistance system 1 comprises a data processing device, e.g., at least one computer 4, configured to execute planning algorithms and motion generation algorithms to plan actions and sequences of actions to be performed by the autonomous device 2. In particular, a program implementing a planning process may be executed on the computer 4, which implements a planning module to devise a plan to address a task by controlling at least one effector trajectory of the effector 8.

[0139] The computer 4 may include at least one processor, at least one memory, including, for example, non-volatile and volatile memory for storing program instructions and program data generated during execution of the method.

[0140] The robotic assistance system 1 acquires a task description of a predetermined task, for example, via a human-machine interface, to acquire an instruction from a user to the robotic assistance system 1. For example, the acquired task description (task) may instruct the robotic assistance system 1 to prepare toasted bread in a domestic application including a kitchen scenario.

[0141] The object tracking device 6 determines the current location of the object 3 within the task environment. The object tracking device 6 generates and provides the sensor signals 11 to the computer 4. The sensor signals 11 may enable the robotic assistance system 1 to determine the state of the autonomous device 2 and the objects and devices 3 within the observed task environment. The observed state and sensor signals may also enable the robotic assistance system 1 to monitor the progress of the task while performing a series of skills.

[0142] The computer 4 executes, in a planning module, a planning algorithm that enables the assistance system 1 to search for a plan for solving a planning problem provided to the planning module.

[0143] The assistance system 1 may include a robot control unit 5 of the robot system 1 to control actuators of the effectors 8 of the autonomous device 2 using actuator control signals 10 generated based on control signals 12 provided by a computer 4 of the assistance system 1.

[0144] The autonomous device 2 may generate a status signal 13 and output the status signal 13 to the robot control unit 5. The robot control unit 5 may provide the information contained in the status signal 13, along with further status information about the autonomous device 2, to the computer 4 in a status signal 14.

[0145] The computer 4 may be provided with a human-machine interface, which includes input / output means, for example an output means such as a monitor 7 for displaying graphical information to a user, and input means such as a keyboard and mouse device or a microphone for receiving input from a user. The computer 4 may in particular execute software implementing a human-machine user interface. The human-machine interface may for example include a graphic user interface GUI for interacting with a user.

[0146] The computer 4 may include a human-machine interface configured to display graphical information to the user that guides the user to achieve the goal state. The human-machine interface may include a virtual reality device configured to visualize a real-world task environment to the user.

[0147] Additionally or alternatively, the human-machine interface may include an augmented reality device configured to present information to the user that guides the user in achieving the goal state as an overlay image on a task environment that exists in the real world. In this embodiment, the human-machine interface may be executed, for example, on a tablet computer equipped with a camera. The display may present the information for guiding the user as a semi-transparent overlay image on an image of the task environment captured by the camera of the tablet computer.

[0148] Alternatively, the human-machine interface may include a virtual reality device configured to visualize the generated virtual environment for the user, for example, the human-machine interface may allow the user to receive guidance for performing tasks in a computer game environment.

[0149] The structure of the robot assistance system 1 shown in FIG. 7 is an example.

[0150] It is noted that some or all structural elements of the robotic assistance system, such as the computer 4, the robot control unit 5, and the object tracking device 6, may be incorporated into the autonomous device 2.

[0151] The autonomous device 2 is not limited to being a stationary device and may be implemented as a mobile autonomous device that moves around within the task environment. The autonomous device 2 may implement an assistive home robot. To assist in the kitchen, the autonomous device 2 needs to know objects, such as food, and devices, such as kitchen utensils, how they can be used, and what effect the kitchen utensils have when applied to specific objects. For this purpose, the hierarchy of all entities present in the kitchen as the current task environment can be pre-taught or extracted from other knowledge sources, such as a publicly available database (knowledge base) accessed via a computer network (not explicitly shown in FIG. 7 ), or it may be pre-taught and extracted from other knowledge sources.

[0152] The autonomous device 2 may have pre-stored knowledge of several tasks, e.g., preparing a sandwich, which may involve skills such as cutting bread, toasting bread, cutting lettuce, and peeling and slicing tomatoes. The proposed method for operating the assistance system 1 using a decomposed program representation in the basic attribute space P' improves the availability of the assistance system 1 in novel situations in the task environment. For example, when tasked with preparing toasted bread, the robotic assistance system 1 may overcome the problem that the only available toaster is defective and utilize an oven instead to solve the decomposed planning problem in the basic attribute space P'.

[0153] All steps performed by various entities and functions described as being performed by various entities described in this disclosure are intended to mean that the respective entities are adapted or configured to perform the respective steps and functions. In the claims and the description, the word "comprises" does not exclude the presence of other elements or steps.

[0154] In the specification and claims, the phrase "at least one of A and B" can be substituted for the phrase "A and / or B," and vice versa, since they are used interchangeably.

[0155] The indefinite article "a" or "an" does not exclude a plurality. A single element or other unit may fulfill the functions of several entities or items recited in the claims. The mere fact that different dependent claims recite certain means and features of a conversion circuit does not exclude that combinations of these means and features cannot be combined in an advantageous implementation.

Claims

1. 1. A method for operating an assistance system based on an attribute space, comprising: obtaining task information defining at least one goal state to be achieved by the assistance system; obtaining initial state information defining an initial state of at least one of an autonomous device or a user and at least one object involved in achieving the goal state or one of the goal states; defining a planning problem based on the obtained task information, the initial state information, and further based on attribute information related to the at least one object stored in a database; selecting, by a planning module, an action or sequence of actions for solving the defined planning problem; generating control signals and outputting the control signals to at least one of a human-machine interface (HMI) to assist the user in achieving the goal state or one of the goal states, or to the autonomous device to achieve the respective goal state; The method comprises: determining whether the selected action or series of actions will achieve the goal state; and if it is determined that the selected action or series of actions will not achieve the goal state, decomposing the defined planning problem based on the attribute information stored in the database to generate a decomposed planning problem, and providing the decomposed planning problem to the planning module; solving the decomposed planning problem with the planning module; generating the control signals based on the solved decomposed planning problem and outputting the control signals to at least one of the human-machine interface to assist the user in achieving the goal state or to the autonomous device to achieve the goal state; The method further comprises:

2. 2. The method for operating an assistance system of claim 1, wherein the stored attribute information includes composite attributes and basic attributes that define at least a state of the at least one object, an initial state of the planning problem, and an effect a device has on the at least one object under at least one predetermined condition.

3. A method for operating the assistance system described in claim 1, wherein decomposing the defined planning problem includes mapping at least one composite attribute included in the defined planning problem to a set of basic attributes based on the stored attribute information.

4. 4. The method for operating an assistance system of claim 3, wherein said stored attribute information comprises a correspondence table storing composite attributes associated with a set of basic attributes for performing said mapping.

5. The mapping step may include mapping at least one of the at least one goal state g, the initial state s, and the planning domain Σ in the composite attribute space to at least one respective basic goal state g′, the basic initial state s in the basic attribute space. ｉ 4. A method for operating an assistance system according to claim 3, comprising mapping the master plan domain Σ′ to the master plan domain Σ′.

6. 4. A method for operating an assistance system as claimed in claim 3, comprising predicting a new mapping of said composite attribute to said set of basic attributes based on a similarity of said new mapping to a predetermined mapping in terms of a similarity of said at least one object to other objects.

7. 7. The method for operating an assistance system of claim 6, wherein predicting a new mapping from the composite attribute to the set of basic attributes based on the similarity includes predicting that the at least one object will have a similar effect as the other object when the at least one object and the other object are exposed to the same means.

8. 7. The method for operating an assistance system of claim 6, wherein predicting a new mapping from the composite attribute to the set of basic attributes based on the similarity comprises generalizing a prediction that a given object has an effect on an object category and predicting that another given object of the same object category will subject the object to the same measures and produce the same effect as the given object.

9. 7. The method for operating an assistance system according to claim 6, wherein predicting a new mapping from the composite attribute to the set of basic attributes is based on at least one of predetermined relationship information provided during a design phase of the assistance system, relationship information generated by an expert, and relationship information generated online during an interaction process of the assistance system with the user.

10. 2. A method for operating an assistance system according to claim 1, comprising iteratively decomposing the defined planning problem based on the stored attribute information to generate the decomposed planning problem until predetermined criteria are met.

11. solving the decomposed planning problem, if the selected action or sequence of actions achieves the goal state or one of the goal states; determining that the predetermined criteria are met; 11. A method for operating an assistance system according to claim 10, comprising:

12. if no further mapping of the composite attribute to a set of corresponding basic attributes is possible based on said stored attribute information; determining that the predetermined criteria are met; 12. A method for operating an assistance system according to claim 11, comprising:

13. 2. The method for operating an assistance system according to claim 1, wherein said method is performed online during a planning process to achieve said goal state or one of said goal states.

14. The method for operating an assistance system according to claim 1 , wherein the assistance system is included in a robotic assistance system that includes the autonomous device.

15. the assistance system is integrated into a controller of the autonomous device; or The assistance system is implemented as software running on a remote server connected to the autonomous device via a network. A method for operating an assistance system according to claim 1.

16. the assistance system guides the user to achieve the at least one goal state via the human-machine interface; the human-machine interface includes a virtual reality device configured to visualize a real-world task environment or a generated virtual environment for the user; or The human-machine interface includes an augmented reality device configured to present information for guiding the user as an overlay on the task environment present in the real world and perceived by the user. A method for operating an assistance system according to claim 1.

17. 1. A non-transitory computer-readable medium having embodied thereon a computer-readable program that, when executed on a computing device, causes the computing device to perform a method for operating an assistance system based on an attribute space, the method comprising: obtaining task information defining at least one goal state to be achieved by the assistance system; obtaining initial state information defining an initial state of at least one of an autonomous device or a user and at least one object involved in achieving the at least one goal state; defining a planning problem based on the obtained task information, the initial state information, and further based on attribute information related to the at least one object stored in a database; selecting, by a planning module, an action or sequence of actions for solving the defined planning problem; generating control signals and outputting the control signals to at least one of a human-machine interface (HMI) to assist the user in achieving the goal state or one of the goal states, or to the autonomous device to achieve the respective goal state; The method comprises: determining whether the selected action or series of actions will achieve the goal state; and if it is determined that the selected action or series of actions will not achieve the goal state, decomposing the defined planning problem based on the attribute information stored in the database to generate a decomposed planning problem, and providing the decomposed planning problem to the planning module; solving the decomposed planning problem with the planning module; generating the control signals based on the solved decomposed planning problem and outputting the control signals to at least one of the human-machine interface to assist the user in achieving the goal state or to the autonomous device to achieve the goal state; a non-transitory computer-readable medium, further comprising:

Images