Method for controlling a technical system

By combining a multi-agent framework with a generative language model and a symbolic planner, the system autonomously generates and adapts planning problem models, solving the problem that requires extensive expert intervention in existing technologies. This enables autonomous and reliable planning in complex environments, improving the system's scalability and ease of use.

CN123060633APending Publication Date: 2026-11-10ROBERT BOSCH GMBH
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202610622863.5
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Priority Date
2025-05-08
Filing Date
2026-05-08
Publication Date
2026-11-10

AI Technical Summary

Technical Problem

Existing control technology systems require extensive human expert intervention in complex and dynamic environments, making it difficult to achieve autonomous and reliable planning. Furthermore, existing solutions are inefficient when the environment changes.

Method used

A multi-agent framework is adopted, which combines generative language model (LLM) with symbolic planner. The planning problem model is autonomously generated and adapted through domain model generator, initial state model generator and target state model generator. The plan is created by symbolic planner, which reduces expert intervention and improves consistency.

Benefits of technology

It enables autonomous and reliable planning in complex environments, reduces expert intervention, improves system scalability and ease of use, and adapts to dynamic environmental changes.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN123060633A_ABST
    Figure CN123060633A_ABST
Patent Text Reader

Abstract

According to different embodiments a method for controlling a technical system is provided, having: receiving a text domain description, the text domain description describing an action which can be performed by the technical system and which changes a state property of the technical system; generating a domain model from the domain description by means of a domain model generator; receiving a text initial state description, the text initial state description describing an initial state of the technical system; generating an initial state model from the initial state description by means of an initial state model generator; receiving a text target state description, the text target state description describing a target state of the technical system; generating a target state model from the target state description by means of a target state model generator; creating a plan by means of a symbolic planner by feeding the domain model, the initial state model and the target state model to the symbolic planner; and controlling the technical system according to the created plan.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This disclosure relates to a method for controlling a technical system. Background Technology

[0002] Control technology systems typically require the planning of action sequences. Traditional approaches to planning in complex environments where many different control scenarios may arise often require extensive manual intervention by human users, which limits their scalability and ease of use.

[0003] For example, symbolic planners require domain models created by experts, which must be manually updated whenever the environment changes, making them unsuitable for dynamic, real-world applications. Large language models (LLMs), while capable of producing structured output from natural language, are prone to errors and inconsistencies.

[0004] Therefore, the desired outcome is a planning scheme that achieves the most autonomous planning possible (without or with only a small amount of human expert intervention) while simultaneously ensuring reliable planning. Summary of the Invention

[0005] According to different implementation methods, a method for controlling a technical system is provided, the method comprising: • Receive a text field description that describes an action that can be performed by the technical system and changes the state characteristics of the technical system (e.g., whether one container is on another container); • Generate a domain model from the domain description using a domain model generator (i.e., the domain model specifies the characteristics and possible actions described in the domain description). • Receive a text initial state description, which describes the initial state of the technical system; • An initial state model is generated based on the initial state description using an initial state model generator; • Receive a text target state description, which describes the target state of the technical system; • A target state model is generated based on the target state description using a target state model generator. The domain model generator, initial state model generator, and target state model generator are implemented using at least one generative language model (e.g., each using a corresponding generative language model, such as at least one LLM), which processes the domain description, initial state description, or target state description. Among them, the domain model generator, the initial state model generator, and the target state model generator exchange feedback about the domain model, the initial state model, and / or the target state model when generating the domain model, the initial state model, and / or the target state model, and adapt the domain model, the initial state model, or the target state model until the domain model, the initial state model, and the target state model are consistent and / or complete (i.e., they together yield a consistent and / or complete planning problem model). • By feeding the domain model, initial state model, and target state model to the symbolic planner, a (control) plan is created using the symbolic planner; and • Based on the established planning and control technology system.

[0006] Unlike traditional planning schemes or systems that require expert intervention or rely on unreliable LLM-based reasoning (making them inefficient for dynamic environments), the above approach transforms planning into a modular, adaptable process where one or more LLMs help structure the problem, while a symbolic planner ensures the correctness of the plan. This results in a scalable, automated, and easy-to-use solution. It enables reliable control in complex environments encompassing diverse objects, scenarios, and tasks. Depending on the implementation, this is achieved through cooperation among multiple agents, where the generative flexibility of a large language model (LLM) is combined with the structured reasoning capabilities of a symbolic planner. This approach is modular, scalable, and adaptable to a wide range of planning problems, particularly those that are simultaneously complex and unstructured.

[0007] Therefore, the above method provides a general and scalable solution for the planning of control technology systems. By combining at least one generative language model (e.g., one or more LLMs) with symbolic planning within a modular multi-agent framework, automatic adaptation to changing environments, improved consistency, and lower dependence on expert intervention are ensured. Thus, this planning scheme is well-suited for real-world applications such as robotics, industrial automation, and autonomous systems.

[0008] A symbolic planner is a component in the cognitive architecture of an autonomous robotic device that uses symbolic representation and logical reasoning to plan tasks. This planner creates a (high-level) plan based on a symbolic representation of the environment and the robot's goals. The plan includes a sequence of actions to be performed (by lower-level control) considering the current state and the desired target state. An example of this is a planner based on PDDL (Planning Domain Definition Language).

[0009] Different embodiments are described below.

[0010] Example 1 is a method for controlling a technical system, as described above.

[0011] Example 2 is the method according to Example 1, wherein the initial state model generator checks whether the state characteristics specified in the initial state description are included in the domain model, and, in response to the initial state model generator confirming that one or more state characteristics specified in the initial state description are not included in the domain model, requests the domain model generator to supplement the domain model with one or more state characteristics.

[0012] This is an example of how to establish consistency and / or integrity in a planning problem model.

[0013] Example 3 is the method according to Example 1 or 2, wherein the target state model generator checks whether the state characteristics specified by the target state description are included in the domain model, and, in response to the target state model generator confirming that one or more state characteristics specified by the target state description are not included in the domain model, requests the domain model generator to supplement the domain model with one or more state characteristics.

[0014] This is another example of how to establish consistency and / or integrity in planning problem models.

[0015] Example 4 is a method according to any one of Examples 1 to 3, wherein the target state model generator checks whether the actions that enable the transition from the initial state to the target state are included in the domain model, and, in response to the target state model generator confirming that one or more actions that enable the transition from the initial state to the target state are not included in the domain model, requests the domain model generator to supplement the domain model with one or more actions.

[0016] This is another example of how to establish consistency and / or integrity in planning problem models.

[0017] Example 5 is a method according to any one of Examples 1 to 4, wherein the target state model generator checks whether the state characteristics specified by the target state description are included in the initial state model, and, in response to the target state model generator confirming that one or more state characteristics specified by the target state description are not included in the initial state model, requests the initial state model generator to supplement the initial state model with one or more state characteristics.

[0018] This is another example of how to establish consistency and / or integrity in planning problem models.

[0019] Example 6 is a method according to any one of Examples 1 to 5, wherein the domain model contains variables, and each variable can have multiple values ​​and describe the state characteristics of the technical system.

[0020] In this way, the domain model can be processed by the symbolic planner.

[0021] Example 7 is the method according to Example 6, wherein the initial state model generator generates an initial state model such that the initial state model confirms the values ​​of at least a portion of the variables based on the initial state description.

[0022] Therefore, the initial state is well-defined for the symbolic planner and follows the corresponding textual description.

[0023] Example 8 is the method according to Example 6 or 7, wherein the target state model generator generates a target state model such that the target state model confirms the values ​​of at least a portion of the variables according to the target state description.

[0024] In this way, the target state is explicitly defined for the symbolic planner and follows the corresponding textual description.

[0025] Example 9 is a method according to any one of Examples 1 to 8, wherein a domain description, an initial state description, and a target state description are fed to at least one generative language model, and a requirement is attached for a component to create a planning problem model or a planning problem model for a symbolic planner.

[0026] Therefore, intuitively, one or more generative language models are used as an interface between text description sources (i.e., human users, documents, instructions, etc.) and the symbol planner. This improves scalability and adaptability compared to using the symbol planner alone.

[0027] Example 10 is a control device designed to perform the method according to any one of Examples 1 to 9.

[0028] Example 11 is a computer program having instructions that, when executed by a processor, cause the processor to perform the method according to any one of Examples 1 to 9.

[0029] Example 12 is a computer-readable medium storing instructions that, when executed by a processor, cause the processor to perform the method according to any one of Examples 1 to 9. Attached Figure Description

[0030] In the accompanying drawings, similar reference numerals generally refer to the same parts in all different views. The drawings are not necessarily drawn to scale, and instead, the emphasis is generally placed on illustrating the principles of the invention. In the following description, different aspects are described with reference to the following drawings.

[0031] Figure 1 A robot is shown as an example of a technological system to be controlled.

[0032] Figure 2 This illustrates an example of a planning problem model that must be adapted.

[0033] Figure 3 This describes a planning system with a multi-agent planning framework architecture, based on one implementation method.

[0034] Figure 4 This shows an example of an agent architecture.

[0035] Figure 5 A flowchart illustrating a method for controlling a technology system according to one embodiment is shown. Detailed Implementation

[0036] The following detailed description is taken with reference to the accompanying drawings, which are illustrated for explanation of specific details and aspects of the present disclosure that enable the implementation of the invention. Other aspects can be used, and structural, logical, and electrical changes can be made without departing from the scope of protection of the invention. Different aspects of the present disclosure are not necessarily mutually exclusive, as some aspects of the present disclosure can be combined with one or more other aspects of the present disclosure to form new aspects.

[0037] Different examples are described in more detail below.

[0038] Figure 1 Robot 100 is shown as an example of a technical system to be controlled.

[0039] Robot 100 includes a robotic arm 101, such as an industrial robotic arm for manipulating or assembling a workpiece or one or more other objects 113. In controlled situations, obstacles 114 may also be present, such as containers containing objects 113, or human users, and the robotic arm 101 should not collide with these obstacles. The robotic arm 101 includes manipulators 102, 103, 104 and a base (or carrier) 105 supporting the manipulators 102, 103, 104. The term "manipulator" refers to a movable element of the robotic arm 101, the actuation of which enables physical interaction with the surrounding environment, for example, to perform a task. For control, robot 100 includes a (robot) control device 106 configured to perform interactions with the environment according to a control program. The last (farthest from the carrier 105) element 104 of the manipulators 102, 103, and 104 is also referred to as the end effector 104 and includes a gripping tool (which may also be a suction cup gripper).

[0040] Other manipulators 102, 103 (closer to carrier 105) are able to form a positioning device, such that together with the end effector 104, a robotic arm 101 is formed having the end effector 104 at its end. The robotic arm 101 is a mechanical arm capable of performing functions similar to those of a human arm.

[0041] The robotic arm 101 may include joint elements 107, 108, and 109, which connect manipulators 102, 103, and 104 to each other and to a carrier 105. Joint elements 107, 108, and 109 may have one or more joints, each of which is capable of rotational movement (i.e., rotational motion) and / or translational movement (i.e., displacement) relative to its associated manipulator. Movement of manipulators 102, 103, and 104 may be initiated by means of actuators controlled by a control device 106.

[0042] The term "actuator" can be understood as a component that, when actuated, is adapted to act on a mechanism or process. An actuator is capable of translating instructions from control device 106 into mechanical motion. An actuator, such as an electromechanical converter, can be configured such that it converts electrical energy into mechanical energy in response to manipulation.

[0043] The term "control device" can be understood as any type of entity implementing logic, which may include, for example, circuitry and / or a processor capable of executing software, firmware, or a combination thereof stored in a storage medium, and capable of issuing commands, for example, to the actuator in this example. The control device can be configured, for example, via program code (e.g., software) to control the operation of a system, in this example, a robot.

[0044] In this example, the control device 106 includes one or more processors 110 and a memory 111, in which code and data are stored, and the processors 110 control the robotic arm 101 based on the code and data. According to different implementations, the control device 106 controls the robotic arm 101 based on a machine learning model 112 (e.g., including one or more neural networks), which is stored in the memory 111.

[0045] When a control technology system performs a task, actions are typically planned, such as grasping, moving, and placing in the case of a robotic arm, and then the actions are executed (this requires precise control of the motion process, i.e., the robot must have specific capabilities, such as safely grasping objects).

[0046] Planning in uncertain, dynamic, and complex environments, such as... Figure 1In robotics, as exemplified by autonomous driving or industrial automation (environments containing a large number of objects and possible control situations, and performing different tasks, referred to below as open-world environments), a typical approach is to use either a classic symbolic planner or a separate large generative language model (typically a large language model (LLM)) for planning.

[0047] Symbolic planners, such as PDDL-based planning frameworks, are very good at creating optimal plans, but require experts to manually create domain models, which is often associated with significant costs (requiring human intervention to confirm the definitions of objects, predicates, and actions).

[0048] Furthermore, the planners' reliance on deterministic task execution and complete environmental awareness limits their applicability in real-world, unpredictable scenarios. Any change to the initial state or objective requires manual updates to the task specifications, making spontaneous adaptation impossible. This renders them inefficient in environments requiring frequent reconfiguration.

[0049] For example, if you want a robot to set a table, a PDDL-based planner can generate the optimal order for placing the cutlery. However, if circumstances change (e.g., the table is moved to another location), the human user must manually modify the initial state or objective in the task specification. If the robot's capabilities are further developed—for example, by gaining the ability to stack plates—the domain specification must also be manually updated to introduce the corresponding new actions (based on the new capabilities).

[0050] This manual intervention can only be poorly extended in dynamic environments with frequent state changes, which limits the effectiveness of symbolic planners in real-world applications.

[0051] On the other hand, LLM has proven promising in bridging the gap between natural language understanding and symbolic reasoning. Frameworks such as LLM+P and NL2Plan have demonstrated LLM's ability to transform natural language input into structured representations (e.g., domain models and plans) using various formats such as PDDL or Python code.

[0052] However, these schemes struggle to maintain consistency, track state changes, and handle iterative corrections. They lack critical components such as a refiner for feedback control and persistent memory, which are essential for adapting to evolving task requirements and maintaining consistency throughout the planning process.

[0053] For example, an LLM-based planner can generate the step of "pick up the plate and put it on the table", but if a new requirement for the color of the plate is added during the task (e.g., when the user explicitly wants to select the "red plate"), the planner will have difficulty taking into account the new requirement without explicitly resetting its planning process.

[0054] Overall, existing approaches for planning are hampered by limited modularity, a lack of robust error correction mechanisms, and poor adaptability to changing environments. These limitations make the control of technical systems susceptible to interference and hinder effective scalability to meet the demands of high-risk environments, such as those requiring continuous adaptation in robotics, healthcare, and automation.

[0055] To achieve planning that is as autonomous as possible (with little or no intervention from human experts) and reliable at the same time, a feedback-controlled multi-agent scheme is provided, depending on the implementation method. This multi-agent scheme has a modular and structured architecture and combines one or more LLMs with symbolic reasoning (i.e., symbolic planners).

[0056] Instead of relying on human experts to manually update planning models, or expecting LLMs to create complete plans, specialized agents (i.e., planning modules, or software agents, for example, in the form of one or more LLMs or supported by one or more LLMs) autonomously generate, validate, and refine planning components based on the provided scheme.

[0057] Each agent focuses on a specific part of the task description—such as generating the domain model, confirming initial conditions, or adapting to the target—while feedback loops between agents ensure the adaptability and correctness of the task description. The symbolic planner then undertakes planning based on the task description provided by the agents.

[0058] This reduces manual work costs, improves scalability, and makes structured planning easy for non-experts to use by enabling users to interact with the corresponding planning system (which uses the provided planning scheme) in natural language (through the use of LLM), while ensuring reliability through symbolic reasoning (as performed by the symbolic planner), since the actual planning is done through the symbolic planner.

[0059] The modular architecture enables seamless adaptation to new domains or integration into other planning systems with minimal reconfiguration. This allows the planning framework to be easily scaled to diverse applications, including autonomous vehicles, drone delivery, and industrial robotics, without significant modifications.

[0060] This planning system uses feedback loops between agents to ensure that all components of the planning problem remain context-dependent and coordinated with each other. This minimizes errors in plan creation and execution, while simultaneously enabling iterative improvement—that is, agents iteratively communicate and refine their results through feedback loops, thereby ensuring consistency and adaptability to evolving scenarios or user constraints.

[0061] For example, when an agent responsible for inventory tracking in a warehouse automation system detects a missing item, the agent can trigger a query to a human user and adapt the planning problem based on the response (which does not necessarily have to be provided by an expert for planning). This ensures that the planning process is consistent with the current state of the warehouse, thereby minimizing disruptions and maintaining efficiency.

[0062] Compared to static symbolic planners that require manual updates to new constraints or evolving objectives, this planning system can therefore automatically adapt to new or uncertain task requirements. Consequently, it is extremely effective in environments where real-time adaptability is crucial. In the case of automated parking, autonomous vehicles using this planning system can adapt to unforeseen constraints during the parking process. For example, if the car initially plans to park in a specific location but identifies an obstacle (e.g., a vehicle parked in the wrong spot or a pedestrian crossing the road), the planning system can refine its parking strategy by adapting its maneuvering or selecting a suitable nearby location—without external intervention. This adaptability ensures smoother parking in dynamic environments where conditions change unpredictably.

[0063] Figure 2 This illustrates an example of a planning problem model that must be adapted.

[0064] In the first control scenario 201, in an environment, such as the environment of the robotic arm 101, there are three containers (labeled b1, b2, and b3). In the initial state 202, b2 is on top of b1, and the user 203 (in natural language) creates a task 204: that b2 should be stacked on top of b3, i.e., the target state 205 should be achieved where b2 is on top of b3.

[0065] Planner 206 creates plan 207 for this goal and triggers control to follow the plan.

[0066] In the second control scenario 208, there are again three containers (labeled b1, b2, and b3), where in the example described, b3 is stacked on top of b1. Furthermore, the containers are visually distinguishable (i.e., in terms of appearance, such as color), indicated here by diagonal lines, cross lines, horizontal lines, and dimensions.

[0067] User 203 now (in natural language) creates task 209: The tower should be formed by these three containers, with the largest container at the bottom, the container with the cross lines in the middle, and the container with the horizontal lines at the top.

[0068] If based on a domain model sufficient to describe the first control case 201, then in the domain model, for example, there are no predicates “container size” and “container appearance”, that is, the domain model must be extended by the predicates and must be assigned values ​​to them in the initial state and the final state.

[0069] Figure 3 The planning system 300 according to one embodiment (e.g., implemented by control device 106) has a multi-agent planning framework architecture.

[0070] exist Figure 3 In this embodiment, three specialized agents—a domain model generator 301, an initial state model generator 302, and a target state model generator 303—collaborate within the planning system 300 to iteratively create a coherent and structured planning problem model 309 in a format suitable for the symbolic planner 310, such as JSON, PDDL, or as Python code. Each agent 301, 302, and 303 is responsible for a specific aspect of the planning task: • Domain model generator 301 creates a domain model (i.e., a model of the environment (and consequently, of possible control situations) and possible actions that the technical system to be controlled (e.g., robotic arm 101) can perform) based on domain information 304 (e.g., at least in part in natural language) received by planning system 300. Domain modeling can include, for example, the following: identifying (appropriate) predicates to represent (e.g., objects), the environment, and the states and properties of transitions between states within that environment; defining applicable actions, their preconditions, and effects; identifying objects and their properties; and specifying constraints. For the first control case 201, the domain information 304 includes, for example, a domain description (excerpt), such as: "This domain is based on a world in which containers can be stacked and unstacked, picked up and placed on the ground or other containers by means of a robotic arm." Actions include: - Pick up: Lift the container from the ground. Prerequisites: - The container is free (no other container stands on it). - The container stands upright on the ground. -The arm is empty. Effect: - The arm now holds the container -The container is no longer free - The container is no longer standing upright on the ground. - The arm is no longer empty -Place down: Place the container to be held on the ground. … - Stacking: Placing the container that will be held onto another container. … - Unloading …” • The initial state model generator 302 specifies the initial state of the system to be controlled based on the initial state description 305 (e.g., at least in natural language) received by the planning system 300. for Figure 1 The first control case 201, with an initial state description 305, is, for example: "There are three containers, b3, b2, and b1. b2 is on top of b1. The robot arm is empty." • The target state model generator 303 specifies the desired result based on the target state description 306 (e.g., at least in part in natural language) received by the planning system 300, which can exist in the form of a task description. for Figure 1 The first control condition 201 in the target state description 306 is, for example: "The target is to bring the container into the following configuration: b2 should stand on b3. b1 should be located next to it." Each agent 301, 302, 303 provides a portion of the planning problem model 309 that can be understood or used by the symbolic planner 310. For example, if each agent 301, 302, 303 provides Python code, then the aforementioned portion (excerpt) could be, for example, as shown below (different functions and variables are represented here using English terms as is common in programming): Domain description: from up_interface import add_action_effect, add_action_precondition,add_fluent_to_problem,create_action, create_fluent, create_problem, create_user_type, add_actions_to_problem from unified_planning.shortcuts import BoolType #Definition of the problem container_world_problem = create_problem("container_world") #Defined necessary types container = create_user_type("container") # A container that can be manipulated #Define Stream clear = create_fluent("clear", BoolType(), parameters={"b":container}) # It has no container. on_ground = create_fluent("on_ground", BoolType(), parameters={"b":container}) # The container is on the ground … #Add stream to the problem add_fluent_to_problem(container_world_problem, clear) add_fluent_to_problem(container_world_problem, on_ground) #Defined Actions #pickup: Pick up a container from the ground pickup = create_action("pickup", {"b": container}) #Prerequisites add_action_precondition(pickup, clear(pickup.b)) # The container must be free. … #Effect add_action_effect(pickup, holding(pickup.b), True) # The arm now holds the container … #Place down: Place the container to be held on the ground. putdown = create_action("putdown", {"b": container}) #Prerequisites … #Add action to question add_actions_to_problem(container_world_problem, [pickup, putdown,stack, unstack]) Initial state: from up_interface import add_objects_to_problem, create_object, set_initial_value # Generate container object b1 = create_object("b1", container) b2 = create_object("b2", container) b3 = create_object("b3", container) #Add object to problem add_objects_to_problem(container_world_problem, [b1, b2, b3]) #Set the initial value of the stream #Container location set_initial_value(container_world_problem, on(b2, b1), True) # b2 is on b1 … #Freedom `set_initial_value(container_world_problem, clear(b2), True)` # b2 is free. … #Arm Status set_initial_value(container_world_problem, arm_empty(), True) # The arm is empty. #Container not kept `set_initial_value(container_world_problem, holding(b1), False)` # Arm not holding b1 … The code for the target state can be similar to the code for the initial state.

[0071] In PDDL, the planning problem model 309 can be represented as follows (where each agent 301, 302, and 303 contributes a corresponding part): The planning problem model 309 is provided to the symbolic planner 310, which calculates a plan with actions to be performed and provides it to the (lower-level) control unit 310, such as a controller, so that the control unit performs the actions and controls the technology system 311 according to the plan.

[0072] Inputs 304, 305, and 306 corresponding to each aspect (domain, initial state, and objective) to the planning system 300 are (e.g., via an input interface or input data preprocessing module) fed to the corresponding agents 301, 302, and 303. Furthermore, update request signals (in...) Figure 3 (Shown by dashed and dotted arrows) This enables agents to communicate with each other or request changes or corrections to ensure that changes in one component (e.g., new restrictions or updated information) trigger corresponding adaptations in other components.

[0073] For example, the initial state model generator 302 and the target state model generator 303 can request a domain model update 307 from the domain model generator 301. This can be achieved, for example, by combining... Figure 2 As in the example mentioned, when, for example, the target state model generator 303 determines that it cannot adequately create the target state (e.g., a container with a specific appearance at a specific location), it is because the domain model lacks the corresponding predicate (e.g., container appearance). Similarly, when, for example, the initial state model does not describe a characteristic that has been described or must be described in the target state model (and thus the initial state model and the target state model are not consistent), the target state model generator 303 can request an initial state model update 308 from the initial state model generator 302.

[0074] The planning system 300 receives inputs 304, 305, and 306 in various formats, such as descriptions in natural language, structured data, or sensor signals (e.g., images and audio inputs from cameras, LiDAR, and radar) that provide information about control conditions and / or initial states. The planning system processes these inputs and converts them into structured representations (e.g., predicates, flows, states, actions), which serve as the basis for planning.

[0075] Possible input types (and the corresponding models and / or (if necessary, processing techniques placed before agents 301, 302, 303) implemented for this purpose) are: • Text-based input: Written summaries and descriptions, such as those about the initial situation (i.e., the current situation) or possible interaction schemes with the relevant application (e.g., application programming interface (API)). This type of data can be processed through language models (e.g., LLM) or text embedding models. Depending on the implementation, the input to each agent 301, 302, 303 is at least partially text-based. • Image-based input: Digital images and video streams are processed using computer vision or deep learning techniques (such as those implemented by agents 301, 302, and 303) to extract relevant spatial and semantic information. Radar and LiDAR inputs: Sensor data from LiDAR and radar can be encoded using deep learning models (e.g., 3D object recognition networks and scene representation models) to create a structured representation of the environment. Thus, the planning system 300 can identify objects, estimate distances, and recognize dynamic changes. • Audio input: Audio signals, such as verbal commands or environmental cues, are converted into text by the planning system 300 using a speech-to-text model, enabling the interpretation of user commands or the deduction of information about the control situation using NLP (Natural Language Processing) technology (similar to text-based input).

[0076] The planning is subdivided into planning subtasks, which can be coarse-grained or fine-grained depending on the complexity and requirements of the (control) task problem to be performed (e.g., can be performed on larger or smaller parts of the (control) task to be performed, respectively). The planning subtasks are: • Generate the domain model (using the domain model generator 301) • Specify initial conditions (via the initial state model generator 302) • Specify the target (via the target state model generator 303) • Resolve inconsistencies between planning components (e.g., by exchanging appropriate feedback (i.e., requests) between agents 301, 302, and 303).

[0077] As described above, the planning subtasks of domain model generation, initial state specification, and target state specification are assigned to specialized agents 301, 302, and 303. These agents operate autonomously but can collaborate by communicating via shared memory or alternative communication channels, such as application programming interfaces (APIs), interrupts, and messaging interfaces. Agents 301, 302, and 303 collaborate iteratively through refinement and feedback loops, whereby they critique, update, and coordinate their results (through communication with each other, for example, using the aforementioned communication possibilities) until a consistent planning problem model 309 is created. This collaboration ensures coherence, consistency, and adaptability among all components of the planning problem.

[0078] Figure 4 An example of the architecture of agent 400 (which can be used for one or more of agents 301, 302, 303) is shown.

[0079] The agent 400 receives an input prompt 401 and processes the input prompt using a generation module 403. The generation module is responsible for creating output (according to the planned subtasks), which is implemented using an LLM and can be (optionally) supplemented by an external tool 404 to enhance its generation capabilities.

[0080] Input information from input prompt 401 and optionally from procedural memory 402 (for long-term, collected experience) flows into the generation process executed by generation module 403, ensuring that not only immediate instructions but also historical context are available when necessary. Once generation module 403 generates output (corresponding to a planning subtask to be executed by agent 400), the output optionally passes through error checking module 405, in which agent 400 checks the correctness and consistency of the output. If an error is found, error checking module 405 sends a signal for iterative improvement back to generation module 403 via a feedback loop.

[0081] If necessary, after iterative improvements, the output is then optionally forwarded to the self-reflection module 406, where it is further refined through internal evaluation and self-criticism. Following self-reflection, an evaluation module 407 optionally assesses whether the output meets a pre-defined quality threshold; otherwise, negative feedback is sent to the generation module 403. Only results exceeding the threshold are output by the agent 400 as the final output 408.

[0082] During the process, a context memory continuously stores recent interactions and data to ensure the agent remains consistent across multiple iterations. This enables effective collaboration between agents, as they are thus able to respond consistently to external requests and maintain coordination between components. With its integrated feedback loop and optional extension modules, the design supports iterative refinement and dynamic learning, making agent 400 both robust and adaptable to complex planning tasks.

[0083] The agent can have a context-sensitive memory 409, which is responsible for interactions (e.g., interactions with other agents), feedback, and coordination between tasks by storing task-related information and the current mode of interaction. It can store the entire course or subset of the last interaction (e.g., the most recent n interactions). This ensures that the agent can maintain the consistency of its output and dynamically adapt to evolving inputs or constraints.

[0084] Based on the planning problem model 309, the symbolic planner 310 creates a sequential plan. Plan creation involves evaluating actions, states, and transitions to ensure the plan corresponds to the defined constraints and objectives.

[0085] The plan can be represented in different formats suitable for execution or communication. Examples of this are natural language instructions (for human operators) and machine-readable formats (for autonomous systems).

[0086] The planning system 300 can include additional components to improve performance, but these additional components are not absolutely necessary for the core functionality, such as: • Program memory 402: The program memory detects teachings from previous planning problems, including user feedback, corrections to recurring errors, and strategies leading to successful results. The components for program memory (i.e., for storing and recalling processes to complete tasks) enable lifelong learning by storing knowledge of the execution (particularly planning) of previous tasks and the corrections made during that process. Thus, by avoiding repetitive errors and incorporating proven methods, the planning system can improve over time. For example, a robotic system used in a logistics center can learn from previous error corrections, thereby improving its efficiency. In a scenario where a robot removes eggs from a refrigerator, program memory enables the system to recognize, for example, that subsequently closing the refrigerator door is a generally expected action, even if it is not explicitly required to achieve the goal. • Self-reflection / criticism mechanism of self-reflection module 406 or evaluation module 407: The planning system is capable of implementing a structured self-reflection process to iteratively refine the output of agents 301, 302, and 303 of generation module 403. After generating a response, an internal critic (e.g., an LLM) evaluates the response according to predefined criteria such as correctness, coherence, and task alignment. If necessary, the output is updated based on the critic's feedback, and the generation process is repeated until an acceptable quality threshold is reached. This iterative refinement improves accuracy, robustness, and adaptability.

[0087] Depending on the complexity of the control task, the planning system 300 can be configured differently, and the intelligent agents 301, 302, and 303 can also be combined, for example: • Coarse configuration: In simpler scenarios, a small number of agents (e.g., three) collaborate to define the domain model, initial state, and goal. • Fine-grained configuration: In more complex scenarios, planning subtasks can be further subdivided.

[0088] Multiple agents collaborate to ensure that the planning problem model is coherently defined and coordinated in relation to the requirement context. For communication between agents used for this collaboration, different communication architectures can be used, including tool call APIs, messaging interfaces, exception-based communication, structured output generation (e.g., JSON, XML), tagged text, and keywords.

[0089] As a symbolic planner 310, it can support different schemes, and therefore the planning system 300 can be adapted to different applications. Possible symbolic planners include, for example, PDDL-based planners, other heuristic planners, and time-series planners.

[0090] In summary, based on different implementation methods, a method is provided, such as... Figure 5 As shown in the image.

[0091] Figure 5 A flowchart 500 is shown illustrating a method for controlling a technology system according to one embodiment.

[0092] In step 501, a text field description is received, which describes an action that can be performed by the technical system and changes the state characteristics of the technical system (e.g., whether one container is on another container).

[0093] In 502, a domain model is generated from the domain description using a domain model generator (i.e., the domain model specifies the characteristics and possible actions as described in the domain description).

[0094] In step 503, a text initial state description is received, which describes the initial state of the technical system.

[0095] In 504, an initial state model is generated based on the initial state description using an initial state model generator.

[0096] In step 505, a text target state description is received, which describes the target state of the technical system.

[0097] In 506, a target state model is generated based on the target state description using a target state model generator.

[0098] The domain model generator, the initial state model generator, and the target state model generator are implemented using at least one generative language model (e.g., each using a corresponding generative language model), which processes the domain description, the initial state description, or the target state description.

[0099] During model generation, the domain model generator, the initial state model generator, and the target state model generator exchange feedback about the domain model, the initial state model, and / or the target state model, and adapt the domain model, the initial state model, or the target state model until the domain model, the initial state model, and the target state model are consistent and / or complete (i.e., they collectively yield a consistent and / or complete planning problem model).

[0100] In 507, a (control) plan is created by feeding the domain model, the initial state model, and the target state model (e.g., combined into a planning problem model) to the symbolic planner.

[0101] In 508, the technical system is controlled according to the created plan.

[0102] It should be noted that, Figure 5 The order of steps described is just one example, and other orders are possible. For example, it is also possible to receive all descriptions first and then generate the model simultaneously. It is also possible to switch back and forth between steps, for example, by providing feedback through another step. For example, the domain model could be adapted again if necessary after it has been previously generated.

[0103] Figure 5The method can be executed by one or more computers having one or more data processing units. The term "data processing unit" can be understood as any type of entity that implements the processing of data or signals. For example, it can process data or signals according to at least one (i.e., one or more) specific functions performed by the data processing unit. The data processing unit can include analog circuits, digital circuits, logic circuits, microprocessors, microcontrollers, central processing units (CPUs), graphics processing units (GPUs), digital signal processors (DSPs), programmable gate arrays (FPGAs), integrated circuits, or any combination thereof. Any other means of implementing the corresponding functions described in more detail herein can also be understood as a data processing unit or logic circuit device. One or more of the method steps described in detail herein can be performed (e.g., implemented) by the data processing unit through one or more specific functions performed by the data processing unit.

[0104] Therefore, depending on the implementation scheme, this method is particularly computer-implemented. The method can be executed automatically.

[0105] The provided control method (or control system or planning system designed based on such control method, in particular planning scheme) can be applied to planning problems in different fields that may require dynamic and adaptable planning (and thus to control problems).

[0106] according to Figure 5 The control method can operate with different data (in addition to textual descriptions), such data being derived, for example, from the following sensor signals: • Digital image data, including video, radar, LiDAR, ultrasound, motion, and thermal imaging. • Audio signals used for task planning or environmental prompts via voice control.

[0107] according to Figure 5 The method can be used in control systems or autonomous systems to create and refine plans in response to changing environmental conditions, user-specific constraints, and user-specific objectives, and to extract structured task representations from unstructured natural language input.

[0108] Examples of application scenarios include: • Automatically create and execute assembly processes during manufacturing. • Task scheduling for assistive robots used in healthcare facilities or personalized care. Generate sequential tasks to satisfy user requests.

[0109] According to different implementation methods, Figure 5The method is used for long-term task planning, where intelligent systems must generate high-level sequences of actions to achieve complex, multi-level goals. For example, the method focuses on goal-oriented planning rather than motion planning, making it suitable for scenarios requiring structured thinking for long tasks, such as preparing food in a kitchen, performing multi-level assembly processes in manufacturing, or executing long-span workflows in healthcare. Depending on the implementation, the strengths of Large Language Models (LLMs) and classic symbolic planners are combined. Traditional symbolic planners can create optimal task plans but require manually created domain models and problem descriptions. These planners are based on static environments and deterministic execution, making them unsuitable for dynamic real-world tasks where goals or initial conditions may change. LLMs, on the other hand, are characterized by handling unstructured natural language input and generating structured outputs, but struggle with domain-specific reasoning, adaptability, and consistency. Depending on the implementation, the method combines the strengths of Large Language Models (LLMs) and classic symbolic planners. Figure 5 The method integrates LLM and symbolic planners into a unified multi-agent framework, in which each agent specializes in a specific task. The modularity of the tasks performed by the agents is extremely flexible and adaptable to the complexity of planning problems. In one configuration, the framework can consist of a small number of agents; for example, one agent creates a domain model (if it does not already exist), and another agent defines the problem or task formulation (i.e., the planning problem model) based on that domain model. In more complex scenarios, the task can be subdivided into subtasks (and then, for example, planning problem models are created for each subtask). This planning problem model is processed by the symbolic planner, i.e., planning actions, and then controlling the corresponding technical system so that these planned actions are executed.

[0110] Depending on the implementation, multiple agents collaborate to ensure that the task problem (especially the planning problem) is consistently defined and coordinated in relation to the required context. Agents iteratively communicate and refine their results through feedback loops, thereby ensuring consistency and adaptability to evolving scenarios or user constraints. The planning scheme is particularly useful for domains requiring structured, long-term decision-making rather than real-time motion planning, such as: • Home robots: For example, planning complete cooking sequences (e.g., making sandwiches, setting the table, emptying the dishwasher). • Industrial automation: For example, long-term task planning in manufacturing (e.g., assembling multiple parts over time). • Healthcare support: For example, implementing multi-level patient care routines, such as dispensing medications and meals. • Autonomous service robots: For example, hotel or restaurant assistants that plan continuous service tasks (e.g., preparation, serving, and post-meal cleanup). • Autonomous driving: Making complex, multi-level decisions in autonomous driving. • Autonomous flight: For example, multi-level decision-making in drone delivery (e.g., coordinating landings in densely populated urban areas).

[0111] Depending on the implementation method Figure 5 This method leverages the fundamental principles of both Natural Language Processing (NLP) and symbolic programming to overcome the limitations of traditional programming schemes. Recent advances in NLP and Large Language Models (LLMs) enable a deeper understanding of natural language and other text- or image-based representations, and can be viewed as... Figure 5 The basis for the planning scheme.

[0112] LLM provides the ability for semantic reasoning and natural language understanding. LLM uses a pre-trained transformer-based architecture to map unstructured inputs to structured outputs. Figure 5 The method builds upon this capability by generating LLM-generated outputs using appropriate prompts, the outputs being consistent with the symbolic representation of the planning model (such as PDDL, Python code, or JSON).

[0113] The prompt words guide LLM: generating modular and consistent planning components (of the planning model) that conform to the syntax and semantic requirements of the symbolic planner and ensure coordination and accuracy.

[0114] The unified planning framework provides a formal system for defining planning problems as a set of states, actions, and transitions. Depending on the implementation, Figure 5 The method bridges the gap between formal systems and unstructured natural language descriptions by (e.g., dynamically) generating modular components (domain model, initial state, target state) of the planning problem model. This modularity enables a symbolic planner to process the planning problem model, working effectively in an open-world environment and dynamically adapting the plan to new constraints, objectives, or environmental changes.

[0115] Depending on the implementation method, training data for planning algorithms can also be generated. This includes, for example, developing synthetic scenarios for co-training LLM and symbolic planners, and / or testing the adaptability and robustness of planning systems under various and challenging conditions.

[0116] Figure 5The processing methods are (ultimately) used to generate control signals for robotic devices, particularly in environments with humans (or environments designed for humans, or "human-centered" environments), and for robotic devices that need to adapt their behavior to constantly changing contexts. The term "robotic device" can be understood to refer to any technical system (especially one with mechanically controlled components), such as computer-controlled machines, vehicles, automated equipment, for example, household appliances, power tools, manufacturing machines, personal assistants, or access control systems.

Claims

1. A method for controlling a technology system (101, 311), the method comprising: Receive (501) a text field description (304) that describes an action that can be executed by the technical system (101, 311) and change the state characteristics of the technical system (101, 311); A domain model (502) is generated based on the domain description (304) using a domain model generator (301); Receive (503) a text initial state description (305), the text initial state description describing the initial state of the technical system (101, 311); An initial state model (504) is generated based on the initial state description (305) using an initial state model generator (302); Receive (505) a text target state description (306), the text target state description describing the target state of the technical system (101, 311); A target state model (506) is generated based on the target state description (306) using a target state model generator (303); in, The domain model generator (301), the initial state model generator (302), and the target state model generator (303) are implemented by means of at least one generative language model, which processes the domain description (304), the initial state description (305), or the target state description (306). The domain model generator (301), the initial state model generator (302), and the target state model generator (303) exchange feedback about the domain model, the initial state model, and / or the target state model when generating the domain model, the initial state model, and / or the target state model, and adapt the domain model, the initial state model, or the target state model until the domain model, the initial state model, and the target state model are consistent and / or complete. A plan (507) is created by means of the symbolic planner (310) by feeding the domain model, the initial state model, and the target state model to the symbolic planner (310); and The technical system (101, 311) is based on the created plan control (508).

2. The method according to claim 1, wherein, The initial state model generator (302) checks whether the state characteristics specified in the initial state description (305) are included in the domain model, and in response to the initial state model generator confirming that one or more state characteristics specified in the initial state description (305) are not included in the domain model, it requests the domain model generator (301) to supplement the domain model with the one or more state characteristics.

3. The method according to claim 1 or 2, wherein, The target state model generator (303) checks whether the state characteristics specified by the target state description (306) are included in the domain model, and in response to the target state model generator confirming that one or more state characteristics specified by the target state description (306) are not included in the domain model, requests the domain model generator (301) to supplement the domain model with the one or more state characteristics.

4. The method according to any one of claims 1 to 3, wherein, The target state model generator (303) checks whether the action that enables the journey from the initial state to the target state is included in the domain model, and, in response to the target state model generator confirming that one or more actions that enable the journey from the initial state to the target state are not included in the domain model, requests the domain model generator (301) to supplement the domain model with the one or more actions.

5. The method according to any one of claims 1 to 4, wherein, The target state model generator (303) checks whether the state characteristics specified by the target state description (306) are included in the initial state model, and, in response to the target state model generator confirming that one or more state characteristics specified by the target state description (306) are not included in the initial state model, requests the initial state model generator (302) to supplement the initial state model with the one or more state characteristics.

6. The method according to any one of claims 1 to 5, wherein, The domain model contains variables, and each variable can have multiple values ​​and describe the state characteristics of the technical system (101, 311).

7. The method according to claim 6, wherein, The initial state model generator (302) generates the initial state model such that the initial state model determines the values ​​of at least a portion of the variables based on the initial state description (305).

8. The method according to claim 6 or 7, wherein, The target state model generator (303) generates the target state model, such that the target state model confirms the values ​​of at least a portion of the variables based on the target state description (306).

9. The method according to any one of claims 1 to 8, wherein, The domain description (304), the initial state description (305), and the target state description (306) are fed to the at least one generative language model, with the requirement to create a planning problem model or a component for the symbolic planner (310).

10. A control device (106) designed to perform the method according to any one of claims 1 to 9.

11. A computer program having instructions that, when executed by a processor, cause the processor to perform the method according to any one of claims 1 to 9.

12. A computer-readable medium storing instructions that, when executed by a processor, cause the processor to perform the method according to any one of claims 1 to 9.