Mobile device, apparatus, data carrier, computer program, and method for planning a trajectory for a mobile device

Abstract

Mobile device, apparatus, data carrier, computer program, and method for planning a trajectory for a mobile device. The method comprises obtaining a natural language instruction indicating a path for

Description

Embodiments of the present disclosure relate to a mobile device, an apparatus, a data carrier, a computer program, and a method for planning a trajectory for a mobile device. Translating natural language into linear temporal logic (LTL) plays a crucial role in controlling mobile devices for technical applications because it bridges human instructions and machine-executable commands. Mobile devices, such as autonomous robots, vehicles, or drones, often operate in dynamic environments requiring precise, time-sensitive behaviors. Natural language is inherently ambiguous and lacks the structure needed for machines to interpret instructions reliably. LTL, a formalism for representing temporal sequences of actions, addresses this challenge by providing a clear, mathematical structure to specify how a system should behave over time. By translating natural language into LTL, human operators can define complex tasks for mobile devices, including movement patterns, obstacle avoidance, or goal completion, using familiar language. This translation ensures that these instructions are executed in a predictable and verifiable manner, which is essential for safety-critical applications like autonomous navigation. So, in some applications it may be desired to translate the unstructured natural language instructions for autonomous agents (such as robots for personal use or industrial setup) into machine understandable LTL. Then this LTL can be used for motion planning of agents. However, some approaches require a fine-tuning dataset, which is many times not feasible in the domain of (custom) robotics. Other approaches lack sufficient accuracy or reliability. Some approaches use neural network-based models and large language models (LLMs) to translate specifications into LTL formulas. However, such approaches use synthetic or real data with intermediate translations to generate the LTL formula. This makes them less applicable in low-data scenarios and requires additional data pre-processing. Hence, there may be a demand for an improved concept of a formalization of instructions for planning a trajectory for a mobile device. This demand may be satisfied by the subject-matter of the appended independent claims. Optional embodiments are disclosed by the appended dependent claims. One idea of the proposed approach is to use large language model and semantic role labeling for translating natural language to LTLs. Optionally, embodiments increase the robustness of a generated translation using a modal checking tool. As a result, the proposed approach does not rely on intermediate translations or fine-tuning of the model. Embodiments of the present disclosure provide a method for planning a trajectory for a mobile device. The method comprises obtaining a natural language instruction indicating a path for the mobile device, assigning semantic role labels to phrases of the natural language instruction, and obtaining a linear temporal logic (LTL) based on the natural language instruction and the semantic role labels using a large language model. In doing so, the semantic role labels provide additional information such that a decrease in accuracy in absence of fine-tuning dataset or data pre-processing may be compensated at least partly. At the same time, the reliability and accuracy of results for the LTL may be improved in comparison with other approaches. In practice, the natural language instruction may include a user prompt. In some embodiments, assigning semantic role labels comprises applying a semantic role labeler to the natural language instruction. In practice, e.g., VerbNet, PropBank, or FrameNet may be used for semantic role identification. The semantic role labels may include one or more atomic propositions, a label indicative of a destination, a task, an action for the mobile device, and / or a label indicating a way or an intermediate goal on the way to the destination. In doing so, e.g., location A may be labeled as destination while another location B may be labeled as an intermediate goal on the way to location A. The method may further include verifying a correctness and synthesizability of the LTL. In doing so, the reliability of the results may be improved. Further, the proposed approach may include a self-consistency check. Accordingly, the method may further comprise generating at least one further LTL based on the natural language instruction and the semantic role labels and checking the LTLs for consistency. To this end, the LTLs may be compared to determine if they are consistent or if they deviate from each other. In the first case, the LTLs may be considered consistent and inconsistent otherwise. If they are not consistent, measures may be taken to avoid undesired behavior. For example, an error message may be issued. In some embodiments, the method further comprises presenting the LTL to a user. In doing so, the transparency may be improved. Also, this enables interaction with the user. In some embodiments, the method further comprises generating a trajectory for the mobile device based on the LTL and maneuvering the mobile device based on the trajectory. In practice, the mobile device may include or correspond to a mobile robot or a vehicle. Further embodiments provide a computer program comprising instructions which, when the computer program is executed by a computer, cause the computer to carry out the method of any one of the proposed. Still further embodiments provide a computer-readable data carrier having stored thereon the proposed computer program. Some embodiments provide an apparatus comprising one or more interfaces for communication and a data processing circuit configured to execute an embodiment of the proposed method. Further embodiments provide a mobile device comprising the proposed apparatus. Further, embodiments are now described with reference to the attached drawings. It should be noted that the embodiments illustrated by the referenced drawings show merely optional embodiments as an example and that the scope of the present disclosure is by no means limited to the embodiments presented: Brief description of the drawings Fig. 1 shows a flow chart schematically illustrating an embodiment of a method for planning a trajectory for a mobile device; Fig. 2 schematically shows a use case of the proposed approach; Fig. 3 shows a flow chart schematically illustrating the proposed translation from natural language to LTLs; Fig. 4 exemplarily shows the semantic role labeling and translation for prompt file creation; and Fig. 5 shows a block diagram schematically illustrating an embodiment of an apparatus according to the proposed approach. It should be understood that various modifications and variations may be made without departing from the scope of the present disclosure. Accordingly, the detailed description and specific examples provided herein are intended to be illustrative rather than limiting. Where appropriate, features of different embodiments may be combined or substituted to form further embodiments. The figures and examples are not necessarily drawn to scale, and like numerals in different figures may refer to similar elements. It is to be noted that the present disclosure can be practiced in the absence of any element not specifically disclosed herein unless otherwise indicated. According to some examples, the proposed approach may be implemented as a method. In practice, the proposed method may be implemented by a computer. So, embodiments of the proposed method may be also understood as computer-implemented method. In practice, the method may include multiple steps. A skilled person having benefit from the present disclosure will appreciate that, where feasible, some method steps may be executed in parallel or sequentially, and that method steps may be executed in a different order when explained herein. Fig. 1 shows a flow chart schematically illustrating an embodiment of a method 100 for planning a trajectory for a mobile device. As can be seen from the flow chart, method 100 comprises obtaining 110 a natural language instruction indicating a path for the mobile device. Natural language, in context of the present disclosure, may be understood as the way humans communicate through spoken or written language, including everyday vocabulary, grammar, and structure. The natural language instruction, e.g., may be obtained from any source of natural language and / or through human-machine interaction, e.g., through a user or human-machine-interface interface. So, in practice, the natural language instruction may be indicative or include a user prompt. Method 100 further includes assigning 120 semantic role labels to phrases of the natural language instruction. The semantic role labels, e.g., indicate a meaning of the phrases in context of the natural language instruction. In practice, the labels may distinguish between different actions and / or a meaning of locations for the trajectory planning. For example, the labels may distinguish between destinations and waypoints on the way to the destination. For this, a semantic role labeler may be applied to the natural language instruction. Further, the semantic role labels may include one or more atomic propositions indicating a sequence of actions or locations. Further, method 100 comprises obtaining 130 an LTL based on the natural language instruction and the semantic role labels using a large language model. For this, the large language model may be applied to the natural language instruction labeled with the semantic role labels. The LTL may be that provided for trajectory planning. For this, a finite-state automaton or graph which represents the LTL in line with possible states and transitions of the mobile device may be generated. Further details and aspects are now laid out in more detail below with reference to Figs. 2 to 4. Fig. 2 schematically shows a use case of the proposed approach. In the present use case, e.g., a user 200 provides a natural language instruction. In this example, the user 200 utters the natural language instruction in a spoken form. In this case, the natural language instruction may be recorded (e.g., using a microphone or the like) and translated using a speech-to-text module 210 to obtain a textual representation 220 of the natural language instruction. The natural language instruction may be been provided to a formalization module 230 for labeling the natural language instruction in line with the proposed approach. Further details about the labeling are described now in more detail below with reference to Fig. 3. The textual representation of the natural language instruction, here numbered 310, e.g., reads “enter a first location (e.g., yellow room) via a second location (e.g., green)”. The natural language instruction 310 is provided to a semantic role labeling module 324 labeling (also referred to herein as semantic role labeling (SRL)) the natural language instruction. SRL, e.g., utilizes VerbNet or any other semantic role labeler for semantic role identification. This, assigns thematic roles to phrases of the natural language instruction, aiding in detecting atomic propositions (APs) for LTL formula generation. This allows to integrate the semantic role labels (numbered 330) (as a soft constraint in the overall prompt for the Language Model (LLM). In step 340, the semantic role labels 330 are included in input for a large language model 350 together with the natural language instruction 310, one or more few-shot chain-of-thought (CoT) prompts, and instructions for CoT reasoning as well as a list of acceptable atomic propositions and allowed operators for the linear temporal logic. In a modular framework, the input may be assembled in a so-called prompt construction module. Accordingly, embodiments of the proposed approach may be referred to as “CoT-TL”. Chain-of-Thought (CoT) Prompting: o Extends traditional CoT reasoning specifically for formal logic generation. o Breaks down complex navigation goals into subgoals, focusing on individual steps to reduce errors. o Uses a systematic approach based on divide-and-conquer methodology to mitigate ambiguity and address compositional generalization. Few-shot Prompting: o Employs carefully designed prompts with a small number of examples (6 in the present case but can be tailored for the individual use case, the present approach applies 1 example for each unique LTL structure) to guide the translation process. o Leverages the intrinsic in-context learning capability of LLMs for LTL translation. Further details and aspects of the proposed labeling are now described below with reference to Fig. 4 schematically illustrating an example of the proposed labeling process. As above, according to the natural language instruction the first location (here, e.g., “blue room”) should be entered via a second location (here, e.g., “red room”). According to embodiments of the proposed approach, one or more subgoals may be extracted from the natural language instruction. In doing so, phrases indicating different actions and / or locations for the mobile device are extracted. Subsequently, the phrases are labeled in accordance with their meaning / semantic. In the present example, “Enter” is identified as action (“[arg 0]”), the first location is identified as destination and the second location as a path. An exemplary list of atomic propositions is: [green_room, red_room, blue_room, first_floor]. So, based on the allowed list of APs, the APs for the present natural language instruction are: blue_room, red_room. Also, allowed temporal and logical operators may be predetermined, e.g., [&, |, !, G, F, U], wherein G means Globally / Always, F means Eventually, X means Next, U means Until &means AND, | means OR, and ! means NOT. The labeled natural language instruction is, then, provided (e.g., as prompt) to the large language model 350 together with the other input as laid out in more detail above. The large language model 350 may use pre-trained LLMs (such as GPT-4, GPT-3, Mistral-7B, or Starcoder) as a core translation engine. When applied to the input, the large language model 350 processes the natural language instruction along with the semantic role labels 330 and few-shot CoT prompts to generate an LTL 240 / 360 in accordance with the natural language instruction. In doing so, the large language model 350 considers the role of the phrases based on the semantic role labels. In doing so for the present example, the large language model 350 recognizes that both the mobile device should move to the first and the second location (see Fig. 4), wherein it needs to move to the second location before it moves to the first location. Accordingly, the large language model 350 links the phrases by appropriate allowed logic operators in accordance with the given few-shot CoT examples and the allowed APs such that the obtained LTL 240 / 360 may be “F(red_room &F(blue_room))”. The LTL 240 / 360 is then provided to a syntax check 370 to ensure that the generated LTL 240 / 360 is synthesizable and correct. For this, e.g., automaton checking (e.g., SPOT library) may be used. In doing so, feedback to the LLM is provided for regeneration if the output is invalid. For this, e.g., an error message is provided to the large language model 350 if the LTL 240 / 360 is invalid. The present example further incorporates a self-consistency check 380. This may include generating multiple independent chains of thought for the same natural language instruction and using majority voting to select one of the LTLs resulting from the different chains of thought for the trajectory planning to improve robustness. As can be seen from Fig. 3, the LTL translation may be implemented as an End-to-End Pipeline which Integrates the described components into a cohesive pipeline that takes natural language specifications as input and produces valid, synthesizable LTL(s) (formulas) as output and operates without the need for fine-tuning or large datasets, making it suitable for low-resource scenarios. The LTL 240 / 360 is then provided to a trajectory planner 250 (see step 390) which generates a trajectory for a mobile device in accordance with the obtained LTL 240 / 360. The present case, the trajectory planner, e.g., generates a trajectory which passes the second location and ends at the first location. For providing the obtained LTL 240 / 360 an Output Module may be used which presents the final, verified LTL formula (to a user) and / or optionally provides reasoning steps (CoT) for interpretability. The mobile device, in the context of the present disclosure, may be understood as a self-contained apparatus capable of independent movement and navigation without direct human control. It may utilize onboard sensors, processors, and actuators to perceive its environment, process information, and execute real-time decisions. Such device can operate in various environments by dynamically adapting to obstacles, terrain, and other external conditions through integrated systems like GPS, cameras, LiDAR, radar, and artificial intelligence. It may be programmed for specific tasks, such as transporting goods, performing inspections, or providing services, and can communicate wirelessly with external systems for updates or coordination. The mobile device may be powered by a self-sustaining energy source, such as a battery or fuel, enabling continuous operation within its designed parameters. In practice, the mobile device, e.g., may correspond to an autonomously driving vehicle, a robotic system, and / or pick and place robots. Optionally, the proposed pipeline may include an Integration Interface which is configured for integration with autonomous agent planning systems, such as robotic system, pick and place robots, autonomous driving. So, in summary, embodiments may implement the following workflow: 1. The natural language instruction / specification is input (to a system according to the proposed approach). 2. The SRL module processes the input and extracts semantic information, i.e., it labels phrases of the natural language instruction. 3. A prompt construction module assembles the input (including the SRL, allowed APs, allowed logic operators, and few-shot prompt examples) for the LLM. 4. The LLM (processing module) generates an (initial) LTL (translation). 5. A model checking module verifies the output, requesting regeneration if necessary. 6. Steps 4-5 may be repeated for a self-consistency check (e.g., 3 times). 7. The final, verified LTL formula is output and can be used for autonomous agent planning. So, a could be shown, the proposed approach provides a training-free framework for translating natural language specifications into LTL for autonomous agents. Further, In particular, embodiments of the proposed approach provide one or more of the following advantages: 1. Data Efficiency: • Achieves sufficient accuracy using less data samples, compared to methods requiring large fine-tuning datasets. • Enables quick adaptation to new domains or tasks without extensive data collection. 2. No Fine-tuning Required: • Operates as a training-free method, eliminating the need for costly and time-consuming model fine-tuning. Improved Accuracy: • Outperforms existing approaches, including fine-tuned models, across multiple datasets. • Demonstrates effectiveness in handling complex LTL structures and formulas. Interpretability: • Provides clear and thorough rationale behind LTL generation through the CoT process. • Enhances user trust and understanding of the translation process. Generalization: • Shows strong performance across diverse datasets and unseen LTL structures. • Adapts well to different autonomous agent planning scenarios. Reliability: • Incorporates model checking to ensure synthesizable and correct LTL outputs. • Uses self-consistency checks to reduce errors and improve robustness. Flexibility: • Compatible with various pre-trained language models, allowing for easy updates as newer models become available. Efficiency in Low-resource Scenarios: • Particularly effective in domains or applications where large annotated datasets are not available. No Intermediate Translations: • Directly translates from natural language to LTL without the need for intermediate representations or preprocessing steps. Practical Applicability: • Can be used for robotics and automated domain such as robots for personal use, industrial use case. As the skilled person will appreciate, the proposed approach may be implemented in various apparatuses, as laid out below in more detail with reference to Fig. 5. Fig. 5 shows a block diagram schematically illustrating an embodiment of such an apparatus 500. The apparatus comprises one or more interfaces 510 for communication and a data processing circuit 520 configured to execute the proposed method. In embodiments, the one or more interfaces 510 may comprise wired and / or wireless interfaces for transmitting and / or receiving communication signals in connection with the execution of the proposed concept. In practice, the interfaces, e.g., comprise pins, wires, antennas, and / or the like. As well, the interfaces may comprise means for (analog and / or digital) signal or data processing in connection with the communication, e.g., filters, samples, analog-to-digital converters, signal acquisition and / or reconstruction means as well as signal amplifiers, compressors and / or any encryption / decryption means. The data processing circuit 520 may correspond to or comprise any type of programable hardware. So, examples of the data processing circuit 520, e.g., comprise a memory, microcontroller, field programable gate arrays, one or more central, and / or graphical processing units. To execute the proposed method, the data processing circuit 520 may be configured to access or retrieve an appropriate computer program for the execution of the proposed method from a memory of the data processing circuit 520 or a separate memory which is communicatively coupled to the data processing circuit 520. In practice, the proposed apparatus may be installed on a mobile device. So, embodiments may also provide a mobile device comprising the proposed apparatus. In implementations, the apparatus, e.g., is part ora component of the mobile device. However, in implementations, computing resources for the mobile device may be outsourced to an external server separate from the mobile device. In such implementations, the proposed approach may be also implemented outside of the mobile device. In the foregoing description, it can be seen that various features are grouped together in examples for the purpose of streamlining the disclosure. This method of disclosure is not to be interpreted as reflecting an intention that the claimed examples require more features than are expressly recited in each claim. Rather, as the following claims reflect, subject matter may lie in less than all features of a single disclosed example. Thus, the following claims are hereby incorporated into the description, where each claim may stand on its own as a separate example. While each claim may stand on its own as a separate example, it is to be noted that, although a dependent claim may refer in the claims to a specific combination with one or more other claims, other examples may also include a combination of the dependent claim with the subject matter of each other dependent claim or a combination of each feature with other dependent or independent claims. Such combinations are proposed herein unless it is stated that a specific combination is not intended. Furthermore, it is intended to include also features of a claim to any other independent claim even if this claim is not directly made dependent to the independent claim. Although specific embodiments have been illustrated and described herein, it will be appreciated by those of ordinary skill in the art that a variety of alternate and / or equivalent implementations may be substituted for the specific embodiments shown and described without departing from the scope of the present embodiments. This application is intended to cover any adaptations or variations of the specific embodiments discussed herein. Therefore, it is intended that the embodiments be limited only by the claims and the equivalents thereof.

Claims

1. A method (100) for planning a trajectory for a mobile device, the method (100) comprising:obtaining (110) a natural language instruction indicating a path for the mobile device;assigning (120) semantic role labels to phrases of the natural language instruction; andobtaining (130) a linear temporal logic, LTL, based on the natural language instruction and the semantic role labels using a large language model.

2. The method (100) of claim 1, wherein the natural language instruction includes a user prompt.

3. The method (100) of claim 1 or 2, wherein assigning semantic role labels comprises applying a semantic role labeler to the natural language instruction.

4. The method (100) of any one of the preceding claims, wherein the semantic role labels include one or more atomic propositions, a label indicative of a destination, and / or a label indicating a way or an intermediate goal on the way to the destination.

5. The method (100) of any one of the preceding claims, wherein the method (100) further includes verifying a correctness and synthesizability of the LTL.

6. The method (100) of any of the preceding claims, wherein the method (100) further comprises generating at least one further LTL based on the natural language instruction and the semantic role labels and checking the LTLs for consistency.

7. The method (100) of any one of the preceding claims, wherein the method (100) further comprises presenting the LTL to a user.

8. The method (100) of any one of the preceding claims, wherein the method (100) further comprises generating a trajectory for the mobile device based on the LTL and maneuvering the mobile device based on the trajectory.

9. The method (100) of any one of the preceding claims, wherein the mobile device includes a mobile robot or a vehicle.

10. A computer program comprising instructions which, when the computer program is executed by a computer, cause the computer to carry out the method (100) of any one of the claims 1 to 9.

11. A computer-readable data carrier having stored thereon the computer program of claim 10.

12. An apparatus (500) comprising:one or more interfaces (510) for communication; anda data processing circuit (520) configured to execute the method (100) of any one of the claims 1 to 9.

13. A mobile device comprising the apparatus (500) of claim 12.

Images