System and method for mobile robot task planning

Abstract

A technique for enabling a robot to autonomously compute solutions to complete sequential logic tasks, and to complete these solutions at runtime. A task formula is obtained on an airborne computer of the mobile robot. And converting the task formula into an automaton. A motion tree is established using the initial map of the space and the automaton. Sampling-based tree search is performed using the motion tree to generate an initial plan for the mobile robot to operate in space and satisfying a task formula, the initial plan including a trajectory. The mobile robot is then moved along the trajectory.

Description

Cross - Reference to Related Applications

[0001] This application claims the priority of U.S. Provisional Patent Application No. 63 / 450,314, filed on March 6, 2023, with the title "Human - in - the - Loop System for Mobile Robot Task Planning", the entire content of which is incorporated herein by reference for all purposes. Technical Field

[0002] The present disclosure relates to mobile robots, and more particularly to a system and method for task planning of mobile robots. Background Art

[0003] A robot is a machine that can be programmed to perform many basic tasks, such as point - to - point navigation. A robot typically consists of a mechanical body, sensors for detecting its surrounding environment, and a control system that enables it to move and interact with the surrounding environment. The control system can be programmed to process sensor data and make action decisions based on that data. Eventually, it is desired that robots can perform many more advanced tasks, such as home care, fire fighting, or building inspection. These tasks usually have complex temporal and spatial constraints and require causal relationships between actions. Linear Temporal Logic (LTL) can describe such tasks in a mathematically precise way. A hybrid robot controller can compute to satisfy tasks expressed as LTL formulas. Summary of the Invention

[0004] An overview of various embodiments of the present invention is provided in the form of a list of examples below. As used hereinafter, any reference to a series of examples should be understood as a disjunctive reference to each of these examples (e.g., "Examples 1 - 4" should be understood as "Example 1, Example 2, Example 3, or Example 4").

[0005] Example 1 is a method for performing task planning for a mobile robot operating in a space, the method comprising: obtaining a task formula on an on - board computer of the mobile robot; converting the task formula into an automaton; constructing a motion tree using an initial map of the space and the automaton; performing a sampling - based tree search using the motion tree to generate an initial plan for the mobile robot to operate in the space and satisfy the task formula, the initial plan including a trajectory; and causing the mobile robot to move along the trajectory.

[0006] Example 2 is the method of Example 1, further comprising: obtaining a second task formula on the on - board computer; converting the second task formula into a second automaton; updating the motion tree using the second automaton; performing a second sampling - based tree search using the updated motion tree to generate a revised plan for the mobile robot to operate in the space and satisfy the second task formula, the revised plan including a second trajectory; and causing the mobile robot to move along the second trajectory.

[0007] Example 3 is the method of Examples 1-2, further comprising: obtaining an environmental model and a robot model on an on-board computer; and constructing an initial map of the space based on the environmental model and the robot model.

[0008] Example 4 is the method of Examples 1-3, wherein the on-board computer executes a natural language processing (NLP) model for generating a task formula according to an input text.

[0009] Example 5 is the method of Examples 1-4, wherein the automaton is a deterministic finite automaton (DFA).

[0010] Example 6 is the method of Examples 1-5, wherein the task formula is a linear temporal logic (LTL) formula.

[0011] Example 7 is the method of Examples 1-6, wherein the on-board computer includes a control circuit configured to receive a trajectory and generate a control signal to move the mobile robot along the trajectory.

[0012] Example 8 is a non-transitory computer-readable medium that contains instructions which, when executed by one or more processors, cause the one or more processors to perform operations for a mobile robot operating in a space, the operations including: obtaining a task formula on the on-board computer of the mobile robot; converting the task formula into an automaton; constructing a motion tree using the initial map of the space and the automaton; performing a sampling-based tree search using the motion tree to generate an initial plan for the mobile robot to operate in the space and satisfy the task formula, the initial plan including a trajectory; and causing the mobile robot to move along the trajectory.

[0013] Example 9 is the non-transitory computer-readable medium of Example 8, wherein the operations further include: obtaining a second task formula on the on-board computer; converting the second task formula into a second automaton; updating the motion tree using the second automaton; performing a second sampling-based tree search using the updated motion tree to generate a revised plan for the mobile robot to operate in the space and satisfy the second task formula, the revised plan including a second trajectory; and causing the mobile robot to move along the second trajectory.

[0014] Example 10 is the non-transitory computer-readable medium of Examples 8-9, wherein the operations further include: obtaining an environmental model and a robot model on the on-board computer; and constructing an initial map of the space based on the environmental model and the robot model.

[0015] Example 11 is the non-transitory computer-readable medium of Examples 8-10, wherein the on-board computer executes a natural language processing (NLP) model for generating a task formula according to an input text.

[0016] Example 12 is a non - transitory computer - readable medium of Examples 8 - 11, where the automaton is a deterministic finite automaton (DFA).

[0017] Example 13 is a non - transitory computer - readable medium of Examples 8 - 12, where the task formula is a linear temporal logic (LTL) formula.

[0018] Example 14 is a non - transitory computer - readable medium of Examples 8 - 13, where the on - board computer includes control circuitry configured to receive a trajectory and generate a control signal to move a mobile robot along the trajectory.

[0019] Example 15 is a system, including: one or more processors; and a non - transitory computer - readable medium containing instructions that, when executed by the one or more processors, cause the one or more processors to perform operations for a mobile robot operating in a space. The operations include: obtaining a task formula on the mobile robot's on - board computer; converting the task formula into an automaton; constructing a motion tree using an initial map of the space and the automaton; performing a sampling - based tree search using the motion tree to generate an initial plan for the mobile robot to operate in the space and satisfy the task formula, the initial plan including a trajectory; and causing the mobile robot to move along the trajectory.

[0020] Example 16 is the system of Example 15, where the operations further include: obtaining a second task formula on the on - board computer; converting the second task formula into a second automaton; updating the motion tree using the second automaton; performing a second sampling - based tree search using the updated motion tree to generate a revised plan for the mobile robot to operate in the space and satisfy the second task formula, the revised plan including a second trajectory; and causing the mobile robot to move along the second trajectory.

[0021] Example 17 is the system of Examples 15 - 16, where the operations further include: obtaining an environment model and a robot model on the on - board computer; and constructing an initial map of the space based on the environment model and the robot model.

[0022] Example 18 is the system of Examples 15 - 17, where the on - board computer executes a natural language processing (NLP) model to generate a task formula according to input text.

[0023] Example 19 is the system of Examples 15 - 18, where the automaton is a deterministic finite automaton (DFA).

[0024] Example 20 is the system of Examples 15 - 19, where the task formula is a linear temporal logic (LTL) formula. Description of the Drawings

[0025] The accompanying drawings are used to provide a further understanding of the present disclosure, are incorporated into and constitute a part of this specification, show embodiments of the present disclosure, and are used together with the detailed description to explain the principles of the present disclosure. When showing the structural details of the present disclosure, it will not exceed the necessary degree required for its basic understanding and implementation.

[0026] Figure 1 An example structure of a human-in-the-loop robot task system is shown.

[0027] Figure 2 An example of a planning process is shown.

[0028] Figure 3 An example of a graphical user interface (GUI) is shown.

[0029] Figure 4 An example of a state diagram including a set of states is shown.

[0030] Figure 5 An example of a sampling-based motion tree growing in a hybrid space is shown.

[0031] Figure 6 A method for performing task planning for a mobile robot operating in a space is shown.

[0032] Figure 7 An example computer system including various hardware components is shown.

[0033] In the drawings, similar components and / or features may have the same numerical reference marks. In addition, various components of the same type can be distinguished by adding a letter after the reference mark, or by adding a dash and / or a second numerical reference mark after the reference mark, and this second numerical reference mark is used to distinguish similar components. If only the first numerical reference mark is used in the specification, the description applies to any one of the similar components and / or features having the same first numerical reference mark, regardless of its suffix. Detailed Description

[0034] Embodiments of the present invention relate to a robot task planning system and a method for controlling a robot to perform tasks such as navigation, search and rescue, inspection, and maintenance. The planning process can automatically plan the movement and actions of the robot, enabling it to navigate in complex or dangerous terrains, or perform specific tasks such as detecting and identifying potential hazards or inspecting infrastructure. The embodiments can be used for remote monitoring and control of the robot, enabling an operator to adjust the actions of the robot in real time as needed. This helps to improve the efficiency and safety of these tasks and utilize resources more effectively.

[0035] Linear temporal logic (LTL) is a formal logical system used in robotics and other fields to reason about the behavior of a system over time. In robotics, LTL can be used to specify and verify the correctness of robotic systems that operate over time, such as autonomous vehicles or robots that interact with humans. One problem with current LTL planning methods is the presence of uncertainty in the environment, which can invalidate the solutions computed by the robot. Due to the limited on-board computing capabilities of mobile robots, they typically cannot recompute the plan from scratch in real time. To address partially unknown and non-static environments, this paper proposes a sampling-based LTL task planning method that can revise the plan according to real-time events. With this method, users can expect the robot to safely and correctly complete LTL tasks without strict assumptions about the environment.

[0036] Another problem with LTL lies in human-robot interaction. In many cases, non-experts need to spend a significant amount of time getting familiar with the syntax and semantics of LTL. Additionally, users typically require expertise to access the source code to initiate the auxiliary modules of the robot, such as motion planning, localization, and mapping, to facilitate autonomous task execution. These processes are error-prone and cumbersome, resulting in a waste of time and manpower in practical applications. To speed up the development and deployment process, a human-in-the-loop robotic task system is designed in the Robot Operating System (ROS). Through a user-friendly interface, users can monitor the robot's observations and progress and accordingly specify high-level tasks for the robot.

[0037] Previous methods to address the above problems include classical controller synthesis methods, in which the environment is abstracted beforehand and the robot's motion is reduced to symbolic transitions. A discrete plan can be retrieved from a product automaton constructed from the product of a transition system and a task-related automaton. However, most existing methods assume that the environment is static and its finite model can be obtained beforehand. In the real world, the environment model is not always available, and due to changes in behavior patterns, the environment model can even be unpredictable. Directly re-synthesizing the entire plan can have adverse consequences because the execution history is lost, and the revised plan may violate the temporal specifications.

[0038] Different from traditional methods, some examples described in this paper may use a sampling tree to explore solutions in the workspace. Sampling points can be reused to connect historical paths for real-time replanning. By using this technology, the LTL planner can quickly respond to real-time updates of workspace knowledge. In addition, a user interface can be adopted to allow users to easily monitor the progress and intervene in task execution. Traditional systems that include an LTL planner and a user interface cannot store and utilize the information collected by the robot during task execution. After being assigned another task, the system may force the planning process to start over, and all historical data will be cleared. As a result, the exploration and correction of partially unknown workspaces will not be remembered. In contrast, the task planning process described in this paper takes into account the information obtained, thereby improving the overall efficiency of task planning.

[0039] In some examples, the task specification is constrained in the form of linear temporal logic (LTL f ), on a finite trace, which has sufficient expressive power to direct a robot with limited power supply. For each LTL f formula φ, there exists a deterministic finite automaton (DFA, or simply "automaton") that accepts a sequence of input words to satisfy φ. A run of length n on a DFA is a sequence of states that starts from the initial state and follows the transition relation.

[0040] Some embodiments include a mobile robot with sensing capabilities moving in a partially unknown workspace to complete high-level goals. The workspace X consists of a set of regions of interest Π (whose properties are combinations of atomic propositions AP) and a set of regions X occ occupied by fixed obstacles. The robot takes the workspace model X init as prior knowledge. It is assumed that X init is sufficient for the robot to compute a trajectory that satisfies the specification φ within the obstacle-free region X free . Since the pre-provided information is incomplete, it is not guaranteed to satisfy the specification because the actual region properties may be inconsistent with the prior knowledge. The workspace can be updated by receiving data from on-board sensors or external observers.

[0041] Figure 1Shows an example structure of a human-in-the-loop robotic task system 100 (or simply "system 100") according to some embodiments of the present disclosure. System 100 may correspond to or be integrated into a robot that includes mechanical and electrical components and can perform tasks autonomously or semi-autonomously. A user 102 of system 100 may speak an instruction statement. A microphone 162 of an input device 166 converts the speech statement into a speech signal 104, and the speech signal 104 is converted by a speech-to-text converter 106 into a task 108 that includes text. The speech-to-text converter 106 may utilize an online speech-to-text service that employs a machine learning model. Alternatively or additionally, the input device 166 may include a keyboard 164, and the user 102 may directly provide the task 108 to an on-board computer 130 through the keyboard 164.

[0042] The task 108 may include one or more instruction statements. These instruction statements are input into a hardware accelerator 112 of the on-board computer 130. The hardware accelerator 112 may be a dedicated processor for executing a natural language processing (NLP) model 110 that can infer a potential task formula 114 (such as an LTL formula) from the instruction statements. The task formula 114 is input into a planning circuit 120 of the on-board computer 130. The planning circuit 120 also receives sensor data 176 from one or more sensors 174 of the system 100, as well as an environment model 122 and a robot model 124. Based on these inputs, the planning circuit 120 may execute a planning process 150 to generate an initial map X of the environment init , an automaton 118 (i.e., DFA), a motion tree 160, an initial plan 156, and / or a revised plan 158 ( Figure 1 not shown in). In some cases, the generated plan can be used to construct a trajectory 170, which is sent to a control circuit 140 to cause the robot to perform mechanical motion. For example, the control circuit 140 may generate a control signal 132 to cause one or more actuators of the robot to move (such as rotating wheels, extending or contracting legs, etc.).

[0043] The mission formula 114 can be converted into an automaton using various tools, such as ltl2dfa or ltl2ba. With the automaton, the on-board computer 130 can plan a series of actions or movements that satisfy the acceptance conditions of the automaton. Then, the robot executes the plan according to its dynamic constraints, such as moving along the trajectory 170. The system 100 may include an output device 168 through which the system 100 provides the execution status 128 to the user 102. The execution status 128 may come from the planning circuit 120 and / or the control circuit 140, and may include information about various data generated by the planning process 150. The output device 168 may include a display screen (such as an LCD or LED panel), an indicator light, an audio output device (such as a speaker or a buzzer), etc.

[0044] During execution, due to a certain degree of uncertainty (such as moving obstacles and incorrect prior knowledge), the planning process 150 may revise or improve the current plan. In some examples, the user 102 can adjust the planning process 150 using the feedback execution information when the robot is online. The user 102 can use the input device 166 to pause, resume, abort the task, or assign a new task, while the output device 168 provides an update of the execution status 128 to the user 102. In one example, the output device 168 may include a screen-based display to show the execution status 128 in the form of an overhead map containing the trajectory 170. The user 102 can view the trajectory 170 and decide to speak into the microphone 162 to provide a new task 108 to change the trajectory 170.

[0045] Figure 2 An example of the planning process 150 according to some embodiments of the present disclosure is shown. In the example shown, the planning process 150 includes two phases: an initialization planning phase 142 and a reactive planning phase 144. Before executing the planning process 150, the system obtains the environment model 122, the robot model 124, and the mission formula 114. In the initialization planning phase 142, the planning process 150 may perform a knowledge base construction 101 to construct an initial map X according to the environment model 122 and the robot model 124 init . The initialization planning phase 142 may also include a step 103 of converting the mission formula 114 into an automaton 118.

[0046] Using the initial map X init and the automaton 118, the sampling-based motion tree 160 grows step by step with dynamically compatible edges. In some examples, the LTL mission formula φ is converted into a DFA and further into a state monitor to guide the expansion of the sampling tree in the sampling-based tree search 105. In some examples, based on the marked region Π and the obstacle X occThe knowledge base of the environmental model construction workspace. In some examples, the knowledge base serves as the initial map X in the initial planning init to obtain an initial solution whose trajectory satisfies the task formula φ.

[0047] In the reactive planning phase 144, once a solution is successfully obtained, the robot can safely track the nominal path by generating a trajectory 170 during trajectory generation 107. During execution, the sampling-based tree search 105 continuously searches for unique solutions and improves existing solutions. Meanwhile, the robot uses the observations 172 generated during observation generation 113, as well as the environmental state 152 and the robot state 154 generated during state estimation 115 for real-time replanning 109. Both observation generation 113 and state estimation 115 may rely on sensor data 176 from external observers and its on-board sensors. The robot uses the updated data 178 to revise the corresponding parts of its knowledge base. In some examples, state estimation 115 assists the planning process 150 by tracking the execution progress and updating the continuous state x of the robot (i.e., the robot state 154). The evolution of both the environmental state 152 and the robot state 154 can trigger a replanning mechanism to handle uncertainties in the domain.

[0048] Figure 3 Shows an example of a graphical user interface (GUI) 180 according to some embodiments of the present disclosure. The GUI 180 can implement one or both functions of the output device 168 and the input device 166. The GUI 180 can assist the user with system preference settings, instruction transmission, and graphical visualization. To make the system scalable in different scenarios, in some examples, a robot configuration panel is provided to the user before task assignment. First, if there are available predefined maps, the user can select from a combo box. These maps may have different formats for task and motion planning. For example, the task planner accepts a configuration file containing propositions marked in polygons, while motion planning may require a more refined map representation, such as a point cloud map. The maps can be loaded silently in the background.

[0049] Then, the user can select the robot type according to their preferences and hardware availability. After that, on-board sensors can be selected for sensing and positioning purposes. In some cases, various options of the planning and sensing modules are filtered, and the user can select from valid autonomous navigation algorithms. Once the confirmation button is clicked, the ground control station (GCS) calls a service to start the plug-in deployed on the vehicle side. Application modules such as positioning and motion planning are launched to achieve safe point-to-point movement.

[0050] After the basic modules are ready, the user can specify high-level tasks for the robot. Although Linear Temporal Logic (LTL) provides a way to instruct the robot without programming, specifying commands as LTL formulas may not be intuitive. The task panel is designed for users with different levels of background knowledge. Expert users familiar with LTL syntax can directly express tasks using LTL formulas. For non-experts, the Graphical User Interface (GUI) 180 provides a natural language input interface. The user can speak or input task specifications, and the system can convert these commands into LTL formulas. In some examples, the speech recognition service (speech-to-text converter 106) converts audio to text, and the Natural Language Processing network (NLP model 110) converts natural language to an LTL formula (task formula 114). The remaining space on the GUI 180 can be used to provide feedback information to the user. The "Plan Framework" text box shows the planned path by displaying the corresponding trajectory. Other notifications indicating the execution status are printed in the "Execution Status" text box.

[0051] Figure 4 Shows an example of a state diagram 182 according to some embodiments of the present disclosure, which includes a set of states 184. In some embodiments, the behavior of the robot can be modeled as a hybrid system. At startup, the robot is in the idle state 184-1 and waits for a task. In some cases, it is assumed that the propositions in the task formula only contain the labels of the regions of interest in the workspace. Once the robot receives the task formula, it enters the planning / replanning state 184-2, in which the LTL path planner grows a randomly explored motion tree 160 to search for a solution that satisfies the specification. Once a feasible solution is found, the robot enters the execution state 184-3 and starts moving along the nominal path while refining the unfinished parts of the plan.

[0052] During execution, if an unexpected obstacle is detected or an error in the knowledge base invalidates the calculated solution, the planning process may switch to the planning replanning mode, and if no valid solution exists, the vehicle will be braked. In cases where the solution is not ideal or leads to danger, the user can stop the task planning and execution by sending an abort signal to the interface. In some cases, if a new task is assigned after the previous task is completed, a task switching program is applied. To adapt to the newly given task, this program updates the current sampling tree so that the transition of each edge in the tree is valid in the new DFA.

[0053] Figure 5 Shows an example of a sampling-based motion tree 160 grown in a hybrid space according to some embodiments of the present disclosure. When constructing the initial map X init and translate LTL fAfter the specification, the robot can construct a motion tree (motion tree) towards the temporal goal. In some cases, the robot may not wait until a near-optimal solution is computed before execution. Instead, once a feasible solution is found, the robot can start moving along the nominal path using the trajectory generator.

[0054] In some examples, the robot may employ sampling-based tree search monitored by an automaton to find a solution. The motion tree 160 may consist of a set of nodes (containing multiple nodes) and a set of edges (containing multiple edges). Additionally, a graph is maintained that contains not only the motion tree but also a set of isolated vertices. Let the set of automaton states be Q and the set of workspace states be X. All states with the same automaton state lie on the same plane. A node (x, q) can be formed by sampling the workspace state x and the automaton state q. If the semantic label λ of x' causes a transition in the DFA, the formed node can be connected to another node (x', q') in the tree. In other words, if there is an edge q' → (λ) → q in the DFA, then the node (x, q) is added to the tree T. An arrival cost value (the cumulative distance from the root of the tree to this state) is assigned to the newly added node in the tree. States that are not added to the tree are added to the graph, and they can be added to the tree in the future by reconnection. The planning process checks whether the newly sampled node can lead to an accepting state in one step. If so, a new solution is determined accordingly.

[0055] Considering the events that may occur in the real world, the robot may need to turn around and pass through visited nodes to satisfy the temporal logic task. During the execution of the plan, the edges that have been traversed remain valid by changing the root node as the robot moves. During the movement of the robot, the classical tree transforms into a double-root tree instead of setting the next unvisited node as the root node. As a result, all nodes in the tree T are updated to automaton states that can be reached in the future, and the arrival cost value of each node is less than or equal to the original value.

[0056] In practical applications, the prior knowledge may be inaccurate, resulting in invalid acceptance conditions. To solve this problem, the robot is able to sense the environment and receive observations from other agents. The mobile robot is considered to be moving in a scenario where the regional attributes may change and unexpected obstacles may block the path. The robot may continuously analyze the sensor data and extract knowledge in two forms: κ reg = (π, L(π)) or κ obs = (υ, x υ )), where υ is an unexpected obstacle instance and x υis the state of υ. If the knowledge stored in the memory is found to be inconsistent with the newly acquired knowledge, the robot will update the knowledge base and make any necessary revisions to the plan.

[0057] Each time κ is obtained reg , the label of region π is updated. The planning process first corrects the states of the nodes in the entire tree T according to the updated workspace, and then solves the state repetition problem. After that, each possible solution is checked to see if its corresponding trajectory satisfies φ. If there is still a feasible solution, the solution with the lowest cumulative cost is set as the current plan. Otherwise, if the knowledge change is critical and all previous solutions are infeasible, the robot will stop completely at x stop . The new root node is set to s stop =(q stop , x stop ). Then the robot waits at x stop until the planning process obtains a new solution.

[0058] Given the knowledge κ obs , if the obstacle υ has not been observed before, or the travel distance Δx υ exceeds the threshold, the planning process updates X free and triggers the replanning procedure. The planning process first blocks the edges passing through υ by setting their weights to positive infinity. After that, similar to the case of processing κ reg , the robot will execute the best executable solution or wait for a feasible solution to be generated by the sampling-based search.

[0059] Figure 6 FIG. shows a method for performing task planning for a mobile robot operating in a space according to some embodiments of the present disclosure. The steps of method 600 can be executed in any order and / or in parallel, and one or more steps of method 600 can be selectively executed. One or more steps of method 600 can be executed by one or more processors, such as those included in the on-board computer 130. Method 600 can be implemented as a computer-readable medium or a computer program product that includes instructions that, when executed by one or more processors, cause the one or more processors to execute the steps of method 600.

[0060] In step 602, a task formula (such as task formula 114) is obtained on the on-board computer of the mobile robot (such as on-board computer 130). In some examples, the task formula can be an LTL formula. An environment model (such as environment model 122) and a robot model (such as robot model 124) can also be obtained on the on-board computer.

[0061] In step 604, the task formula is converted into an automaton (e.g., automaton 118). In some examples, the automaton can be a DFA.

[0062] In step 606, an initial map of the space (e.g., initial map X init ) is constructed based on the environment model and the robot model.

[0063] In step 608, a motion tree (e.g., motion tree 160) is constructed using the initial map of the space and the automaton.

[0064] In step 610, a sampling-based tree search (e.g., sampling-based tree search 105) is performed using the motion tree to generate an initial plan (e.g., initial plan 156) for the mobile robot to operate within the space and satisfy the task formula. In some examples, the initial plan can include a trajectory (e.g., trajectory 170). In some examples, if it is determined that the initial plan no longer satisfies the task formula, the motion tree can be reshaped and a new sampling-based tree search can be performed using the updated motion tree to generate a revised plan (e.g., revised plan 158).

[0065] In step 612, the mobile robot is made to move along the trajectory. In some examples, the on-board computer includes a control circuit (e.g., control circuit 140) configured to receive the trajectory and generate a control signal (e.g., control signal 132) to make the mobile robot move along the trajectory.

[0066] Figure 7 An example computer system 700 including various hardware elements is shown in accordance with some embodiments of the present disclosure. The computer system 700 can be incorporated or integrated into the devices described herein, and / or can be configured to perform some or all of the steps of the methods provided by the various embodiments. For example, in various embodiments, the computer system 700 can be incorporated into system 100, and / or can be configured to perform method 600. It should be noted that Figure 7 it is only intended to provide a general description of the various components, any or all of which can be used as needed. Thus, Figure 7 it generally shows how the various system elements can be implemented in a relatively separated or relatively more integrated manner.

[0067] In the example shown, computer system 700 includes a communication medium 702, one or more processors 704, one or more input devices 706, one or more output devices 708, a communication subsystem 710, and one or more storage devices 712. Computer system 700 can be implemented using various hardware implementation methods and embedded system technologies. For example, one or more components of computer system 700 can be implemented in an integrated circuit (IC), an application-specific integrated circuit (ASIC), an application-specific standard product (ASSP), a field-programmable gate array (FPGA, such as those commercially sold by companies like Intel or Lattice Semiconductor (LATTICE )) and other products), a system-on-chip (SoC), a microcontroller, a printed circuit board (PCB), and / or a hybrid device (such as an SoC FPGA), etc.

[0068] The various hardware components of computer system 700 can be communicatively connected via communication medium 702. For clarity, communication medium 702 is shown as a single connection, but it should be understood that communication medium 702 can include various quantities and types of communication media for transmitting data between hardware components. For example, communication medium 702 can include one or more wires (e.g., conductive traces, paths, or leads on a printed circuit board or integrated circuit (IC), microstrip lines, stripline, coaxial cables), one or more optical waveguides (e.g., optical fibers, strip waveguides), and / or one or more wireless connections or links (e.g., infrared wireless communication, radio communication, microwave wireless communication), etc.

[0069] In some embodiments, communication medium 702 can include one or more buses for connecting the pins of the hardware components of computer system 700. For example, communication medium 702 can include a bus connecting processor 704 and main memory 714, called the system bus; and a bus connecting main memory 714 with input device 706 or output device 708, called the expansion bus. The system bus itself can be composed of several buses, including an address bus, a data bus, and a control bus. The address bus can transmit the memory address of processor 704 to the address bus circuit associated with main memory 714 so that the data bus can access and transfer the data stored at that memory address back to processor 704. The control bus can transmit commands from processor 704 and return status signals from main memory 714. Each bus can include multiple wires for transmitting multiple bits of information, and each bus can support serial or parallel transmission of data.

[0070] The processor 704 may include one or more central processing units (CPUs), graphics processing units (GPUs), neural network processors or accelerators, digital signal processors (DSPs), and / or other general-purpose or special-purpose processors capable of executing instructions. The CPU may be in the form of a microprocessor, which may be fabricated on a single IC chip composed of metal-oxide-semiconductor field-effect transistors (MOSFETs). The processor 704 may include one or more multi-core processors, where each core may read and execute program instructions simultaneously with other cores, thereby increasing the running speed of programs that support multi-threading.

[0071] The input device 706 may include one or more of various user input devices, such as a mouse, keyboard, microphone, and various sensor input devices, such as image acquisition devices, temperature sensors (e.g., thermometers, thermocouples, thermistors), pressure sensors (e.g., barometers, tactile sensors), motion sensors (e.g., accelerometers, gyroscopes, tilt sensors), light sensors (e.g., photodiodes, photodetectors, charge-coupled devices), etc. The input device 706 may also include a device for reading and / or receiving a removable storage device or other removable media. Such removable media may include optical discs (e.g., Blu-ray discs, DVDs, CDs), memory cards (e.g., CompactFlash cards, Secure Digital (SD) cards, memory sticks), floppy disks, universal serial bus (USB) flash drives, external hard disk drives (HDDs), or solid-state drives (SSDs), etc.

[0072] The output device 708 may include various devices that convert information into a human-readable form, such as but not limited to display devices, speakers, printers, tactile or haptic devices, etc. The output device 708 may also include a device for writing to a removable storage device or other removable media, such as those mentioned in the input device 706 section. The output device 708 may also include various actuators for causing one or more components to move physically. These actuators may be hydraulic, pneumatic, electric, and may be controlled using control signals generated by the computer system 700.

[0073] The communication subsystem 710 may include hardware components for connecting the computer system 700 to systems or devices located outside the computer system 700 (e.g., via a computer network). In various embodiments, the communication subsystem 710 may include wired communication devices, optical communication devices (e.g., optical modems), infrared communication devices, radio communication devices (e.g., wireless network interface controllers, Bluetooth devices, IEEE 802.11 devices, Wi-Fi devices, Wi-Max devices, cellular devices) connected to one or more input / output ports (e.g., universal asynchronous receiver / transmitter (UART)).

[0074] The storage device 712 may include various data storage devices of the computer system 700. For example, the storage device 712 may include various types of computer memories with different response times and capacities, ranging from memories with faster response times and lower capacities, such as processor registers and caches (e.g., L0, L1, L2), to memories with moderate response times and moderate capacities, such as random access memory (RAM), and then to memories with slower response times and lower capacities, such as solid state drives and hard disk drives. Although the processor 704 and the storage device 712 are shown as separate elements, it should be understood that the processor 704 may include different levels of on-processor memory, such as processor registers and caches, which may be used by a single processor or shared among multiple processors.

[0075] The storage device 712 may include a main memory 714, which the processor 704 can directly access via the address bus and data bus of the communication medium 702. For example, the processor 704 may continuously read and execute instructions stored in the main memory 714. Thus, various software elements may be loaded into the main memory 714 for reading and execution by the processor 704, as Figure 7 shown. Generally, the main memory 714 is volatile memory and loses all data when power is lost, so a power source is required to preserve the stored data. The main memory 714 may also include a small portion of non-volatile memory that contains software (e.g., firmware such as BIOS) for reading other software stored in the storage device 712 into the main memory 714. In some embodiments, the volatile memory of the main memory 714 is implemented as RAM, such as dynamic random access memory (DRAM), and the non-volatile memory of the main memory 714 is implemented as read-only memory (ROM), such as flash memory, erasable programmable read-only memory (EPROM), or electrically erasable programmable read-only memory (EEPROM).

[0076] The computer system 700 may include software elements, as shown currently located within the main memory 714, which may include an operating system, device drivers, firmware, a compiler, and / or other code, such as one or more application programs, which may include the computer programs provided by the various embodiments of the present disclosure. By way of example only, one or more steps described in any of the methods discussed above may be implemented as instructions 716 that may be executed by the computer system 700. In one example, such instructions 716 may be received by the computer system 700 via the communication subsystem 710 (e.g., via a wireless or wired signal carrying the instructions 716), transmitted via the communication medium 702 to the storage device 712, stored within the storage device 712, read into the main memory 714, and executed by the processor 704 to perform one or more steps of the described method. In another example, the instructions 716 may be received by the computer system 700 via the input device 706 (e.g., via a reader for removable media), transmitted via the communication medium 702 to the storage device 712, stored within the storage device 712, read into the main memory 714, and executed by the processor 704 to perform one or more steps of the described method.

[0077] In some embodiments of the present disclosure, the instructions 716 are stored on a computer-readable storage medium (or simply referred to as a computer-readable medium). Such a computer-readable medium may be non-transitory and may thus be referred to as a non-transitory computer-readable medium. In some cases, the non-transitory computer-readable medium may be incorporated within the computer system 700. For example, the non-transitory computer-readable medium may be one of the storage devices 712 (as Figure 7 shown). In some cases, the non-transitory computer-readable medium may be separate from the computer system 700. In one example, the non-transitory computer-readable medium may be a removable medium provided to the input device 706 (as Figure 7 shown), such as those mentioned in the input device 706 section, and the instructions 716 are read into the computer system 700 by the input device 706. In another example, the non-transitory computer-readable medium may be a component of a remote electronic device (such as a mobile phone) that may wirelessly transmit a data signal carrying the instructions 716, and the signal is received by the communication subsystem 710 (as Figure 7 shown).

[0078] Instruction 716 can be in any suitable form for being read and / or executed by computer system 700. For example, instruction 716 can be source code (written in a human-readable programming language such as Java, C, C++, C#, Python), object code, assembly language, machine code, microcode, executable code, etc. In one example, instruction 716 is provided to computer system 700 in the form of source code, and a compiler is used to translate instruction 716 from the source code into machine code, and then the machine code can be read into main memory 714 for execution by processor 704. As another example, instruction 716 is provided to computer system 700 in the form of an executable file, which contains machine code that can be immediately read into main memory 714 for execution by processor 704. In various examples, instruction 716 can be provided to computer system 700 in an encrypted or unencrypted form, a compressed or uncompressed form, in the form of an installation package or as the initialization of a broader software deployment, etc.

[0079] In one aspect of the present disclosure, a system (such as computer system 700) is provided for performing methods according to various embodiments of the present disclosure. For example, some embodiments can include a system that includes one or more processors (such as processor 704) communicatively coupled to a non-transitory computer-readable medium (such as storage device 712 or main memory 714). Instructions (such as instruction 716) can be stored in the non-transitory computer-readable medium, and when executed by the one or more processors, cause the one or more processors to perform the methods described in the various embodiments.

[0080] In another aspect of the present disclosure, a computer program product is provided that includes instructions (such as instruction 716) for performing methods according to various embodiments of the present disclosure. The computer program product can be embodied specifically in a non-transitory computer-readable medium (such as storage device 712 or main memory 714). The instructions are configured to cause one or more processors (such as processor 704) to perform the methods described in the various embodiments.

[0081] In another aspect of the present disclosure, a non-transitory computer-readable medium (such as storage device 712 or main memory 714) is provided. Instructions (such as instruction 716) are stored in the non-transitory computer-readable medium, and when executed by one or more processors (such as processor 704), cause the one or more processors to perform the methods described in the various embodiments of the present disclosure.

[0082] The above methods, systems, and devices are all examples. Various configurations may omit, substitute, or add various steps or components as needed. For example, in alternative configurations, the order of execution of the method may be different from that described, and / or individual stages may be added, omitted, and / or combined. Additionally, features described for certain configurations may be combined in various other configurations. Elements of different aspects and configurations may also be combined in a similar manner. Moreover, technology is constantly evolving, so many elements are merely examples and do not limit the scope of the present disclosure or the claims.

[0083] Specific details are given in the specification to provide a thorough understanding of the exemplary configurations, including implementations. However, the configurations may be practiced without these specific details. For example, well-known circuits, processes, algorithms, structures, and techniques are not unnecessarily elaborated to avoid obscuring the configurations. This specification merely provides example configurations and does not limit the scope, applicability, or configurations of the claims. Instead, the foregoing configuration description will provide an enabling description for those skilled in the art to implement the described technology. Various changes may be made to the function and arrangement of the elements without departing from the spirit or scope of the present disclosure.

[0084] After describing several example configurations, various modifications, alternative constructions, and equivalents may be used without departing from the spirit of the present disclosure. For example, the above elements may be components of a larger system, where other rules may take precedence over or otherwise modify the application of the technology. Additionally, many steps may be taken before, during, or after considering the above elements. Therefore, the above description does not limit the scope of the claims.

[0085] As used herein and in the appended claims, unless the context clearly indicates otherwise, the singular forms "a," "an," and "the" include plural references. Thus, for example, a reference to "a user" includes a reference to one or more such users, a reference to "a processor" includes a reference to one or more processors and their equivalents known to those skilled in the art, and so on.

[0086] Furthermore, when used in this specification and the following claims, the words "comprises," "comprising," "includes," "including," "has," "having," "contains," and "containing" are intended to specify the presence of the stated features, integers, components, or steps, but do not preclude the presence or addition of one or more other features, integers, components, steps, acts, or groups.

[0087] It should also be understood that the examples and embodiments described herein are for illustrative purposes only, and those skilled in the art will make various modifications or changes based on these examples and embodiments, and such modifications or changes should be included within the spirit and scope of this application and the scope of the appended claims.

Claims

1. A method for performing mission planning for a mobile robot operating in space, the method comprising: Obtaining a task formulation on an onboard computer of the mobile robot; Convert task formulas into automata; Use the initial map of the space and the automaton to build a movement tree; Performing a sampling-based tree search using the kinematic tree to generate an initial plan for the mobile robot to operate in the space and satisfy the task formula, the initial plan including a trajectory; as well as Make the mobile robot move along the trajectory.

2. The method according to claim 1, further comprising: Obtain the second mission formula on the onboard computer; Convert the second task formula into a second automaton; Update the movement tree using the second automaton; performing a second sampling-based tree search using the updated kinematic tree to generate a revised plan for the mobile robot to operate in the space and satisfy the second task formula, the revised plan including a second trajectory; as well as Make the mobile robot move along the second trajectory.

3. The method according to claim 1, further comprising: Obtaining the environment model and the robot model on an onboard computer; as well as Build an initial map of the space based on the environment model and the robot model.

4. The method of claim 1, wherein the onboard computer executes a natural language processing (NLP) model for generating a task formula based on input text. The method of claim 1 , wherein the automaton is a deterministic finite automaton (DFA). The method of claim 1 , wherein the task formula is a linear temporal logic (LTL) formula.

7. The method of claim 1, wherein the onboard computer includes a control circuit configured to receive the trajectory and generate a control signal to cause the mobile robot to move along the trajectory.

8. A non-transitory computer-readable medium comprising instructions that, when executed by one or more processors, cause the one or more processors to perform operations for a mobile robot operating in a space, the operations comprising: Obtaining a task formulation on an onboard computer of the mobile robot; Convert task formulas into automata; Use the initial map of the space and the automaton to build a movement tree; Performing a sampling-based tree search using the kinematic tree to generate an initial plan for the mobile robot to operate in the space and satisfy the task formula, the initial plan including a trajectory; as well as Make the mobile robot move along the trajectory.

9. The non-transitory computer readable medium of claim 8, wherein the operations further comprise: Obtain the second mission formula on the onboard computer; Convert the second task formula into a second automaton; Update the movement tree using the second automaton; performing a second sampling-based tree search using the updated kinematic tree to generate a revised plan for the mobile robot to operate in the space and satisfy the second task formula, the revised plan including a second trajectory; as well as Make the mobile robot move along the second trajectory.

10. The non-transitory computer readable medium of claim 8, wherein the operations further comprise: Acquiring an environment model and a robot model on an onboard computer; and Build an initial map of the space based on the environment model and the robot model.

11. The non-transitory computer-readable medium of claim 8, wherein the onboard computer executes a natural language processing (NLP) model for generating a task formula from input text.

12. The non-transitory computer-readable medium of claim 8, wherein the automaton is a deterministic finite automaton (DFA).

13. The non-transitory computer-readable medium of claim 8, wherein the task formula is a linear temporal logic (LTL) formula.

14. The non-transitory computer-readable medium of claim 8, wherein the onboard computer comprises a control circuit configured to receive the trajectory and generate a control signal to cause the mobile robot to move along the trajectory.

15. A system comprising: one or more processors; as well as A non-transitory computer-readable medium comprising instructions that, when executed by one or more processors, cause the one or more processors to perform operations for a mobile robot operating in a space, the operations comprising: Obtaining a task formulation on an onboard computer of the mobile robot; Convert task formulas into automata; Use the initial map of the space and the automaton to build a movement tree; Performing a sampling-based tree search using the kinematic tree to generate an initial plan for the mobile robot to operate in the space and satisfy the task formula, the initial plan including a trajectory; and Make the mobile robot move along the trajectory.

16. The system of claim 15, wherein the operations further comprise: Obtain the second mission formula on the onboard computer; Convert the second task formula into a second automaton; Update the movement tree using the second automaton; performing a second sampling-based tree search using the updated kinematic tree to generate a revised plan for the mobile robot to operate in the space and satisfy the second task formula, the revised plan including a second trajectory; as well as Make the mobile robot move along the second trajectory.

17. The system of claim 15, wherein the operations further comprise: Acquiring an environment model and a robot model on an onboard computer; and Build an initial map of the space based on the environment model and the robot model.

18. The system of claim 15, wherein the onboard computer executes a natural language processing (NLP) model for generating a task formulation based on input text.

19. The system of claim 15, wherein the automaton is a deterministic finite automaton (DFA).

20. The system of claim 15, wherein the task formula is a linear temporal logic (LTL) formula.

Images