A robot's skill control system for performing parallel missions

The system optimally manages parallel missions for humanoid robots by using an interpretation and supervision module with task planners and low-level runes to ensure deterministic task execution, addressing the challenges of probabilistic models and dynamic environments.

FR3175005A1Pending Publication Date: 2026-11-13ENCHANTED TOOLS
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
FR2025004854
Authority / Receiving Office
FR · FR
Patent Type
Applications
Current Assignee / Owner
Filing Date
2025-05-07
Publication Date
2026-11-13

AI Technical Summary

Technical Problem

Existing technologies fail to optimally manage and execute parallel missions for robots, particularly humanoid robots, due to the probabilistic nature of foundation models and the lack of real-time consideration of changing environmental conditions and resource constraints.

Method used

A system comprising an interpretation and supervision module coupled with a task planner, utilizing foundation models to process information from perception, skills, and world state, determining an ordered list of missions with priority levels, and incorporating low-level runes for deterministic task execution.

Benefits of technology

Enables efficient management of parallel missions by humanoid robots, ensuring real-time adaptation to changing environments and resource constraints, while prioritizing tasks effectively.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 00000000_0000_ABST
    Figure 00000000_0000_ABST
Patent Text Reader

Abstract

The present invention relates to a system for controlling the skills of a robot for carrying out a set of missions. The robot has a set of skills, a set of perception means, and a set of available material resources. The system comprises an interpretation and supervision module coupled to a task planner, configured to manage the missions and to determine, based on information received from the set of skills, the set of perception means, and the set of available material resources, an ordered list of missions. Each mission has a priority level. The task planner defines a list of tasks to be performed by the robot. Several tasks from the task list can, if necessary, be implemented in parallel by the robot. Abstract figure: Figure 1
Need to check novelty before this filing date? Find Prior Art

Description

Title of the invention: System for controlling the skills of a robot for performing parallel missions. Technical field

[0001] The present invention relates to the field of robots, particularly humanoid robots. Such robots possess diverse capabilities. A PDDL (Planning Domain Definition Language) interface can manage instruction sets, that is, a sequence of assigned tasks, to enable a robot to accomplish a mission or a series of missions, executed using the robot's capabilities. The invention specifically aims to optimally manage the execution of a mission assigned to the robot, or even situations in which several missions are assigned to the robot in parallel. STATE OF THE ART

[0002] The principle of foundation models, in particular large language models or large visual models, is known for interpreting instructions received and setting a list of missions assigned to a robot.

[0003] The principle of definition language interfaces is also known per se. Such definition language interfaces are known as "PDDL interfaces," for "Planning Domain Definition Language." As is known, in the field of artificial intelligence, this involves formatting the input data of a task scheduler. A modified PDDL interface of this type will be referred to, within the scope of the present invention, as an interpretation and monitoring module.

[0004] The present invention relates to the control of parallel missions assigned to a robot, equipped with skills and material resources, within an evolving surrounding context, also referred to as the "state of the world." In other words, the invention lies at the intersection of an interpretation and supervision module, the skills of a robot, and the implementation of a foundation model, for the purpose of accomplishing a set of missions assigned to the robot.

[0005] A "competence" is understood to mean a robot's ability to perform a task. In other words, a competence is, in a sense, a partial task such as "turn 45 degrees," "turn towards the target," "animate the arm to simulate an emotion," "give a thumbs-up," "move forward one meter," etc. Competencies are configurable, for example, in terms of "degrees," "distance in meters," etc. Some competencies can be described as "high-level." High-level competencies are competencies coupled with constraints. Competencies thus vary in complexity, particularly in that they require the control of more or fewer hardware sub-assemblies of the robot: for example, two base motors to turn, or more actuators and components to animate the arms, display a specific dynamic projected face, etc.

[0006] The robot's skills are intended to be combined to perform tasks and accomplish missions.

[0007] The determination of how the robot's skills should be combined to perform tasks and accomplish missions is carried out by the interpretation and supervision module, in particular through the implementation of a foundation model.

[0008] The general problem is to propose a deterministic operation of an interpretation and supervision module in a context where the missions assigned to the robot are determined by a foundation model, in particular a grand language model or a visual language model, whose operation is probabilistic.

[0009] In the prior art, a PDDL interface determines, for a robot, an objective, a main mission. The robot then carries out successive missions, without any real-time questioning of the relevance of the way in which the different missions are ordered, including when the state of the world changes.

[0010] With the present invention, it is desired to manage several parallel missions.

[0011] In other words, the technical problem can be expressed as follows: how to use the robot's skills while taking into account the constraints of reality, which evolve over time, and the limited material resources of the robot, particularly in the context of managing parallel missions.

[0012] The main parameters are the available material resources of the robot, the constraints of the surrounding real world and the prioritization of missions. PRESENTATION OF THE INVENTION

[0013] More specifically, the invention relates to a system for controlling the skills of a robot for carrying out a set of missions, the set of missions being assigned to the robot, the robot having a set of skills enabling the performance of tasks, a set of perception means and a set of available material resources, the system comprising an interpretation and supervision module coupled to a task planner, the interpretation and supervision module being configured to manage the missions of the set of missions in parallel, and to determine, based on information, including textual information, received from the set of skills, the set of perception means and the set of available material resources, an ordered list of missions, each mission having a priority level, the planner tasks defining a list of tasks to be performed by the robot, the tasks in the task list being each associated with information, including textual information, relating to the mission to which said task is dedicated, several tasks in the task list being able to be implemented in parallel by the robot if necessary.

[0014] The robot may in particular be a humanoid robot, specifically configured to move on a single spherical wheel.

[0015] Advantageously, the interpretation and supervision module implements a foundation model.

[0016] According to one embodiment, the foundation model is adapted to process together the information received from the perception set, the skills set, and the ordered list of missions, as well as a world state set, a set of the task(s) in progress associated with the mission(s) to which it / they is / are devoted, and a set of the task(s) to be executed; each of said received information being described by information, including textual information, and processed by the foundation model of the interpretation and supervision module which updates the task planner in real time.

[0017] According to one embodiment, the foundation model is adapted to process together, in addition, a history of past information received from the perception set, the skills set, and / or the ordered list of missions, and / or the state of the world set, and / or the set of the task(s) in progress associated with the mission(s) to which it is / are devoted, and / or the set of the task(s) to be executed; said past information being each described by information, including textual information, and processed by the foundation model of the interpretation and supervision module which updates the task planner in real time.

[0018] According to one embodiment, the task(s) being executed leads to the activation of the robot's actuators.

[0019] Advantageously, the skill control system of a robot according to the invention may include means for inhibiting the activation of robot actuators.

[0020] According to one embodiment, the robot's skill set is configured to be enriched by one or more learned skills.

[0021] According to one embodiment, at least one learned skill being acquired by the robot through behavioral programming via runes.

[0022] According to one embodiment, at least some of the tasks in the task list are assigned by instructions given in natural language interpreted by the interpretation and supervision module, by means of a foundation model, in particular a large language model or a visual language model.

[0023] According to one embodiment, at least part of the missions in the mission list is assigned to the robot through behavioral programming via runes.

[0024] The present invention also relates to a robot comprising a robot skills control system as briefly described above.

[0025] In particular, such a robot can be configured to move on a single spherical wheel. PRESENTATION OF THE FIGURES

[0026] The invention will be better understood upon reading the following description, given solely by way of example, and referring to the accompanying drawings, given by way of non-limiting example, and on which:

[0027] [Fig.1] is a schematic representation showing the operation of the system according to the invention;

[0028] [Fig.2] is a schematic representation showing the operation of the system according to the invention, taking into account the history.

[0029] It should be noted that the figures set out the invention in detail to enable implementation of the invention; although not limiting, said figures serve in particular to better define the invention where appropriate. DETAILED DESCRIPTION OF THE INVENTION

[0030] The invention relates to a system for determining and controlling a mission or a set of parallel missions assigned to a robot.

[0031] The system includes in particular an interpretation and supervision module. The interpretation and supervision module is coupled to a set of robot skills, a perception set, a set of surrounding context, also referred to as a world state set, and a task planner.

[0032] The interpretation and supervision module determines, from the set of parallel missions assigned to the robot, an ordered list of missions to be accomplished by the robot, each mission being associated with a level of priority, for example "low priority" or "medium priority" or "high priority".

[0033] It is noted in parallel that, in the field of robots, in particular humanoid robots, in order to overcome the structural constraints of the robot and in particular to avoid a "specialization" of robots, the applicant has developed, as described in the patent application whose registration number is FR2212216, a robotic network architecture comprising a set of electronic labels, designated "tags" or even "runes", and at least one robot, in particular including at least one predefined gripper.

[0034] In such a robotic network architecture, the runes are configured to be linked to an element of the robotic network and contain identification information for the element, as well as information enabling the element's location and / or other information allowing the robot to determine its behavior towards the element. Thanks to the information contained in the rune, the robot can, in particular, determine whether it has the possibility, including the capacity and / or the right, to grasp and / or manipulate and / or move said element, especially using the predefined gripper. The robot is configured to interact with any type of element comprising at least one rune, and said robot includes a module for interpreting said identification information, as well as any other information possibly contained in the rune.

[0035] In this general context, generative artificial intelligence algorithms based on foundation models (LxM), also referred to in the prior art, i.e., deep-learning models trained on very large datasets, are implemented. In particular, such foundation models can be large language models (LLMs). Other foundation models can be implemented in this context, such as visual large models (VLMs), i.e., visual language models that simultaneously interpret images from the robot's cameras.

[0036] These foundation models are developed to allow the robot to interact with its environment, particularly its "social" environment. The robot is thus equipped with means of perception, including listening capabilities via microphones, and a module for interpreting instructions that can be given to it in "natural language".

[0037] Instructions can also be given to the robot via runes as described above. In this case, in particular, a method for programming the robot's behavior is implemented, using physical or virtual runes, within the robotic network architecture. In this case, the robotic network architecture comprises the robot and at least one element, the runes belonging to one of the identification and action types, the identification type including subtypes such as location, object, and person. The behavioral programming method then comprises the following steps:

[0038] the association of at least one identification-type rune with at least one element of the robotic network;

[0039] the coupling of at least one identification type rune to at least one action type rune;

[0040] the activation of at least one rune to generate the execution of a task by the robot with respect to the element associated with said identification type rune coupled with said action type rune.

[0041] The coupling step corresponds to the definition or modification of a rule configured so that the robot performs an interpretation of the activation of at least one rune to perform a predefined task with respect to the element associated with said rune of identification type coupled with said rune of action type, said interpretation being carried out by means of the implementation of the interpretation module, in particular based on a foundation model, for example a large language model or a visual language model, the task to be carried out by the robot corresponding to an output of the interpretation module, the task planner having the function of ordering the succession of tasks to be carried out by the robot.

[0042] Within the framework of the present invention, the system includes an interpretation and supervision module, with expanded capabilities compared to the interpretation module described above.

[0043] Thus, the foundation models implemented in the context of the present invention allow the interpretation and supervision module to determine an ordered list of tasks to be performed by the robot. These tasks are associated with priority levels and are translated, using the task planner, into a list of tasks to be carried out by the robot in the real world. These tasks involve, for example, the activation of the robot's actuators.

[0044] The robot's abilities may include: moving to a place, object or person equipped with a rune, moving to a location, grasping an object equipped with a rune, smiling, saying.

[0045] Preferably, the interpretation and supervision module is also coupled to a set of tasks that are currently being executed or whose execution is planned and is yet to come.

[0046] The perception assembly may include: one or more optical sensors, including cameras, one or more touch sensors, one or more Bluetooth® antennas, etc.

[0047] The world state set, in other words the representation of the world as taken into consideration by the interpretation and supervision module, may include textual information on the location of the robot or on its state: it is holding something, etc. This corresponds to the state of the world as it can be perceived at time t.

[0048] Each set can be enriched and / or evolve over time, including in parallel with the execution of tasks by the robot.

[0049] Each set is associated with information, including textual information, to serve as input to the foundation model.

[0050] In particular, each mission is described by textual information and presents a level of priority.

[0051] In particular, each task is described by textual information and has a priority level. Each task is dedicated to a mission.

[0052] The tasks may cause the robot's actuators to activate. These activations may be inhibited by low-level runes.

[0053] Indeed, in this context, to set limits in the real world on what the robot can do, the low-level runes interact with the robot "without the robot's knowledge," that is, by "short-circuiting" the interpretation module implementing the generative artificial intelligence algorithms. For example, a verification module processes all output from ongoing tasks to take into account the existence of instructions from low-level runes imposing "physical" limits on the robot. The low-level runes are therefore runes with an active "low-level flag," also referred to as a "low-level field."

[0054] The principle is that the implementation of these low-level runes is deterministic: the robot is configured to implement the instructions contained in the low-level runes in a non-restrictive manner. It is therefore known with certainty that the rules and prohibitions imposed by these low-level runes, via the instructions they contain, will be implemented by the robot.

[0055] The instructions contained in the low-level runes are interpreted by the robot independently of the interpretation module.

[0056] The instructions contained in the low-level runes are configured to allow the robot to force its functions. If the instructions and physical limits from the low-level runes are received by the robot, including after instructions from the interpretation module, said limits are implemented directly and without delay by the robot.

[0057] Low-level runes can thus inhibit the activation of actuators necessary for the implementation of a task assigned to the robot.

[0058] Furthermore, the robot can acquire new skills, through behavioral programming, notably via runes.

[0059] Missions are determined following instructions given in natural language or by behavioral programming, notably via runes.

[0060] The missions are processed in parallel under the control of the interpretation and supervision module.

[0061] The interpretation and supervision module takes into account a set of available hardware resources of the robot, in particular to update the ordered list of missions or to directly modify the list of tasks defined by the task planner.

[0062] The interpretation and supervision module acts in a way as a "primordial brain" of the robot.

[0063] In particular, the foundation model of the interpretation and supervision module is adapted to process together the information received from the perception set, the skills set, and the ordered list of missions, as well as a set of world state, a set of the task(s) in progress associated with the mission(s) to which it / they is / are devoted, and a set of the task(s) to be executed; each of said received information being described by information, including textual information, and processed by the foundation model of the interpretation and supervision module, which updates the task planner in real time. Examples

[0064] In the example of [Fig.1], the robot's set of skills includes: "go to a rune"; "go to a place (coupled with a rune)"; "grab (an object equipped with) a rune"; "smile"; "say"; etc.

[0065] The state of the world includes in particular the following information: "the robot is at the entrance"; "the robot is carrying flowers"; etc.

[0066] The robot's perception system allows the robot, for example, to have the following information: one or more vision sensors (cameras); one or more touch sensors; one or more rune detectors; one or more ASR sensors; etc.

[0067] Parallel missions are assigned to the robot, in particular the following missions: "guide customers", with a high degree of priority; "provide information to Jerome", with a medium degree of priority; "take flowers at the entrance", with a low degree of priority; "express a joyful mood", with a low degree of priority; etc.

[0068] From then on, the LxM foundation model is implemented.

[0069] The ordered list of tasks is processed by the task planner to generate tasks for the robot to be performed sequentially or in parallel, said tasks being associated with the missions in the ordered list of missions assigned to the robot, in particular:

[0070] “telling customers: 'follow me'”, associated with the mission “guiding customers”; then "Go to a place (where Jerome is)," associated with the mission "provide information to Jerome"; then "tell Jerome," associated with the mission "provide information to Jerome"; then "go to a rune (paired with the flowers)," associated with the mission "take the flowers at the entrance"; "grab the flowers (paired with a rune) with your left hand"; and, in parallel:

[0071] “smile”, associated with the mission “express a joyful mood”.

[0072] Some tasks require the activation of actuators. Where appropriate, the verification module may include means to inhibit these tasks by prohibiting the activation of actuators.

[0073] These robot tasks may require the activation of robot actuators.

[0074] The interpretation and supervision module forms a kind of "primordial brain" of the robot. In particular, the interpretation and supervision module is linked to the perception set, the skills set, and the ordered list of missions. The interpretation and supervision module is also linked to the world state set, the set of tasks currently being executed associated with the mission(s) to which they are dedicated, and the set of tasks to be executed. Each of these received pieces of information is described by a specific piece of information, including text, and processed by the LxM foundation model of the interpretation and supervision module, which updates the task scheduler in real time.Preferably, the history of this information is also described by information, including text, and processed by the LxM foundation model of the interpretation and monitoring module, which updates the task scheduler in real time.

[0075] The interpretation and monitoring module is implemented, in particular, using hardware resources (computer, memory, etc.) that are embedded on the robot. Similarly, with reference to [Fig. 2], the history of information processed by the interpretation and monitoring module can be stored in the robot's memory. Alternatively, the interpretation and monitoring module is implemented, in particular, using remote hardware resources (computer, memory, etc.) installed on a remote server. Likewise, the history of information processed by the interpretation and monitoring module can be stored in the memory of a remote server.

[0076] It should be noted that the invention is not limited to the embodiments described above. It will indeed become apparent to a person skilled in the art that various modifications can be made to the embodiments described above, in light of the instruction that has just been disclosed to them.

[0077] In the detailed presentation of the invention given above, the terms used shall not be interpreted as limiting the invention to the embodiment set forth in this description, but shall be interpreted to include all equivalents which can be foreseen by a person skilled in the art by applying their general knowledge to the implementation of the teaching which has just been disclosed to them.

Claims

Demands

1. A system for controlling the skills of a robot for the performance of a set of missions, the set of missions being assigned to the robot, the robot having a set of skills enabling the performance of tasks, a set of perception means and a set of available material resources, the system comprising an interpretation and supervision module coupled to a task planner, the interpretation and supervision module being configured to manage the missions of the set of missions in parallel, and to determine, based on information, including textual information, received from the set of skills, the set of perception means and the set of available material resources, an ordered list of missions, each mission having a priority level, the task planner defining a list of tasks to be performed by the robot,Each task in the task list is associated with information, including text, relating to the mission to which that task is dedicated; several tasks in the task list can, if necessary, be implemented in parallel by the robot.

2. A robot skill control system according to claim 1, wherein the interpretation and supervision module implements a foundation (LxM) model.

3. A robot skill control system according to claim 2, wherein the foundation model (LxM) is adapted to process together the information received from the perception set, the skill set, and the ordered list of missions, as well as a world state set, a set of the task(s) being executed associated with the mission(s) to which it / they is / are devoted, and a set of the task(s) to be executed; each of said received information being described by information, including textual information, and processed by the foundation model (LxM) of the interpretation and supervision module which updates the task planner in real time.

4. A robot skill control system according to claim 3, wherein the foundation model (LxM) is adapted to process together, in addition, a history of past information received from the perception set, the skills set, and / or the ordered list of missions, and / or the state of the world set, and / or the set of the task(s) in progress associated with the mission(s) to which it is / are devoted, and / or the set of the task(s) to be executed; said past information being each described by information, including textual information, and processed by the foundation model (LxM) of the interpretation and supervision module which updates the task planner in real time.

5. A robot skill control system according to any one of the preceding claims, wherein the task(s) being performed results in the activation of robot actuators.

6. A robot skill control system according to claim 5, comprising means for inhibiting the activation of robot actuators.

7. A robot skill control system according to any one of the preceding claims, wherein the robot's skill set is configured to be enriched by one or more learned skills.

8. A robot skill control system according to claim 7, at least one learned skill being acquired by the robot through behavioral programming via runes.

9. A robot skills control system according to any one of the preceding claims, wherein at least some of the missions in the mission list are assigned by instructions given in natural language interpreted by the interpretation and supervision module, by means of a foundation model (LxM), in particular a large language model or a visual language model.

10. A robot skill control system according to any one of the preceding claims, wherein at least some of the missions in the mission list are assigned to the robot through behavioral programming via runes.

11. Robot, comprising a system for controlling the skills of a robot according to one of the preceding claims.

12. Robot according to the preceding claim, configured to move on a single spherical wheel.