Information processing device, information processing method, and program

Abstract

A technology is provided for obtaining highly explainable output regarding the control of a robot device. [Solution] An information processing device according to one aspect of the present invention acquires observation data of an environment in which a robot device operates and instruction information relating to a task goal to be given to the robot device, generates a problem description of the task from the acquired observation data and instruction information using an inference module, and outputs the generated problem description. The problem description is configured to include descriptions of initial states and goal states of objects in the environment.

Description

[Technical Field]

[0001] This invention relates to an information processing apparatus, an information processing method, and a program. [Background technology]

[0002] Interacting with robotic devices using natural language is a promising means for non-experts to enable robotic devices to solve complex and diverse tasks. In recent years, research using large language models (LLMs) to output the sequence of actions of a robotic device in a workspace from linguistic instructions has attracted attention.

[0003] For example, Non-Patent Document 1 proposes a system for generating a sequence of actions for a robot device from linguistic instructions. The system proposed in Non-Patent Document 1 accepts natural language instructions as input. The system uses an LLM (Say module) to infer from the linguistic instructions the next most likely action for the robot device to be performed. The system also uses a value function (Can module) to infer highly feasible actions from observed data. The system then integrates the two inference results and, based on the result of the integration, determines the sequence of actions to be given to the robot device (a sequence of actions that is highly likely to be performed and is feasible). [Prior art documents] [Non-patent literature]

[0004] [Non-Patent Document 1] Michael Ahn et al., “Do As I Can, Not As I Say: Grounding Language in Robotic Affordances”, [online], [Retrieved October 24, 2023], Internet<URL:https: / / arxiv.org / abs / 2204.01691> [Overview of the Initiative] [Problems that the invention aims to solve]

[0005] The inventors of this case have found the following problems with the conventional system described above. Specifically, the generated motion sequences in the conventional system are specific to the trial environment and are not necessarily interpretable to humans. Because the motion sequences are output as they are, there is a problem in that the explainability of the output obtained is low when it comes to controlling the robot device.

[0006] In one respect, this invention has been made in consideration of these circumstances, and its purpose is to provide a technology for obtaining highly explainable output in relation to the control of robotic devices. [Means for solving the problem]

[0007] To solve the above-mentioned problems, the present invention employs the following configuration. The configurations of the invention described below can be combined as appropriate.

[0008] In other words, an information processing device according to one aspect of the present invention includes a control unit configured to perform the steps of: acquiring observation data of the environment in which a robot device operates and instruction information relating to the objective of a task to be given to the robot device; generating a problem description of the task from the acquired observation data and instruction information using an inference module; and outputting the generated problem description. The problem description includes a description of the initial state and target state of objects present in the environment.

[0009] The problem description, along with the domain description, is used by the planner to generate an action plan (i.e., obtain an action sequence). This problem description describes the initial and target states of objects in the environment in order to accomplish the task, and is highly explainable (interpretable to humans). (This is possible.) Therefore, with this configuration, it is possible to obtain a highly explainable output for obtaining the motion sequence (command) to be given to the robot device.

[0010] In the information processing apparatus according to the above aspect, the generated problem description may conform to a predetermined format. According to this configuration, since the generated problem description conforms to the predetermined format, it is possible to facilitate the planner to generate an action plan from the problem description.

[0011] In the information processing apparatus according to the above aspect, the generated problem description may further include a description of the object existing in the environment. According to this configuration, by the description of the object, it becomes easier to identify the object existing in the environment, so that it is possible to facilitate the planner to generate an action plan from the problem description.

[0012] In the information processing apparatus according to the above aspect, the observation data may be constituted by sensing data of a sensor, and the instruction information may be constituted by language information that instructs the target in natural language. According to this configuration, in a scenario where the environment is observed by a sensor and the target of a task is given by a language instruction, it is possible to obtain a highly explanatory output regarding the control of the robot apparatus.

[0013] In the information processing apparatus according to the above aspect, the inference module may be configured to include a trained model of in-context learning. The control unit , in the process of giving the generated problem description to the planner to generate an action plan for the robot apparatus, when the planner outputs an error message, further execute the step of obtaining the output error message, and the step of using the inference module to generate a new problem description from the problem description and the error message. It may be configured as such. According to this configuration, when an appropriate problem description cannot be obtained, the problem description can be automatically corrected. Note that the trained model is a trained machine learning model.

[0014] The control unit of the information processing apparatus according to the above aspect may be configured to further acquire environmental information regarding the environment in the acquiring step. Generating the problem description from the acquired observation data and the instruction information may be constituted by generating the problem description from the acquired observation data, the instruction information, and the environmental information. According to this configuration, in addition to the observation data and the instruction information, by further using the environmental information as an explanatory variable, the environment for performing the task can be more specified, and thus an improvement in the accuracy of generating the problem description can be expected.

[0015] In the information processing apparatus according to the above aspect, the generated problem description may further include a description of the object existing in the environment. The inference module may include an object estimator. Generating the problem description using the inference module may include generating a description of the object existing in the environment from the acquired observation data using the object estimator. According to this configuration, the description part of the object in the problem description can be appropriately generated.

[0016] In the information processing apparatus according to the above aspect, the object estimator may include a trained model of in-context learning. According to this configuration, since the object estimator includes a trained model of in-context learning, the object existing in the environment can be generally estimated, and thus it can be expected to cope with the generation of problem descriptions for various tasks.

[0017] The control unit of the information processing apparatus according to the above aspect may be configured to further acquire the attribute information of the object existing in the environment in the acquiring step. Generating the description of the object from the acquired observation data may be constituted by generating the description of the object from the acquired observation data and the attribute information. According to this configuration, since the attribute information makes the object existing in the environment more specifiable, an improvement in the accuracy of generating the description part of the object in the problem description can be expected. Generating the description of the object from the acquired observation data may be constituted by generating the description of the object from the acquired observation data and the attribute information. According to this configuration, since the attribute information makes the object existing in the environment more specifiable, an improvement in the accuracy of generating the description part of the object in the problem description can be expected.

[0018] In the information processing device relating to the above aspect, the inference module may include an initial state estimator. Generating the problem description using the inference module may include generating a description of the initial state of the object present in the environment using the initial state estimator. With this configuration, the initial state description portion of the problem description can be appropriately generated.

[0019] In the information processing device relating to the above aspect, the initial state estimator may include a pre-trained model for in-context learning. With this configuration, the initial state estimator includes a pre-trained model for in-context learning, which allows for the general inference of the initial state of objects present in the environment, thereby enabling the generation of problem descriptions for various tasks.

[0020] In the information processing device relating to the above-described aspect, the inference module may include a target estimator. Generating the problem description using the inference module may include generating a description of the target state of the object present in the environment using the target estimator. With this configuration, the portion describing the target state in the problem description can be appropriately generated.

[0021] In the information processing device relating to the above aspect, the target estimator may include a pre-trained model for in-context learning. With this configuration, the target estimator includes a pre-trained model for in-context learning, which allows for the general inference of the target state of an object, thereby enabling the generation of problem descriptions for various tasks.

[0022] Furthermore, the embodiments of the present invention are not limited to the information processing apparatus described above. As another aspect of the information processing apparatus relating to each of the above aspects, one aspect of the present invention may be an information processing method that realizes all or part of the above configurations, a program, or a storage medium that stores such a program and is readable by a machine such as a computer. A storage medium that is readable by a machine such as a computer is a medium that stores information such as programs by electrical, magnetic, optical, mechanical, or chemical action.

[0023] For example, an information processing method relating to one aspect of the present invention may be an information processing method in which a computer performs the steps of: acquiring observation data of the environment in which a robot device operates and instruction information relating to the objective of a task to be given to the robot device; using an inference module to generate a problem description of the task from the acquired observation data and instruction information; and outputting the generated problem description.

[0024] Furthermore, for example, a program relating to one aspect of the present invention may be a program that causes a computer to perform the following steps: acquiring observational data of the environment in which the robot device operates and instructional information relating to the objective of a task to be given to the robot device; using an inference module to generate a problem description of the task from the acquired observational data and instructional information; and outputting the generated problem description. [Effects of the Invention]

[0025] According to the present invention, a highly explainable output can be obtained with respect to the control of a robotic device. [Brief explanation of the drawing]

[0026] [Figure 1] Figure 1 schematically shows an example of a scenario in which the present invention is applied. [Figure 2] Figure 2 schematically shows an example of the input and output of the inference module according to the embodiment. [Figure 3]Figure 3 schematically shows an example of the process of revising the problem description according to the embodiment. [Figure 4] Figure 4 schematically shows an example of the configuration of an inference module according to the embodiment. [Figure 5A] Figure 5A schematically shows an example of an object estimator according to an embodiment. [Figure 5B] Figure 5B schematically shows an example of an initial state estimator according to an embodiment. [Figure 5C] Figure 5C schematically shows an example of a target estimator according to an embodiment. [Figure 6] Figure 6 schematically shows an example of the hardware configuration of the information processing device according to the embodiment. [Figure 7] Figure 7 schematically shows an example of the software configuration of the information processing device according to the embodiment. [Figure 8] Figure 8 is a flowchart showing an example of the processing procedure of the information processing device according to the embodiment. [Figure 9] Figure 9 schematically shows an example of the configuration of an inference module related to another form. [Figure 10] Figure 10 shows the descriptions included in the domain description for each domain prepared. [Figure 11A] Figure 11A shows an example of observation data and instruction information provided in the domain (Cooking). [Figure 11B] Figure 11B shows an example of observation data and indication information provided in the domain (Blocksworld). [Figure 11C] Figure 11C shows an example of observation data and indication information provided in the domain (Hanoi). [Figure 12] Figure 12 shows the results of the first experimental example. [Figure 13] Figure 13 shows the results of the second experimental example. [Figure 14] Figure 14 shows the results of the third experimental example. [Modes for carrying out the invention]

[0027] Hereinafter, an embodiment relating to one aspect of the present invention (hereinafter also referred to as "this embodiment") will be described based on the drawings. However, this embodiment described below is merely illustrative in all respects of the present invention. Various improvements or modifications may be made without departing from the scope of the present invention. In carrying out the present invention, specific configurations according to the embodiment may be adopted as appropriate. In this embodiment, the data appearing is described in natural language, but more specifically, it is specified in pseudo-language, commands, parameters, machine code, etc. that can be recognized by a computer.

[0028] §1 Examples of Application Figure 1 schematically shows an example of a scenario in which the present invention is applied. The information processing device 1 according to this embodiment is one or more computers configured to generate a problem description 5 used by a planner to generate an action plan. Specifically, the information processing device 1 acquires observational data 20 of the environment in which the robot device R operates, and instructional information 21 regarding the objective of the task to be given to the robot device R. The information processing device 1 uses an inference module 3 to generate a problem description 5 of the task from the acquired observational data 20 and instructional information 21. The problem description 5 includes descriptions (51, 52) of the initial state and target state of one or more objects present in the environment. The information processing device 1 outputs the generated problem description 5.

[0029] Problem description 5 includes descriptions of the initial and target states (51, 52) for one or more objects present in the environment, so that the planner can generate an action plan to reach the target state from the initial state of the task. The initial state is the state before the task is performed. The initial state is the start This may also be called a state. The target state is the state after the task has been completed and the objective has been achieved. The initial state description 51 describes the initial state of one or more objects present in the environment. On the other hand, the target state description 52 describes the target state of one or more objects that is reached upon completion of the task. That is, each description (51, 52) indicates the states before and after the completion of the task and is interpretable to humans. In one example, each description (51, 52) indicates the state of an object rather than the actions of the robot device R, and is therefore easy for humans to interpret without a special understanding of the robot device R (for example, without deciphering the program code). Therefore, this problem description 5 has high explainability. Accordingly, according to this embodiment, a highly explainable output can be obtained for obtaining the action sequence (control command) to be given to the robot device R.

[0030] [Input / Output Data] In this embodiment, observation data 20 and instruction information 21 are provided to the inference module 3 as input data, and as a result of the calculation processing performed by the inference module 3, the problem description 5 is obtained as output data.

[0031] The observation data 20 can be anything that represents the initial state of an object before performing the task. The type of observation data 20 is not particularly limited and may be appropriately selected depending on the embodiment. The observation data 20 may consist of data from one or more modalities. The environment may be observed in any way. The environment may include all events related to the circumstances under which the robot device R performs the task. The environment may consist of at least one of a real environment and a virtual environment. For example, the environment may consist of both a real environment and a virtual environment, such as AR (Augmented Reality) or MR (Mixed Reality). The environment may also include VR (Virtual Reality). The number of objects present in the environment may be arbitrary. The data 20 may be acquired at any time before the start of task execution. When multiple tasks are given, the initial state may be the state before performing all of the tasks, or the state before performing one of the tasks included in the multiple tasks (including intermediate states between tasks). In other words, "initial" does not necessarily mean "first". A task may be any task (work) that transitions from the initial state to the target state. The state before performing this task may be the initial state, and the state after the task has been properly performed may be the target state. The interval between tasks may be determined as appropriate depending on the embodiment. Since the initial state of the object is represented in the observed data 20, the inference module 3 can infer the description of the initial state 51 in the problem description 5 from the observed data 20.

[0032] The instruction information 21 can be anything that can identify the task's objective. The type of instruction information 21 is not particularly limited and may be appropriately selected depending on the embodiment. The instruction information 21 may consist of data from one or more modalities. The objective may be given as appropriate. The instruction information 21 may be acquired at any time before the task execution begins. The instruction information 21 may be acquired before or after the observation data 20. In addition, the instruction information 21 may be acquired at least partially in parallel with the observation data 20. Since the task's objective is expressed in the instruction information 21, the inference module 3 can infer the description of the objective state 52 in the problem description 5 from the instruction information 21.

[0033] Problem description 5 includes descriptions of the initial state and the target state (51, 52), so that, given the skills of the robot device R, the planner can generate an action plan for the robot device R, including its feasibility (whether or not it is feasible). As long as it is such, the configuration of problem description 5 is not particularly limited and may be appropriately selected depending on the embodiment. Since problem description 5 is independent of the robot device, it may be used to generate an action plan for any robot device. The planner may be appropriately configured to generate an action plan from problem description 5.

[0034] As shown in FIG. 1, in one example, the skills of the robot device R may be given by the domain description 23. Also, in one example, the planner may include a symbolic planner 61 and a motion planner 65. The symbolic planner 61 may be configured to generate an action sequence 63, which is a series of abstract actions, from the problem description 5 and the domain description 23. The abstract actions may be any collection of actions that include one or more actions of the robot device R and may be defined as a collection of actions that can be expressed by a desire (e.g., words, etc.). The abstract actions may be defined, for example, as a collection of meaningful (i.e., understandable by humans) actions such as grasping an object, carrying it, positioning it, etc. The symbolic planner 61 may use, for example, known symbolic planners such as Fast Downward of Reference 1 (“The Fast Downward Planning System”, [online], [searched on October 24, 2023], Internet <URL:https: / / planning.wiki / ref / planners / fd>), LAMA planner of Reference 2 (Silvia Richter et al., “The LAMA planner: guiding cost-based anytime planning with landmarks”, [online], [searched on October 24, 2023], Internet <URL:https: / / dl.acm.org / doi / 10.5555 / 1946417.<1946420>). For example, known symbolic planners such as the LAMA planner may be used.

[0035] The motion planner 65 may be configured to generate a sequence of control commands 67 for the robot device R to execute each action included in the motion sequence 63, given the motion sequence 63 by the symbolic planner 61. The motion planner 65 may also be configured to determine whether the robot device R can execute each action included in the motion sequence 63. For example, the motion planner 65 may be configured as described in Reference 3 (James J. Kuffner, Jr., Steven M. LaValle, “RRT-Connect: An Efficient Approach to Single-Query Path Planning”, [online], [Retrieved October 24, 2023], Internet).<URL:https: / / www.cs.cmu.edu / afs / cs / academic / class / 15494-s12 / readings / kuffner_icra2000.pdf> ) RRT-connect, Reference 4 (Nathan Ratliff et al., “CHOMP: Gradient Optimization Techniques for Efficient Motion Planning”, [online], [Retrieved October 24, 2023], In Tarnet<URL:https: / / www.ri.cmu.edu / pub_files / 2009 / 5 / icra09-chomp.pdf> CHOMP (Covariant Hamiltonian Optimization for Motion Planning), Reference 5 (Mrinal Kalakrishnan et al., “STOMP: Stochastic Trajectory Optimization for Motion Planning”, [online], [Retrieved October 24, 2023], Internet) <URL:http: / / ros.fei.edu.br / roswiki / attachments / Papers(2f)ICRA2011_Kalakrishnan / kalakrishnan_icra2011.pdf> A known motion planner such as STOMP may be used. The generated control command 67 may be given to the robot device R as appropriate. The robot device R may start performing the task at any timing according to the given control command 67. Note that the configuration of the planner is not limited to the example in Figure 1 and may be changed as appropriate depending on the embodiment. In another example, the planner may be configured to generate an action plan by directly inferring the control command 67 from the problem description 5 and the domain description 23.

[0036] The format of the problem description 5 is not particularly limited and may be appropriately selected depending on the embodiment, as long as it includes descriptions of the initial state and the target state (51, 52) and is usable for generating an action plan. In one example, the generated problem description 5 may be configured to follow a predetermined format. The predetermined format may be, for example, PDDL (Planning Domain Definition Language), PDDLStream (Reference 6: “pddlstream”, [online], [Retrieved October 24, 2023], Internet).<URL:https: / / github.com / caelan / pddlstream> ) etc. This may be provided by a language for planners (planning description language). In one example of this embodiment, the generated problem description 5 conforms to a predetermined format, making it easier for a planner (e.g., symbolic planner 61) to generate an action plan from the problem description 5.

[0037] In another example, the generated problem description 5 does not have to follow a specific format. In this case, the generated problem description 5 may be converted to conform to a predetermined format by intermediate processing such as a conversion program. Alternatively, the generated problem description 5 may be used to generate an action plan without conforming to a predetermined format by a planner that can accept input in any format, such as a planner composed of a pre-trained model of in-context learning.

[0038] The input and output data of the inference module 3 may be modified as appropriate depending on the embodiment. The inference module 3 may also accept input of arbitrary data other than the observation data 20 and instruction information 21. Furthermore, the inference module 3 may also output arbitrary data other than the problem description 5. The problem description 5 may also include arbitrary information other than the descriptions of the initial state and the target state (51, 52).

[0039] (Example of input / output data) Figure 2 schematically shows an example of the input and output of the inference module 3 according to this embodiment. In the example in Figure 2, the input data includes observation data 20, instruction information 21, and environmental information 22. The problem description 5 (output data) includes descriptions of the object, initial state, and target state (50, 51, 52).

[0040] In one example, the environment may be observed by one or more sensors S, and accordingly, the observation data 20 may consist of sensing data from one or more sensors S. The type of sensor S is not particularly limited and may be appropriately selected depending on the embodiment. Sensor S may include, for example, a camera, depth sensor, infrared sensor, optical sensor, radar, LiDAR (Light Detection And Ranging), microphone, position sensor, and other measuring sensors. The position sensor may be, for example, a GPS (Global Positioning System) sensor, a GNSS (Global Navigation Satellite System) sensor, etc. Sensing data may consist of, for example This includes image data, depth data, infrared data, optical sensor measurement data (marker detection results, etc.), radar data, LiDAR data, sound data, position data, and other measurement data. That's fine. Also, sensor S may include one or more measuring sensors for measuring the state of the robot device R. Measuring sensors may include, for example, encoders, motion capture devices, tactile sensors, force sensors, inertial measurement units, etc. Accordingly, the sensing data may include measurement data of the robot device R (e.g., joint angles, end-effector position, tactile data at the end-effector, force data at the end-effector, posture measurement data, etc.). The sensor S may be located outside the robot device R or located inside the robot device R.

[0041] In one example, the instruction information 21 may consist of language information that instructs the target in natural language. The natural language instruction may be acquired by any method. The natural language instruction may be acquired by methods such as text input, voice input, image input, or other inputs. The data format of the instruction information 21 (language information) may be appropriately selected depending on the embodiment. The instruction information 21 may consist of, for example, text data, voice data, image data, or other data formats. The acquired natural language instruction data may be used as instruction information 21 as is, or it may be used as instruction information 21 after its data format has been converted. As an example of the latter, the instruction data may be converted by a conversion model, for example, from voice to text, or from text to voice, and the converted instruction data may be used as instruction information 21. The conversion may include any analysis process such as voice analysis. The conversion model may be included in the inference module 3, or it may be prepared separately from the inference module 3. The conversion model may consist of at least one of a trained model (trained machine learning model) and a rule-based model. Instructions in natural language may be given manually by an operator or other person, or they may be given automatically by computer processing. Instruction information 21 may be given as appropriate depending on the task. In one example, instruction information 21 may be given when the task is given. As in the case of "every," it may be given each time. In another example, if the target is predetermined, such as the Tower of Hanoi, the instruction information 21 may be given. In one example of this embodiment, when the environment is observed by the sensor S and the task target is given by verbal instruction, a highly explainable output can be obtained regarding the control of the robot device R.

[0042] The configuration of the observation data 20 and the instruction information 21 may be appropriately modified depending on the embodiment. The observation data 20 may include, along with the sensing data, any data other than sensing data, such as data provided manually or data generated by computer processing. In another example, the observation data 20 may not include sensing data and may consist of any data other than sensing data. In yet another example, the instruction information 21 may be provided in a format other than natural language. For example, the instruction information 21 may be provided in text format using symbols other than natural language. Also, for example, the instruction information 21 may consist of an image showing the target state of an object present in the environment. The image may consist of at least one of a real image and a virtual image.

[0043] In one example, the information processing device 1 may be configured to further acquire environmental information 22 related to the environment. Generating a problem description 5 from the acquired observation data 20 and instruction information 21 may be configured by generating the problem description 5 from the acquired observation data 20, instruction information 21, and environmental information 22. According to one example of this embodiment, the environmental information 22 can be used to restrict the environment in which the task is performed, thereby narrowing down the conditions for generating the problem description 5. That is, by further using environmental information 22 as an explanatory variable in addition to the observation data 20 and instruction information 21, the environment in which the task is performed can be more specifically identified. Therefore, an improvement in the accuracy of generating the problem description 5 can be expected.

[0044] Furthermore, the environmental information 22 may include any information relating to the robot device R that may be involved in the generation of the problem description 5. For example, the environmental information 22 may include at least one of the domain description 23 and the domain information 24.

[0045] Domain description 23 defines events common to all problems, including the skills of the robot device R. As described above, domain description 23 may be used together with problem description 5 to generate an action plan by the planner. If problem description 5 is given in a predetermined format such as PDDL or PDDLStream, domain description 23 may also be given in a predetermined format. For example, if PDDL is adopted, problem description 5 may be problem.pddl and domain description 23 may be domain.pddl. Also, in one example, domain description Description 23 defines the skills of the robotic device R (e.g., actions), and the environment present. The description may include a description defining the state of the target object (e.g., predicates), a description defining the type of target object (e.g., types), and a description to check compatibility with the planner (e.g., requirements). The state of the target object is defined by the robot device R. This may include the states that the target object can be in when the process is executed.

[0046] Domain information 24 may include arbitrary information that limits the domain from which the problem description 5 is generated. Domain information 24 may also be called domain knowledge. Domain information 24 may consist of arbitrary information about the environment other than the domain description 23. Domain information 24 may supplement the conditions for generating the problem description 5 together with the domain description 23. For example, domain information 24 may include attribute information 241 of objects present in the environment. Attribute information 241 may include, for example, the name and characteristics of the object. Characteristics may include, for example, appearance-related features such as color, shape, and size. Appearance-related features include, for example, a cutting board being round, a counter being black, etc. This makes it possible to narrow down the features of objects appearing in the observed data 20, and as a result, an improvement in the accuracy of inference on the observed data 20 can be expected. Also, for example, as will be described later, the inference module 3 may be configured to include a trained model for in-context learning. If this is done, the domain information 24 may include one or more input / output samples 243 for the trained model. For example, the output sample may be a ground truth sample of the portion of the problem description 5 that is generated by the trained model. The input sample may be, for example, a sample of the portion of the observation data 20, instruction information 21, and environmental information 22 that are input to the trained model in order to obtain the ground truth sample of the problem description 5.

[0047] The format in which the input data (observation data 20, instruction information 21, and environmental information 22) is provided to the inference module 3 is not particularly limited and may be determined as appropriate depending on the embodiment. In one example, the observation data 20, instruction information 21, and environmental information 22 may be input to the inference module 3 as is. In another example, preprocessing may be applied to at least one of the data from the observation data 20, instruction information 21, and environmental information 22, and the preprocessed data may be input to the inference module 3. Preprocessing may include any computational processing such as processing to analyze information, processing to add information, or processing to reduce information. The computational model that performs the preprocessing may be included in the inference module 3 or may be prepared separately from the inference module 3. The computational model may consist of at least one of a trained model (trained machine learning model) and a rule-based model. For example, in the case where the observation data 20 consists of image data, the inference module 3 may be configured to accept image data as input. Alternatively, the image data may be analyzed by any method such as image processing or an analysis model. This analysis process is an example of preprocessing. The analysis results may include, for example, bounding box detection results, object identification results in the image data, etc. The inference module 3 may be configured to accept analysis results as input for the image data.

[0048] In one example, the generated problem description 5 may further include a description 50 of objects present in the environment. The object description 50 may correspond to a list of objects present in the environment. The scope of objects included in the description 50 may be determined as appropriate depending on the embodiment. For example, the object description 50 may consist of a list of all objects that can be observed in the target environment. Alternatively, for example, the object description 50 may consist of a list of some objects that can be observed. Some objects may be objects of interest, such as objects that can be involved in the task. In this case, objects of no interest (e.g., objects that cannot be involved in the task) may be omitted in the object description 50. According to one example of this embodiment, the object description 50 makes it easier to identify objects present in the environment, thereby making it easier for the planner (e.g., symbolic planner 61) to generate an action plan from the problem description 5. Note that if the descriptions of the initial state and target state (51, 52) are obtained in accordance with a predetermined format, the object description 50 may also be obtained in accordance with a predetermined format. For example, when using PDDL, the description of objects in problem.pddl is an example of an object description 50, the description of init (initial state) is an example of an initial state description 51, and the description of goal is an example of a target state description 52.

[0049] Note that the input / output configuration of the inference module 3 is not limited to the example in Figure 2 and may be modified as appropriate depending on the embodiment. At least a portion of the environmental information 22 may be omitted. In problem description 5, the object description 50 may be omitted. For example, when the robot device R is driven in an environment where no other objects exist besides the robot device R (for example, when a drone flies in the air where nothing exists), the object description 50 may be omitted in problem description 5.

[0050] [Revision of the problem description] For example, the information processing device 1 may be configured to perform processing related to modifying the generated problem description 5. The processing to modify the problem description 5 may be performed automatically by a computer or manually by an operator or other person.

[0051] As an example, the information processing device 1 may determine whether the generated problem description 5 fits a planner (e.g., a symbolic planner 61). The determination is made using a trained model and This may be done by any method, such as using at least one of the rule-based models. For example, if the problem description 5 is generated to conform to a predetermined format, the information processing device 1 may determine whether the generated problem description 5 conforms to the predetermined format by evaluating whether it conforms to the predetermined format. If it is determined to conform to the planner, the generated problem description 5 may be provided to the planner as appropriate. On the other hand, if it is determined that the generated problem description 5 does not conform to the planner, the information processing device 1 may modify the generated problem description 5 as appropriate. For example, the information processing device 1 may modify the generated problem description 5 using a modification model. The modification model may consist of at least one of a trained model and a rule-based model. Also, for example, the information processing device 1 may output the problem description 5 to an output device along with factors that make it unsuitable for the planner (for example, indicating parts that do not conform to the predetermined format). The information processing device 1 may accept modifications to the problem description 5 from an operator via an input device. The information processing device 1 may then modify the problem description 5 according to the received content. As another example, the information processing device 1 may omit the process of determining whether or not it conforms to the planner and then perform the above process of modifying the problem description 5.

[0052] The timing for executing the process of modifying problem description 5 is not particularly limited and may be determined as appropriate depending on the embodiment. In one example, the information processing device 1 may execute the process of modifying problem description 5 before the planner (e.g., symbolic planner 61) uses problem description 5 to generate an action plan. In another example, after the generated problem description 5 has been given to the planner, the information processing device 1 may execute the process of modifying problem description 5 in response to an error occurring in the planner's attempt to generate an action plan, such as outputting an error message or being unable to generate a proper action plan. Note that the process of modifying problem description 5 may be executed by an external computer other than the information processing device 1. Furthermore, the process of modifying problem description 5 may be omitted.

[0053] Figure 3 schematically shows an example of the modification process of problem description 5 according to this embodiment. In one example, the inference module 3 may be configured to include a trained model 39 for in-context learning. In-context learning is the acquisition of the ability to perform specific inferences through the context of inputs (prompts), such as input / output samples. In one example, the trained model 39 can acquire the ability for in-context learning by including a self-attention mechanism and an autoregressive model. This trained model 39 may be, for example, a Large Vision-Language Model (LLM), a Large Vision-Language Model (LVLM), etc. A Large Vision-Language Model may include a visual question answering model, an open vocabulary object detection model, an open vocabulary object segmentation model, etc. In addition, the trained model 39 may be a Large Vision-Language Model combined with one or more other modalities (sound, etc.), such as Audio Question Answering. It may also be Dell. Furthermore, the trained model 39 may be, for example, Large Audio Model. It may also be a large-scale model of one or more modalities other than language. The input data format for the trained model 39 may be appropriately selected depending on the embodiment.

[0054] In the example in Figure 3, the inference module 3 includes a trained model 39 with the ability to learn within context, so that if an error occurs, it can adapt to the error and modify the problem description 5 accordingly. In this example in Figure 3, the information processing device 1 may generate the problem description 5 by providing input data 200 to the inference module 3 and executing the calculations of the inference module 3. The input data 200 includes observation data 20 and instruction information 21 in the target scene for which an action plan is to be generated. In one example, the input data 200 may further include environmental information 22.

[0055] The generated problem description 5 may be given to the planner as appropriate. The planner will then consider the given problem Using problem description 5, an attempt may be made to generate an action plan. If the planner has the configuration shown in Figure 1, the generated problem description 5 may be given to the symbolic planner 61, which may attempt to generate an action sequence 63 from the problem description 5. The process of generating an action plan by this planner may be executed by at least one of the information processing device 1 and an external computer other than the information processing device 1. As described above, the domain description 23 may be used together with the problem description 5 to generate the action plan.

[0056] If problem description 5 is correct, the planner can generate a correct action plan from problem description 5. In the example in Figure 1, a correct motion sequence 63 is obtained, and accordingly, the motion planner 65 can generate a sequence of control commands 67. The generated action plan may be used as appropriate for the motion control of the robot device R. On the other hand, if problem description 5 is incorrect, an error occurs in the generation of the action plan by the planner. An incorrect problem description 5 may include, for example, that problem description 5 is not compatible with the planner, that it is not possible to generate an action plan while avoiding prohibited items, or that an action plan cannot be generated with the given skills of the robot device R. An incorrect problem description 5 may include, for example, that problem description 5 does not follow a predetermined format. In this case, the planner outputs an error message 615.

[0057] In the process of generating an action plan for the robot device R by providing the generated problem description 5 to the planner, if the planner outputs an error message 615, the information processing device 1 may acquire the outputted error message 615. The structure of the error message 615 is not particularly limited and may be appropriately determined according to the embodiment, such as the configuration of the planner. The information processing device 1 may then provide the acquired error message 615 and problem description 5 to the inference module 3 and execute the calculation process of the inference module 3 again. That is, the information processing device 1 may use the problem description 5 and error message 615 as a re-prompt and execute the process of generating the problem description 5 again. In this way, the information processing device 1 may use the inference module 3 to generate a new problem description 5 from the problem description 5 and error message 615 (i.e., modify the problem description 5). According to one example of this embodiment, if an appropriate problem description 5 cannot be obtained, the problem description 5 can be automatically modified.

[0058] The data provided to the inference module 3 during re-prompting is not limited to the problem description 5 and the error message 615. The configuration of re-prompting may be modified as appropriate depending on the embodiment. In another example, the information processing device 1 may provide the inference module 3 with at least a portion of the input data 200 along with the problem description 5 and the error message 615. This is expected to improve the accuracy of correcting the problem description 5.

[0059] Furthermore, the information processing device 1 may recursively repeat the process of correcting the problem description 5 by the re-prompting described above. The number of repetitions is not particularly limited and may be determined as appropriate depending on the embodiment. In one example, the information processing device 1 may repeatedly execute the process of correcting the problem description 5 by the re-prompting until the error message 615 is no longer output. In another example, the number of times the process of correcting the problem description 5 by the re-prompting is executed may be predetermined. If the error message 615 is still output after the correction process by the re-prompting has been executed a predetermined number of times, the information processing device 1 may stop the correction process by the re-prompting and output the execution results up to that point (for example, the generated problem description 5, etc.).

[0060] Furthermore, the method of re-prompting is not limited to the above example and may be modified as appropriate depending on the embodiment. In another example, the information processing device 1 may modify the problem description 5 by using a prompt modified in response to the error message 615 as a re-prompt, together with or in place of the problem description 5 and the error message 615. For example, if the error message 615 is output, the information processing device 1 uses the modified model to output the error message In response to the message 615, the prompt (input data 200) given to the inference module 3 may be modified as appropriate. The modified model may be composed of at least one of a trained model and a rule-based model as appropriate. The information processing device 1 may then give the modified prompt to the inference module 3 again and generate a new problem description 5 by executing the calculation process of the inference module 3.

[0061] [Inference Module] The inference module 3 is configured to perform an inference process that generates a problem description 5 from input data including observation data 20 and instruction information 21. The configuration of the inference module 3 is not particularly limited as long as it can perform such an inference process, and may be determined as appropriate depending on the embodiment. In one example, the inference module 3 may consist of at least one of a rule-based model and a trained model (a trained machine learning model).

[0062] A rule-based model is configured to derive an inference result (in this embodiment, the generation result of problem description 5) from a given input according to rules. The rules may be set as appropriate. A machine learning model is configured to have one or more computational parameters that can be adjusted by machine learning. One or more computational parameters are used for the computation of the desired inference (in this embodiment, the generation of problem description 5). The machine learning model may consist of, for example, a neural network, a regression model, a decision tree model, a support vector machine, or other functional expressions (computational models). The machine learning method may be selected as appropriate depending on the machine learning model adopted (for example, backpropagation).

[0063] Machine learning involves adjusting (optimizing) the values ​​of computational parameters using training samples. Typically, a trained model may be generated by supervised learning using multiple training datasets, each composed of a combination of input samples (training samples) and output samples (teacher signals, labels). For example, input samples may be samples of input data (observation data 20, instruction information 21, etc.), and output samples may be samples of output data (problem description 5). In supervised learning, the values ​​of the computational parameters of the machine learning model may be adjusted so that the output obtained from the machine learning model fits the corresponding output sample when given input samples. However, the method for generating a trained model is not limited to this example and may be appropriately modified depending on the embodiment. The training dataset is not limited to the above example and may be appropriately selected depending on the embodiment. For example, data other than those mentioned above may be used as the training dataset, such as when acquiring the ability to learn in context. Furthermore, the learning method is not limited to supervised learning; for example, other methods such as unsupervised learning (including self-supervised learning) and reinforcement learning may be used. The machine learning model may be trained online and offline, or at least both. Machine learning models may be appropriately tuned through methods such as transfer learning, retraining, and additional training. Additional training can be performed, for example, as described in Reference 7 ("LoRA", [online], [Retrieved October 24, 2023], Internet).<URL:https: / / github.com / microsoft / LoRA> ) LoRA (Low-Rank Adaptation of Large Language Models), reference text Reference 8 (Neil Houlsby et al., “Parameter-Efficient Transfer Learning for NLP”, [online], [Retrieved October 24, 2023], Internet)<URL:http: / / proceedings.mlr.press / v97 / houlsby19a / houlsby19a.pdf> ) Adapter, Reference 9 (Brian Lester et al.) al., “The Power of Scale for Parameter-Efficient Prompt Tuning”, [online], [ [Searched on October 24, 2023], Internet<URL:https: / / arxiv.org / abs / 2104.08691> This may include machine learning methods such as Prompt Tuning, which adds a module to an existing pre-trained model and adjusts the parameters of the added module while keeping the parameters of the existing trained model unchanged. Additional learning is also described in Reference 10 (Kecheng Zheng et al., “Regularized Mask Tuning: Uncovering Hidden Knowledge in Pre-trained Vision-Language Models”, [online], [Retrieved October 24, 2023], Internet). <URL:https: / / It may include machine learning techniques that adjust masks to exclude parts of a pre-trained model, such as Mask Tuning (openaccess.thecvf.com / content / ICCV2023 / papers / Zheng_Regularized_Mask_Tuning_Uncovering_Hidden_Knowledge_in_Pre-Trained_Vision-Language_Models_ICCV_2023_paper.pdf). When a pre-trained model capable of in-context learning, such as the large-scale language model mentioned above, is used in at least part of inference module 3, these additional training techniques may be used to specialize the trained model for specific problems, domains, or other situations.

[0064] In one example, inference module 3 may include a neural network. The structure of the neural network is not particularly limited and may be determined as appropriate depending on the embodiment. The structure of the neural network may be specified, for example, by the number of layers from the input layer to the output layer, the type of each layer, the number of nodes (neurons) included in each layer, and the connection relationships between the nodes in each layer. In one example, the neural network may include any mechanism such as a recursive structure, a self-attention mechanism, or an autoregressive model. The neural network may also include any layer such as a fully connected layer, a convolutional layer, a pooling layer, an inverse convolutional layer, an unpooling layer, a normalization layer, a dropout layer, or an LSTM (Long short-term memory). The neural network may include any type of model such as a diffusion model, a transformer model, or a generative model. The weights of the connections between each node included in the neural network and the thresholds of each node are examples of computational parameters.

[0065] As described above, in one example, the inference module 3 may include a pre-trained model 39 for in-context learning. In this case, the input data provided to the inference module 3 may include one or more input / output samples 243 for the pre-trained model 39 as domain information 24 of the environment information 22. The input / output samples 243 may be prepared as appropriate depending on the domain for which the problem description 5 is generated. In this example, by providing the input / output samples 243 to the pre-trained model 39, in-context learning can be performed and the pre-trained model 39 can be adapted to the target domain. That is, by preparing input / output samples 243 for each domain, the inference module 3 can generate problem descriptions 5 in a general manner without changing the pre-trained model 39. Therefore, according to this example, it can be expected that the generation of problem descriptions 5 for tasks in various domains can be handled.

[0066] Furthermore, in one example, if the inference module 3 includes a pre-trained model 39 for in-context learning, a domain description 23 may be provided for each type of robot device R. In addition, output samples (samples of problem description 5) corresponding to the domain description 23 may be provided as domain information 24 for each type. Accordingly, the information processing device 1 may accept a specification of the type. The specification method is not particularly limited and may be appropriately selected depending on the embodiment. Typically, the type may be specified by manual methods such as operation via an input device by an operator, text input, or voice input. The specification of the type may be included in the instruction information 21. The information processing device 1 may provide the domain description 23 and output samples (domain information 24) corresponding to the specified type as environment information 22 to the inference module 3 (pre-trained model 39). This makes it possible to generate a problem description 5 for generating an action plan for a robot device R having the specified type of skills. In other words, according to this example, it is possible to increase the versatility of the inference module 3 and improve the accuracy of generating a problem description 5 that fits the type.

[0067] Furthermore, the input and output configuration of the inference module 3 may be determined as appropriate depending on the embodiment. As described above, in one example, the input data (observation data 20, instruction information 21, and environmental information 22) may be provided to the inference module 3 as is. In another example, at least a portion of the input data may be subjected to arbitrary preprocessing before being provided to the inference module 3. Preprocessing may include computational processing such as feature extraction, other analysis, and inference. The inference module 3 may be provided with preprocessed input data. Also, in one example, the inference module The output of module 3 may be configured to directly show the inference result (problem description 5). In another example, the output of inference module 3 may be configured to indirectly show the inference result. In this case, the inference result may be obtained by performing arbitrary information processing (interpretation processing) on ​​the output of inference module 3. The generation of problem description 5 may be performed in real time on data obtained at the present time (which may include past data), or it may be performed retrospectively on data obtained in the past.

[0068] Furthermore, the inference module 3 may be configured as a single unit or as a set of multiple sub-elements. For example, when the inference module 3 is configured as a set of multiple sub-elements, each sub-element may be appropriately configured to generate a part (corresponding part) of the problem description 5. For example, one or more sub-elements may be prepared for each description of the object, initial state, and target state (50, 51, 52) of the problem description 5. Each sub-element may be composed of at least one of a machine learning model and a rule-based model.

[0069] (Example configuration of an inference module) Figure 4 schematically shows an example of the configuration of the inference module 3 according to this embodiment. In the example in Figure 4, the problem description 5 is assumed to include a description 50 of an object present in the environment. In one example, the inference module 3 may include an object estimator 31, an initial state estimator 33, and a target estimator 35. The object estimator 31 is an example of one or more subelements corresponding to the description 50 of an object. The initial state estimator 33 is an example of one or more subelements corresponding to the description 51 of the initial state. The target estimator 35 is an example of one or more subelements corresponding to the description 52 of the target state.

[0070] The object estimator 31 may be configured to generate descriptions 50 of one or more objects present in the environment from the acquired observation data 20. Generating the object descriptions 50 may include detecting the objects. Generating the problem description 5 using the inference module 3 may include generating the descriptions 50 of objects present in the environment from the acquired observation data 20 using the object estimator 31. The observation data 20 contains information about objects present in the environment. Therefore, according to one example of this embodiment, the object descriptions 50 in the problem description 5 can be appropriately generated.

[0071] The configuration of the object estimator 31 is not particularly limited and may be appropriately selected depending on the embodiment, as long as it is possible to generate an object description 50 from the observation data 20. The above description of the configuration of the inference module 3 may also apply to the object estimator 31. The object estimator 31 may be composed of at least one of a rule-based model and a trained model.

[0072] The object estimator 31 may use a model capable of inferring object descriptions 50 only in a specific domain. However, in order to be commonly usable for tasks in various domains, it is desirable to use a model capable of inferring object descriptions 50 in a general way, rather than a model capable of inferring object descriptions 50 only in such a specific domain, as the object estimator 31. For example, the object estimator 31 may have a pre-trained model that performs in-context learning. By having a pre-trained model that has the ability to perform in-context learning, the versatility of the object estimator 31 can be increased. That is, it can infer descriptions 50 of objects present in the environment in a general way, and this can be expected to enable the generation of object descriptions 50 for tasks in various domains.

[0073] The input / output configuration of the object estimator 31 may be determined as appropriate depending on the embodiment. The object estimator 31 may also accept input of arbitrary data other than the observation data 20. The input data may be given to the object estimator 31 as is, or it may be given to the object estimator 31 after preprocessing has been applied. The object estimator 31 may further accept arbitrary data other than the object description 50. You can output it there.

[0074] In one example, the information processing device 1 may further acquire environmental information 22 related to the environment. The object estimator 31 may further use the acquired environmental information 22 to generate an object description 50. That is, generating an object description 50 from acquired observation data 20 may be configured by generating an object description 50 from acquired observation data 20 and environmental information 22. According to one example of this embodiment, by further using environmental information 22, it is possible to expect an improvement in the accuracy of generating the object description 50 in the problem description 5.

[0075] The environmental information 22 provided to the object estimator 31 may be appropriately selected depending on the embodiment. The environmental information 22 may include at least one of the domain description 23 and domain information 24. For example, the environmental information 22 may include at least a part of the domain description 23. Also, for example, the environmental information 22 may include attribute information 241 of objects present in the environment as domain information 24. This makes it possible to more accurately identify objects present in the environment using attribute information 241, and thus improves the accuracy of generating the object description 50 in the problem description 5. Also, for example, if the object estimator 31 has a trained model for in-context learning, the environmental information 22 may include one or more input / output samples 243 as domain information 24. The input samples may consist of samples of the observed data 20. If environmental information 22 other than the input / output samples 243 is provided to the object estimator 31, the input samples may include samples of that environmental information 22. The output samples may consist of ground truth samples of the object description 50 corresponding to the input samples.

[0076] The domain description 23 and domain information 24 (attribute information 241, input / output samples 243) may be prepared as appropriate depending on the embodiment. In one example, the domain description 23 and domain information 24 may be prepared in advance. The domain description 23 and domain information 24 may be held in the memory resources of the information processing device 1, or they may be provided to the information processing device 1 from an external source when used. In another example, at least a part of the domain description 23 and domain information 24 may be generated as appropriate when used. For example, reference information including a list of objects and attribute information may be prepared in advance. The information processing device 1 may extract candidate objects present in the environment from at least one of the instruction information 21 (e.g., language information) and the domain description 23 (e.g., a description defining the type of target object). The information processing device 1 may compare the extracted list of candidate objects with the list of reference information and generate attribute information 241 to be given to the object estimator 31 by extracting attribute information of objects that match the candidate objects from the reference information. The process of generating this attribute information 241 may be performed as a preprocessing step or as a process within the inference module 3.

[0077] The initial state estimator 33 may be configured to generate a description 51 of the initial state of one or more objects present in the environment from the acquired observation data 20. Generating the problem description 5 using the inference module 3 may include generating the description 51 of the initial state of objects present in the environment using the initial state estimator 33. The observation data 20 contains the initial state of objects present in the environment. Therefore, according to one example of this embodiment, the description 51 of the initial state in the problem description 5 can be appropriately generated.

[0078] The configuration of the initial state estimator 33 is not particularly limited and may be appropriately selected depending on the embodiment, as long as it is possible to generate a description 51 of the initial state from the observed data 20. The above description of the configuration of the inference module 3 may also apply to the initial state estimator 33, similar to the object estimator 31. The initial state estimator 33 may be composed of at least one of a rule-based model and a trained model.

[0079] The initial state estimator 33 is a model that can infer the initial state description 51 only in a specific domain. A model can be used. However, in order to be commonly usable for tasks in various domains, it is desirable to use a model that can infer the initial state description 51 in a general way as the initial state estimator 33, rather than a model that can infer the initial state description 51 only in such a specific domain. For example, the initial state estimator 33 may have a pre-trained model that performs in-context learning. By having a pre-trained model that has the ability to perform in-context learning, the generality of the initial state estimator 33 can be increased. That is, it can infer the initial state description 51 of objects present in the environment in a general way, and this can be expected to enable the generation of initial state descriptions 51 for tasks in various domains.

[0080] The input / output configuration of the initial state estimator 33 may be determined as appropriate depending on the embodiment, similar to the object estimator 31. The initial state estimator 33 may also accept input of arbitrary data other than the observed data 20. The input data may be given to the initial state estimator 33 as is, or it may be given to the initial state estimator 33 after preprocessing has been applied. The input data may be given to the object estimator 31 first, and the calculation results of the object estimator 31 on the input data (including intermediate calculation results) may be given to the initial state estimator 33. For example, providing the observed data 20 to the initial state estimator 33 may include providing the observed data 20 to the initial state estimator 33 as is, providing the preprocessed results of the observed data 20 to the initial state estimator 33, and providing the calculation results of the object estimator 31 on the observed data 20 to the initial state estimator 33. The preprocessing results and the calculation results of the object estimator 31 may be, for example, the results of object detection in the observed data 20. However, the order in which the input data is provided is not limited to this example. The input data may be provided to the initial state estimator 33 first, and the result of the calculation performed by the initial state estimator 33 on the input data may be provided to the object estimator 31. For example, providing the observation data 20 to the object estimator 31 may include providing the observation data 20 to the object estimator 31 as is, providing the object estimator 31 with the preprocessed result of the observation data 20, and providing the object estimator 31 with the result of the calculation performed by the initial state estimator 33 on the observation data 20. In addition, the initial state estimator 33 may output any data other than the description 51 of the object's initial state.

[0081] In one example, similar to the object estimator 31, the initial state estimator 33 may further use the environment information 22 to generate a description of the initial state 51. That is, generating a description of the initial state 51 from the observed data 20 may be performed by generating a description of the initial state 51 from the observed data 20 and the environment information 22. The environment information 22 provided to the initial state estimator 33 may be appropriately selected depending on the embodiment. The environment information 22 may include at least one of the domain description 23 and domain information 24. For example, the environment information 22 may include at least a part of the domain description 23. Also, for example, the environment information 22 may include attribute information 241 as domain information 24. Also, for example, if the initial state estimator 33 has a trained model for in-context learning, the environment information 22 may include one or more input / output samples 243 as domain information 24. The input samples may consist of samples of the observed data 20 (including preprocessing results and calculation results of the object estimator 31). If environmental information 22 other than the input / output samples 243 is provided to the initial state estimator 33, the input samples may include samples of that environmental information 22. The output samples may consist of correct samples of the initial state description 51 corresponding to the input samples. The environmental information 22 (domain description 23, domain information 24) may be prepared as appropriate depending on the embodiment. By further using the environmental information 22, it is possible to expect an improvement in the accuracy of generating the initial state description 51 in the problem description 5.

[0082] In one example, the calculation process of the object estimator 31 may be completed before that of the initial state estimator 33. In this case, the description of the object 50 generated by the object estimator 31 may be given to the initial state estimator 33 as input data for generating the description of the initial state 51. That is, the initial state estimator 33 receives the description of the object 50 generated by the object estimator 31. The initial state description 51 may be generated by further using 0. If the initial state estimator 33 has a trained model for in-context learning and the environmental information 22 provided to the initial state estimator 33 includes input / output samples 243, the input samples may include samples of the object description 50. However, the order of operations within the inference module 3 is not limited to this example. The operations of the initial state estimator 33 may be executed at least partially in parallel with the object estimator 31, or they may be executed to completion before the object estimator 31. If the operations of the initial state estimator 33 are executed to completion before the object estimator 31, the initial state description 51 generated by the initial state estimator 33 may be provided to the object estimator 31. If the object estimator 31 has a trained model for in-context learning and the environmental information 22 provided to the object estimator 31 includes input / output samples 243, the input samples may include samples of the initial state description 51.

[0083] The target estimator 35 may be configured to generate a description 52 of the target state of one or more objects present in the environment from the acquired instruction information 21. Generating the problem description 5 using the inference module 3 may include generating the description 52 of the target state of objects present in the environment using the target estimator 35. The instruction information 21 contains information about the task's objective. Therefore, according to one example of this embodiment, the description 52 of the target state in the problem description 5 can be appropriately generated.

[0084] The configuration of the target estimator 35 is not particularly limited and may be appropriately selected depending on the embodiment, as long as it is possible to generate a description 52 of the target state from the instruction information 21. The above description of the configuration of the inference module 3 may also apply to the target estimator 35, similar to the object estimator 31, etc. The target estimator 35 may be composed of at least one of a rule-based model and a trained model.

[0085] The target estimator 35 may use a model capable of inferring a description of the target state 52 only in a specific domain. However, in order to be commonly usable for tasks in various domains, it is desirable to use a model capable of inferring a description of the target state 52 in a general way, rather than a model capable of inferring a description of the target state 52 only in such a specific domain, as the target estimator 35. For example, the target estimator 35 may include a pre-trained model that performs in-context learning. By including a pre-trained model capable of performing in-context learning, the versatility of the target estimator 35 can be increased. That is, it can infer a description of the target state 52 of an object present in the environment in a general way, and this can be expected to enable the generation of a description of the target state 52 for tasks in various domains.

[0086] The input / output configuration of the target estimator 35 may be appropriately determined according to the embodiment, similar to the object estimator 31, etc. The target estimator 35 may also accept input of arbitrary data other than the instruction information 21. The input data may be given to the target estimator 35 as is, or it may be given to the target estimator 35 after preprocessing has been applied. Similar to the relationship between the object estimator 31 and the initial state estimator 33, the calculation result (including intermediate calculation results) of at least one of the object estimator 31 and the initial state estimator 33 may be given to the target estimator 35. Alternatively, the calculation result (including intermediate calculation results) of the target estimator 35 may be given to at least one of the object estimator 31 and the initial state estimator 33. Furthermore, the target estimator 35 may output arbitrary data other than the description 52 of the target state of the object.

[0087] In one example, similar to the object estimator 31, the target estimator 35 may further use the environmental information 22 to generate a description of the target state 52. That is, generating a description of the target state 52 from the instruction information 21 may be configured by generating a description of the target state 52 from the instruction information 21 and the environmental information 22. The environmental information 22 provided to the target estimator 35 may be appropriately selected depending on the embodiment. The environmental information 22 is the domain description 23 and the domain information 24. It may include at least one of the following. For example, the environment information 22 may include at least a part of the domain description 23. Also, for example, the environment information 22 may include attribute information 241 as domain information 24. Also, for example, if the target estimator 35 has a trained model of in-context learning, the environment information 22 may include one or more input / output samples 243 as domain information 24. The input samples may consist of samples of instruction information 21. If environment information 22 other than the input / output samples 243 is given to the target estimator 35, the input samples may include samples of that environment information 22. The output samples may consist of correct samples of the target state description 52 corresponding to the input samples. The environment information 22 (domain description 23, domain information 24) may be prepared as appropriate depending on the embodiment. By further using the environment information 22, it is possible to expect an improvement in the accuracy of generating the target state description 52 in the problem description 5. The environmental information 22 provided to the object estimator 31, the initial state estimator 33, and the target estimator 35 may or may not overlap at least partially.

[0088] In one example, the calculation process of the object estimator 31 may be completed before that of the target estimator 35. In this case, the object description 50 generated by the object estimator 31 may be provided to the target estimator 35 as input data for generating the target state description 52. Similarly, the calculation process of the initial state estimator 33 may be completed before that of the target estimator 35. In this case, the initial state description 51 generated by the initial state estimator 33 may be provided to the target estimator 35 as input data for generating the target state description 52. That is, the target estimator 35 may further use at least one of the object description 50 and the initial state description 51 generated by the object estimator 31 and the initial state estimator 33 to generate the target state description 52. If the target estimator 35 has a trained model for in-context learning and the environmental information 22 provided to the target estimator 35 includes input / output samples 243, the input samples may include at least one of the object description 50 and the initial state description 51. However, the order of operations within the inference module 3 is not limited to these examples. The operation of the target estimator 35 may be performed at least partially in parallel with the object estimator 31, or it may be performed to completion before the object estimator 31. If the operation of the target estimator 35 is performed to completion before the object estimator 31, the description of the target state 52 generated by the target estimator 35 may be provided to the object estimator 31. Similarly, the operation of the target estimator 35 may be performed at least partially in parallel with the initial state estimator 33, or it may be performed to completion before the initial state estimator 33. If the operation of the target estimator 35 is performed to completion before the initial state estimator 33, the description of the target state 52 generated by the target estimator 35 may be provided to the initial state estimator 33. If at least one of the object estimator 31 and the initial state estimator 33 has a trained model for in-context learning, and the given environment information 22 includes input / output samples 243, the input samples may include samples of the description of the target state 52.

[0089] The configuration of the inference module 3 is not limited to this example and may be modified as appropriate depending on the embodiment. In another example, at least one combination of the object estimator 31, the initial state estimator 33, and the target estimator 35 may be configured as a single unit. At least one of the object estimator 31, the initial state estimator 33, and the target estimator 35 may be omitted as appropriate. Also, in the example of Figure 4, the inference module 3 including the in-context trained model 39 may be configured by having at least one of the object estimator 31, the initial state estimator 33, and the target estimator 35 include the in-context trained model. If the inference module 3 includes the in-context trained model 39 and the environmental information 22 provided to the inference module 3 includes input / output samples 243, the information processing device 1 may use the input / output samples 243 as few-shot prompting. For example, the information processing device 1 may provide only the input / output samples 243 to the trained model 39 before generating the problem description 5, or it may provide the input / output samples 243 along with other input data when generating the problem description 5. The trained model 39 may be given this information. The information processing device 1 may then perform calculations on the trained model 39. In this way, the information processing device 1 may adapt the trained model 39 to a domain where in-context learning is performed and the problem description 5 is generated.

[0090] (Specific examples of object estimators) Figure 5A schematically shows an example of an object estimator 31 according to this embodiment. In one example, the object estimator 31 may include a trained model 311 and a rule-based model 313. The trained model 311 may be configured to detect objects appearing in the observed data 20 using the environmental information 22 as a clue when observed data 20 and environmental information 22 are input, and to output an object detection result 201. The trained model 311 may be a trained model that has the ability to perform in-context learning.

[0091] For example, sensor S may be a camera, and observation data 20 may be image data. Environmental information 22 may be expressed in language, such as "white bowl" or "round cutting board". This may be a list of attribute information 241 indicating the attributes of an object. The trained model 311 is referenced in Reference 11 (Shilong Liu et al., “Grounding DINO: Marrying DINO with grounded pre-training for open-set object detection”, [online], [Retrieved October 24, 2023]). internet<URL:https: / / arxiv.org / abs / 2303.05499> ), Reference 12 (Alexander Kirillov et al., “Segment Anything”, [online], [Retrieved October 24, 2020], Internet<URL:https: / / arxiv.org / abs / 2304.02643> This may be an open vocabulary object detection model or an open vocabulary object segmentation model as proposed in (etc.). The detection result 201 may include at least one of the results of detecting the range in which the object is captured as a bounding box and the results of identifying the object. The rule-based model 313 may be configured to generate an object description 50 from the object detection result 201 by the trained model 311 according to the rules. The rules may be set as appropriate.

[0092] Note that the configuration shown in Figure 5A is merely one example of the object estimator 31. The configuration of the object estimator 31 is not limited to the example shown in Figure 5A and may be modified as appropriate depending on the embodiment. In another example, the trained model 311 may be provided with environmental information 22 in addition to the attribute information 241. The observation data 20 may further include data in a format other than image data, or it may consist only of data in a format other than image data. The trained model 311 and the rule-based model 313 may be configured as a single unit. The rule-based model 313 may be replaced by the trained model. The trained model 311 may be configured to generate the object description 50. Accordingly, the rule-based model 313 may be omitted.

[0093] (Specific example of an initial state estimator) Figure 5B schematically shows an example of an initial state estimator 33 according to this embodiment. In one example, the initial state estimator 33 may include a detector 331 and a trained model 333. The information processing device 1 may extract partial data 205 corresponding to each object from the observed data 20 according to the object detection result 201. The detector 331 may be configured to generate a caption 206 for each object from the extracted partial data 205 for each object. The caption 206 for each object may be configured to include information about the object that appears in the partial data 205. The trained model 333 may be configured to generate a description 51 of the initial state of the object from the object detection result 201 and the caption 206. The use of the object detection result 201 by the trained model 333 is an example of the initial state estimator 33 using the calculation result of the object estimator 31 as observed data 20. The detector 331 and the trained model 333 may each be trained models capable of performing in-context learning.

[0094] For example, partial data 205 is the portion extracted by the bounding box of each object. Image data is acceptable. Detector 331 is referenced in Reference 13 (Junnan Li et al., “BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large "Language Models", [online], [Retrieved October 24, 2023], Internet<URL:https: / / arxiv.org / abs / 2301.12597> This may be a visual question answering model as proposed in (etc.). The detector 331 is appropriately provided with the question text along with the partial data 205. It is also possible. For example, the question may consist of predetermined text data that asks about an object shown in a partial image (e.g., "Q: what does this object describe? A: ."). Model 333 is based on reference 14 (OpenAI, “GPT-4 Technical Report”, [online], [Reiwa]). [Searched on October 24, 1930], Internet<URL:https: / / arxiv.org / pdf / 2303.08774.pdf> ) may be a large-scale language model (LLM) as proposed in ) etc. Accordingly, the caption 206 may be text data that describes the partial data 205 (partial image data). The object detection result 201 may be appropriately converted into text data 202, and the obtained text data 202 may be given to the trained model 333 as the detection result 201 (observation data 20). In one example, the text data 202 may consist of the name of the detected object (identification result) and text indicating the coordinate range of the object's bounding box. The trained model 333 may also be given an object description 50 generated by the object estimator 31. Furthermore, input / output samples 243 may be given to the trained model 333 as environmental information 22 (domain information 24). In one example, the input samples may consist of samples of text data 202, caption 206, and object description 50, and the output samples may consist of ground truth samples of the corresponding initial state description 51. Input / output samples 243 may be used for few-shot prompting of the trained model 333.

[0095] Note that the configuration shown in Figure 5B is merely one example of the initial state estimator 33. The configuration of the initial state estimator 33 is not limited to the example in Figure 5B and may be modified as appropriate depending on the embodiment. In another example, the trained model 333 may be provided with environmental information 22 other than the input / output samples 243. In yet another example, the trained model 333 may have an input section that accepts input of data from modalities other than text, such as a large-scale visual language model. This allows the trained model 333 to be configured to accept input of data from other modalities. Accordingly, the observed data 20 or detection result 201 may be provided directly to the trained model 333. The caption 206 may consist of data in a format other than text data. Furthermore, the partial data 205 (observed data 20) may further include data in a format other than image data, or it may consist only of data in a format other than image data. The data format of the detection result 201 is not particularly limited and may be selected as appropriate depending on the embodiment. The detector 331 may be omitted.

[0096] (Specific example of a target estimator) Figure 5C schematically shows an example of a target estimator 35 according to this embodiment. In one example, the target estimator 35 may include a trained model 351. The trained model 351 may be configured to generate a description 52 of the target state of an object from instruction information 21. The trained model 351 may have the ability to perform in-context learning.

[0097] For example, the trained model 351 may be a large-scale language model (LLM) as proposed in reference 14, etc. Accordingly, the instruction information 21 is text that indicates the target in natural language. The data may consist of stock data. The trained model 351 may also be further provided with at least one of the object description 50 generated by the object estimator 31 and the initial state description 51 generated by the initial state estimator 33. In addition, input / output samples 243 may be further provided to the trained model 351 as environmental information 22 (domain information 24). In one example, the input samples may consist of instruction information 21, object description 50 and initial state description 51 samples, and the output samples may consist of ground truth samples of the corresponding target state description 52. The configuration may be as follows. The input / output samples 243 may be used for few-shot prompting of the trained model 351. The trained model 351 may be provided in common with the trained model 333 of the initial state estimator 33, or it may be provided separately.

[0098] Note that the configuration in Figure 5C is merely one example of the target estimator 35. The configuration of the target estimator 35 is not limited to the example in Figure 5C and may be modified as appropriate depending on the embodiment. In another example, the trained model 351 may be provided with environmental information 22 other than the input / output samples 243. In yet another example, the trained model 351 may be configured to accept input of data from other modalities, similar to the trained model 333 described above. Accordingly, the trained model 351 may be provided with data in a format other than text data. The instruction information 21 may further include data in a format other than text data, or it may consist only of data in a format other than text data.

[0099] [Robot equipment] The robot device R is not particularly limited and may be appropriately selected depending on the embodiment. The robot device R may be, for example, an industrial robot on a production line, an autonomous robot configured to operate autonomously, or a mobile body configured to move. Industrial robots may be, for example, vertical articulated robots, horizontal articulated robots (SCARA robots), parallel link robots, orthogonal robots, etc. Autonomous robots may be, for example, humanoid robots, guide robots, agricultural robots, nursing care robots, security robots, transport robots, etc. The content of autonomous processing may be appropriately selected depending on the embodiment. Mobile bodies may include, for example, cleaning robots, the above-mentioned autonomous robots configured to move (including mobile robots), vehicles configured to operate automatically, and aircraft capable of automatic flight (drones, etc.). The robot device R may exist in real space or virtual space. The action plan using the generated problem description 5 may be performed for the control of the robot device R in real space, or for the simulation of the robot device R in virtual space.

[0100] The generation of problem description 5 according to this embodiment may be applied to any task that the robot device R can accomplish. Tasks may be, for example, work, movement, etc. Work may be, for example, assembly, cooking, cleaning, chemical experiments, etc. If the robot device R is equipped with a communication device, the sensing data (observation data 20) may include data obtained by communication. For example, in the case of generating an action plan (movement plan) for a moving object, the sensing data may include data obtained by vehicle-to-infrastructure communication, vehicle-to-vehicle communication, etc.

[0101] Furthermore, the robot device R may be operated in a setting where it coexists with humans. In this setting, instruction information 21 may be provided by a human (operator). The generated problem description 5 may be output in a way that is perceptible to humans (for example, it may be output to a display). The information processing device 1 may accept human intervention (manual modification) on the generated problem description 5. For example, human intervention may modify the movement amount, angle, posture, sway, etc., of the robot device R. The problem description 5 may be generated through a dialogue-style interaction between the information processing device 1 and the operator by using a trained model capable of performing in-context learning in the inference module 3 (at least one of the object estimator 31, initial state estimator 33, and target estimator 35). Human input may be provided directly to the information processing device 1 or indirectly to the information processing device 1 via the robot device R. Human input may be provided by any method, such as operating an input device (including text input), recording, or imaging. The planner may be configured to generate an action plan from problem description 5, taking into account the presence of human beings.

[0102] [Output of problem description] The output format of problem description 5 is not particularly limited and can be appropriately selected depending on the embodiment. That's fine. In a simple example, the generated problem description 5 may be output as is. Outputting the problem description 5 may include outputting the problem description 5 to at least one of the memory resources of the information processing device 1, an output device connected to the information processing device 1, and an external computer. The output method may be appropriately selected depending on the embodiment. For example, the problem description 5 may be output as text, an image, audio, or a combination thereof.

[0103] In another example, outputting problem description 5 may include at least part of providing problem description 5 to the planner and having the planner generate an action plan. If this includes the planner generating an action plan, outputting problem description 5 may further include controlling the operation of the robot device R (i.e., driving the robot device R) according to the generated action plan.

[0104] The generation of an action plan by the planner may be performed by at least one of the information processing device 1 and one or more external computers. If the planner includes a symbolic planner 61 and a motion planner 65, the information processing device 1 may perform the calculation processing for one of the symbolic planner 61 and the motion planner 65, while the external computer performs the calculation processing for the other. The calculation processing for both the symbolic planner 61 and the motion planner 65 may be performed by either the information processing device 1 or one or more external computers. If the calculation processing for both the symbolic planner 61 and the motion planner 65 is performed by one or more external computers, the external computer performing the calculation processing for the symbolic planner 61 and the external computer performing the calculation processing for the motion planner 65 may be the same or different.

[0105] Furthermore, the control of the robot device R based on the generated action plan may be performed by at least one of the information processing device 1 and an external computer. When at least part of the generation of the action plan by the planner and the control of the robot device R are performed by an external computer, the external computer that performs the generation of the action plan by the planner and the external computer that controls the robot device R may be the same or different. One or more external computers may perform at least part of the generation of the action plan, and the information processing device 1 may perform the control of the robot device R according to the generated action plan. The information processing device 1 and the external computer may exchange data as appropriate by any method such as data communication.

[0106] §2 Example Configuration [Hardware configuration] Figure 6 schematically illustrates an example of the hardware configuration of the information processing device 1 according to this embodiment. In the example in Figure 6, the information processing device 1 according to this embodiment is a computer in which a control unit 11, a storage unit 12, an external interface 13, an input device 14, an output device 15, and a drive 16 are electrically connected.

[0107] The control unit 11 is a hardware processor, a CPU (Central Processing Unit), It includes RAM (Random Access Memory), ROM (Read Only Memory), etc., and is configured to perform information processing based on programs and various data. The control unit 11 (CPU) is an example of processor resources. The storage unit 12 may be composed of, for example, a hard disk drive, a solid-state drive, etc. The storage unit 12, RAM, and ROM are examples of memory resources. In this embodiment, the storage unit 12 stores various information such as the program 81 and module data 300.

[0108] Program 81 is a program that causes the information processing device 1 to execute information processing related to the generation of problem description 5 (Figure 8, described later). Program 81 includes a series of instructions for said information processing. Module data 300 indicates information related to inference module 3. Problem description 5 The configuration of the module data 300 is not particularly limited and can be determined as appropriate depending on the embodiment, as long as the inference module 3 can be regenerated when generating the module data. For example, if a machine learning model is included, the module data 300 may include information indicating the values ​​of computation parameters adjusted by machine learning. In some cases, the module data 300 may further include information indicating the configuration of the machine learning model (e.g., the structure of a neural network). If a rule-based model is included, the module data 300 may include information indicating the rules. If the inference module 3 has the configuration shown in Figure 4, information regarding the object estimator 31, the initial state estimator 33, and the target estimator 35 may be held in separate data (files), or at least one combination may be held in the same data. The module data 300 may be incorporated into the program 81.

[0109] The external interface 13 is configured to connect to an external device by wire or wireless connection. The external interface 13 may be, for example, a USB (Universal Serial Bus) port, a communication port, a dedicated port, etc. If the external interface 13 includes a communication port, the communication standard of the communication port may be arbitrarily selected. In this embodiment, the information processing device 1 may be connected to an external device (for example, a sensor S, a robot device R, an external computer, etc.) via the external interface 13.

[0110] The input device 14 is, for example, a device for inputting data such as a mouse or keyboard. The output device 15 is, for example, a device for outputting data such as a display or speaker. The operator can operate the information processing device 1 by using the input device 14 and the output device 15. The input device 14 and the output device 15 may be connected via an external interface 13. The input device 14 and the output device 15 may be integrated into a single unit, such as a touch panel display.

[0111] The drive 16 is a device for reading various types of information, such as programs, stored in the storage medium 91. At least one of the program 81 and module data 300 may be stored in the storage medium 91 in place of or together with the storage unit 12. The storage medium 91 is configured to store various types of information (stored programs, etc.) by electrical, magnetic, optical, mechanical, or chemical means so that a machine such as a computer can read the information. The information processing device 1 may acquire at least one of the program 81 and module data 300 from the storage medium 91. The storage medium 91 may be a disk-type storage medium such as a CD or DVD, or a non-disk-type storage medium such as semiconductor memory (e.g., flash memory). The type of drive 16 may be appropriately selected according to the type of storage medium 91. The drive 16 may be connected via an external interface 13.

[0112] Regarding the specific hardware configuration of the information processing device 1, components can be omitted, replaced, and added as appropriate depending on the embodiment. For example, the control unit 11 may include multiple hardware processors. Hardware processors include microprocessors, FPGAs (field-programmable gate arrays), DSPs (digital signal processors), and GPs. It may be composed of a U (Graphics Processing Unit), an ASIC (application-specific integrated circuit), etc. The storage unit 12 contains RAM and ROM included in the control unit 11. It may be composed of more. At least one of the external interface 13, input device 14, output device 15, and drive 16 may be omitted. The information processing device 1 may be composed of multiple computers. In this case, the hardware configuration of each computer may or may not be the same. In addition, the information processing device 1 may be an information processing device designed specifically for the service to be provided, or it may be a general-purpose server device, a general-purpose PC (Personal Computer), a tablet PC, a terminal device, etc. An external computer is used. In this case, the hardware configuration of the external computer may be the same as that of the information processing device 1. The external computer may be an information processing device designed specifically for the service being provided, or it may be a general-purpose server device, a general-purpose PC, a tablet PC, a terminal device, etc.

[0113] [Software Configuration] Figure 7 schematically illustrates an example of the software configuration of the information processing device 1 according to this embodiment. The control unit 11 of the information processing device 1 loads the program 81 stored in the storage unit 12 into RAM, and the CPU executes the instructions contained in the program 81. As a result, the information processing device 1 operates as a computer equipped with an acquisition unit 111, a generation unit 112, and an output unit 113 as software modules. In other words, in this embodiment, each software module of the information processing device 1 is realized by the control unit 11 (CPU).

[0114] The acquisition unit 111 is configured to acquire observational data 20 of the environment in which the robot device R operates, and instruction information 21 regarding the task objectives to be given to the robot device R. In one example, the acquisition unit 111 may be configured to further acquire environmental information 22 regarding the environment. Since the information processing device 1 holds module data 300, the generation unit 112 includes an inference module 3. The generation unit 112 is configured to use the inference module 3 to generate a task problem description 5 from the acquired observational data 20 and instruction information 21. In one example, generating a task problem description 5 from observational data 20 and instruction information 21 may be configured by generating a task problem description 5 from observational data 20, instruction information 21, and environmental information 22. The output unit 113 is configured to output the generated problem description 5.

[0115] In this embodiment, an example is described in which each software module of the information processing device 1 is implemented by a general-purpose CPU. However, some or all of the above software modules may be implemented by one or more dedicated processors or chipsets. Each of the above modules may also be implemented as a hardware module. Regarding the software configuration of the information processing device 1, modules may be omitted, replaced, and added as appropriate, depending on the embodiment.

[0116] §3 Example of Operation Figure 8 is a flowchart showing an example of the processing procedure of the information processing device 1 according to this embodiment. The following processing procedure is an example of an information processing method executed by a computer. However, the following processing procedure of the information processing device 1 is merely an example, and each step may be modified as much as possible. Furthermore, depending on the embodiment, steps in the following processing procedure can be omitted, replaced, and added as appropriate.

[0117] (Step S101) In step S101, the control unit 11 operates as an acquisition unit 111 and acquires observation data 20 and instruction information 21. In one example, the observation data 20 may consist of sensing data from one or more sensors S, and the instruction information 21 may consist of language information that indicates a target in natural language. In another example, the control unit 11 may further acquire environmental information 22. The environmental information 22 may include at least one of a domain description 23 and domain information 24. The domain information 24 may include attribute information 241. If the inference module 3 includes a trained model 39 for in-context learning, the domain information 24 may include input / output samples 243. After acquiring the observation data 20 and instruction information 21, the control unit 11 proceeds to the next step S102.

[0118] (Step S102) In step S102, the control unit 11 operates as a generation unit 112 and uses the inference module 3 to generate a task problem description 5 from the acquired observation data 20 and instruction information 21. In one example, if environmental information 22 is acquired further, the control unit 11 uses the inference module 3 Then, a problem description 5 may be generated from the observation data 20, instruction information 21, and environmental information 22.

[0119] The generated problem description 5 includes descriptions (51, 52) of the initial and target states of one or more objects present in the environment. In one example, the generated problem description 5 may follow a predetermined format. The generated problem description 5 may also further include descriptions 50 of the objects present in the environment.

[0120] In one example, generating a problem description 5 using the inference module 3 may include at least one of the following: generating an object description 50 using the object estimator 31, generating an initial state description 51 using the initial state estimator 33, and generating a target state description 52 using the target estimator 35. If the inference module 3 includes a pre-trained model 39 for in-context learning and the domain information 24 includes input / output samples 243, the control unit 11 may use the input / output samples 243 as few-shot prompting to adapt the pre-trained model 39 to the domain for generating the problem description 5. Once the problem description 5 is generated, the control unit 11 proceeds to the next step S103.

[0121] (Step S103) In step S103, the control unit 11 operates as an output unit 113 and outputs the generated problem description 5. In one example, the control unit 11 may output the problem description 5 to at least one of the memory resources of the information processing device 1, the output device 15, and an external computer. In another example, as part of the process in step S103, the control unit 11 may perform at least part of the process of providing the problem description 5 to the planner and having the planner generate an action plan. When the information processing device 1 performs calculation processing of the planner, the control unit 11 may appropriately acquire information about the planner. For example, information about the planner may be stored in at least one of the memory resources of the information processing device 1 (e.g., RAM, storage unit 12) and an external computer, and the control unit 11 may acquire information about the planner from any of these. When the process of generating an action plan by the planner is performed by an external computer, the control unit 11 may provide the generated problem description 5 to the external computer as part of the process in step S103. In another example, when an action plan is generated by the information processing device 1 or an external computer, the control unit 11 may, as part of the process in step S103, control the operation of the robot device R according to the generated action plan. Controlling the operation of the robot device R may include directly controlling the robot device R and indirectly controlling the robot device R by giving instructions to the robot device R's controller. Note that at least one of the generation of the action plan by the planner and the control of the robot device R may be performed separately from the process in step S103. Control of the robot device R may be omitted. If the control of the robot device R is omitted, the generation of the action plan by the planner may also be omitted. Once the output of problem description 5 is completed, the control unit 11 terminates the processing procedure of the information processing device 1 related to this example of operation.

[0122] For example, in the process of generating an action plan by the planner, if the planner outputs an error message 615, the control unit 11 may execute a process to correct the problem description 5. For example, as shown in Figure 3, if the inference module 3 includes a trained model 39 for in-context learning, the control unit 11 may obtain the outputted error message 615. The control unit 11 may then provide the obtained error message 615 and the problem description 5 to the inference module 3 and execute the calculation process of the inference module 3 again. At this time of re-prompting, the control unit 11 may further provide at least a portion of the input data 200 to the inference module 3. As a result, the control unit 11 may generate a new corrected problem description 5. The control unit 11 may recursively repeat the process of correcting the problem description 5 by this re-prompting.

[0123] [Features] In this embodiment, the problem description 5 generated in step S102 includes descriptions (51, 52) of the initial state and target state of one or more objects present in the environment, so that the planner can generate an action plan to reach the target state from the initial state of the task. The initial state description 51 describes the initial state of one or more objects present in the environment. On the other hand, the target state description 52 describes the target state of one or more objects that is reached upon completion of the task. That is, each description (51, 52) shows the states before and after the execution of the task and is interpretable to humans. Therefore, this problem description 5 has high explainability. Accordingly, according to this embodiment, a highly explainable output can be obtained for obtaining an action sequence (control command) to be given to the robot device R.

[0124] §4 Variant Although embodiments of the present invention have been described in detail above, the above description is merely illustrative in all respects of the present invention. Various improvements or modifications may be made to the above embodiments as appropriate. For example, the following modifications are possible. In the following, the same reference numerals are used for components similar to those in the above embodiments, and explanations of points similar to those in the above embodiments have been omitted as appropriate. The following modifications can be combined as appropriate.

[0125] <4.1> In the above embodiment, the sensor S may be installed so that the observation direction can be changed by being attached to or including a drive device. For example, if the sensor S includes a camera, the camera may be installed so that the shooting direction can be changed by being attached to an electric pan / tilt head. An electric pan / tilt head is an example of a drive device. The information processing device 1 may be configured to drive the drive device directly or indirectly. As a result, the information processing device 1 may change the observation direction by the sensor S as appropriate.

[0126] In addition, the environmental information 22 may include a list of specified objects to be observed. In step S101, the control unit 11 of the information processing device 1 may acquire the environmental information 22 including the specified list. The attribute information 241 may also serve as the list of specified objects. In step S102, the control unit 11 may determine whether or not all objects included in the specified list have been detected from the observation data 20. In the example of Figure 4, the control unit 11 may determine whether or not all objects included in the specified list have been detected from the observation data 20 in the calculation processing of the object estimator 31 or the initial state estimator 33. In one example, if an object detector is provided, such as the trained model 311 in Figure 5A, the control unit 11 may determine whether or not all objects included in the specified list have been detected based on the object detection result by the detector for the observation data 20. In another example, the control unit 11 may determine whether or not all objects included in the specified list have been detected based on whether or not all objects included in the specified list are included in at least one of the object description 50 and the initial state description 51. If at least some of the objects included in the specified list are not detected from the observation data 20, the control unit 11 may drive the sensor S drive device and appropriately change the observation direction of the sensor S. The direction and amount of change may be determined by any method, such as randomly or according to a predetermined rule. The control unit 11 may then acquire the observation data 20 again and repeat the process in step S102. The control unit 11 may repeat these processes until all objects included in the specified list are detected. On the other hand, if all objects included in the specified list are detected from the observation data 20, the control unit 11 may proceed with the process of generating the problem description 5. According to this modification, it is possible to correct the observation direction of the sensor S to an appropriate direction and improve the accuracy of generating the problem description 5.

[0127] The above process for verifying detected objects may be modified as appropriate depending on the embodiment. For example, the designated list may further include objects that can be observed or not (recommended objects). If only the recommended object is not detected, the control unit 11 may output an alert and proceed with the process of generating problem description 5. Alternatively, the control unit 11 may stop the process of generating problem description 5 and ask the operator whether to continue the process of generating problem description 5.

[0128] <4.2> The domain description 23 may include parts that depend on the robot device R (e.g., a description defining the skills of the robot device R) and parts that relate to the environment (e.g., a description defining the type of target object). Therefore, similar to the method for generating the problem description 5 in the above embodiment, the information processing device 1 may use a calculation module to generate at least a portion of the environment-related portion of the domain description 23 from at least a portion of the observation data 20, instruction information 21, and environment information 22. The calculation module may consist of at least one of a trained model and a rule-based model. According to this modification, it is possible to reduce the effort required to prepare the domain description 23.

[0129] <4.3> In the example shown in Figure 4 above, the inference module 3 is configured to generate the descriptions of the object, initial state, and target state (50, 51, 52) in the problem description 5 individually. However, the configuration of the inference module 3 is not limited to this example. In another example, the inference module 3 may be configured to generate at least one combination of the descriptions of the object, initial state, and target state (50, 51, 52) in the problem description 5 as a single unit.

[0130] Figure 9 schematically shows an example of the configuration of an inference module 3A in another form. The inference module 3A comprises an object detector 371, a caption model 373, and a large-scale language model 375. In the example in Figure 9, the observation data 20 may consist of image data, and the instruction information 21 may consist of text data that indicates the target in natural language. The object detector 371 is an open vocabulary object as proposed in the above references 11 and 12. A detection model or an open vocabulary object segmentation model may be used. For caption model 373, the visual question answering model proposed in reference 13, etc., may be used. For large-scale language model 375, the visual question answering model proposed in reference 14, etc., may be used. The proposed model may be used.

[0131] First, the control unit 11 of the information processing device 1 may detect objects from the observation data 20 using the object detector 371. Depending on the object detection result by the object detector 371, the control unit 11 may extract partial data (partial image data) for each object from the observation data 20 and provide the extracted partial data to the caption model 373. The control unit 11 may use the caption model 373 to generate captions for each object from the extracted partial data for each object. Then, the control unit 11 may use the large-scale language model 375 to generate the descriptions (50, 51, 52) of the problem description 5 from the text data of the object detection result by the object detector 371, the text data of the captions from the caption model 373, and the instruction information 21. According to this modified example, the calculation process can be simplified.

[0132] <4.4> In the above embodiment, the re-prompting method shown in Figure 3 may be modified as appropriate depending on the embodiment. For example, the re-prompting method in Figure 3 is described in Reference 15 (Jason Wei et al., “Chain-of-Thought Prompting Elicits Reasoning in Large Language Models”, [online], [Retrieved October 24, 2023], Internet).<URL:https: / / arxiv.org / pdf / 2201.11903.pdf> ), Reference 16 (Takeshi Kojima et al., “Large Language Models are Zero-Shot Reasoners”, [online], [Retrieved October 24, 2020], In Tarnet<URL:https: / / arxiv.org / abs / 2205.11916> ), Reference 17 (Denny Zh ou et al., “Least-to-Most Prompting Enables Complex Reasoning in Large Language Models, [online], [Retrieved October 24, 2023], Internet<URL:https: / / arxiv.org / abs / 2205.10625> The Chain-of-Thought prompt proposed in ) etc. is used. This is acceptable. The Chain-of-Thought prompt is a method that obtains the final inference result (in this embodiment, the result of generating the problem description 5) by performing stepwise inference through intermediate inference steps. For example, the control unit 11 of the information processing device 1, along with the problem description 5 and the error message 615, may ask, "What part of the PDDL problem do you think is causing this error?" The control unit 11 may provide the inference module 3 with template data for asking about errors, causing the inference module 3 to generate an explanation for the errors. Subsequently, the control unit 11 may provide the inference module 3 with the problem description 5 and the error message 615, causing the inference module 3 to generate a new problem description 5.

[0133] §5 Experimental Examples To verify the effectiveness of the above embodiments, the following experiments were conducted. However, the present invention is not limited to the following embodiments.

[0134] (Inference module) As the inference module for the first embodiment, the configuration of the inference module shown in Figure 4 was adopted. The configuration of the object detector was adopted as shown in Figure 5A. The trained model for the object detector was the model proposed in Reference 11. The configuration of the initial state estimator was adopted as shown in Figure 5B. The detector for the initial state estimator was the model proposed in Reference 13. The trained model for the initial state estimator was the model proposed in Reference 14. The configuration of the target estimator was adopted as shown in Figure 5C. The trained model for the target estimator was the model proposed in Reference 14.

[0135] Furthermore, image data was used for observation data. Text data indicating the target in natural language was used for instruction information. The trained model of the object estimator was provided with observation data (image data) along with attribute information indicating the object's attributes in linguistic representation. The trained model of the initial state estimator was provided with text data indicating the object detection result, text data of the detector's caption, a description of the object generated by the object estimator, and input / output samples used for few-shot prompting. The trained model of the target estimator was provided with instruction information (text data), a description of the object generated by the object estimator, a description of the initial state generated by the initial state estimator, and input / output samples used for few-shot prompting. There were three input / output samples.

[0136] As the inference module for the second embodiment, the configuration of the inference module shown in Figure 9 was adopted. For the object detector of the inference module, the model proposed in Reference 11 was adopted. For the caption model, the model proposed in Reference 13 was adopted. For the large-scale language model, the model proposed in Reference 14 was adopted. The large-scale language model was provided with instruction information (text data), text data of the detection result by the object detector, text data of the caption by the caption model, and input / output samples used for few-shot prompting.

[0137] The PDDL format was adopted for the problem description. A symbolic planner was prepared to verify whether an action plan could be generated using the generated problem description. The environment for running the symbolic planner was Fast Downward and VAL (Reference 18: “KCL-Planning / VAL”, [online], [Retrieved October 24, 2023]), Internet.<URL:https: / / github.com / KCL-Planning / VAL> ) was used. The results of the first and second examples In the theory module, a correction mechanism using re-prompting (Figure 3) was provided. Re-prompting consisted of input data, a generated problem description, and an error message. The Chain-of-Thought prompt method described above was adopted. The maximum number of times the problem description can be corrected by re-prompting was set to two.

[0138] (Dataset) Figure 10 shows the types (object types), predicates, and actions of the domain description for each prepared domain. As shown in Figure 10, Cooking, Blockwar We prepared datasets in three domains: Blocksworld and the Tower of Hanoi. The cooking task was assumed to involve slicing vegetables and placing them in a bowl. The robotic device R was assumed to consist of two robotic arms positioned on the left and right. The target state was specified by the state and location of the vegetables.

[0139] Blocks World is referenced in Reference 19 (Naresh Gupta et al., “On the Complexity of Blocks-World Planning”, [online], [Retrieved October 24, 2023], Internet). The ancients used in URL:https: / / www.semanticscholar.org / paper / On-the-Complexity-of-Blocks-World-Planning-Gupta-Nau / db01349fd0d29c9443e37b0b0aa4ddb948ace5ce>) etc. This is a typical domain. Seven unique colored blocks were used for each problem. The robotic arm was configured so that it did not always start by grasping something. The target state specified the relationships between the blocks.

[0140] Hanoi is referenced in reference 20 (Ron Alford et al., “Translating HTNs to PDDL: A Small "Amount of Domain Knowledge Can Go a Long Way", [online], [Retrieved October 24, 2023], Internet<URL:https: / / www.cs.umd.edu / ~nau / papers / alford2009translating.pdf> This is a classic domain used in applications such as [examples of applications]. It consists of 3 pegs and 10 pieces in 6 colors. The disks were configured. Disks of the same color were numbered in descending order of width. The three pegs were numbered from left to right. The initial state and target state specified the disk positions.

[0141] Figures 11A, 11B, and 11C show examples of observational data and instructional information given in the Cooking, Block World, and Hanoi domains. One domain description was prepared for each domain. Additionally, a sample problem description was provided for each problem. In Hanoi, the same instructional information was used for all problems.

[0142] (Evaluation metrics) To evaluate whether the problem description was generated appropriately, four indicators (R syntax , R plan , R part , R all ) was prepared. R syntaxwas defined as the ratio of the generated problem description conforming to the PDDL syntax. When VAL did not return a warning and an end code, the generated problem description was regarded as syntactically correct. R plan was defined as the ratio when no error message was output by the symbolic planner when generating an action plan from the generated problem description . When attempting to generate an action plan by the symbolic planner and VAL did not return an error message , the action plan was regarded as being generated effectively. R part and R all were defined as the ratio of the generated problem description containing all the descriptions of the correct answer samples for the purpose of evaluating reproducibility . R part In , the descriptions of the object, the initial state, and the target state were calculated individually. R all For, it was calculated for the entire generated problem description.

[0143] (Experimental Example) In the first experimental example, using the inference module of the first embodiment, problem descriptions were generated for each of the three domains. The correction of the problem description by re-prompting was set to be executed up to two times. 100 problems were prepared for each domain. By changing the combination of input / output samples, 10 problem descriptions were generated for each problem. For the generated problem descriptions, the above four evaluation indexes (R syntax , R plan , R part , R all ) were calculated.

[0144] In the second experimental example, using the inference module of the second embodiment, problem descriptions were generated for each of the three domains. For the generated problem descriptions, three evaluation indexes (R syntax , R plan , R all ) were calculated. Other conditions were set the same as in the first experimental example.

[0145] In the third experimental example, using the inference module of the first embodiment, (A) correction re-prompt (CR) and Chain-of-Thought (CoT) prompts were given (corrections were limited to two), and (B) correction re-prompt Rompto (CR) and Chain-of-Thought (CoT) prompts are available (only one correction allowed), (C) Correction prompts (CR) are present, but Chain-of-Thought (CoT) prompts are absent (correction is performed only once). (D) until, as well as the correction re-prompt (CR) and Chain-of-Thought (CoT) prompts. Problem descriptions were generated for each of the three domains under four conditions (none). Three evaluation metrics (R) were applied to the generated problem descriptions. syntax , R plan , R all The following calculation was performed. All other conditions were set to be the same as in the first experimental example. All other conditions were set to be the same as in the first experimental example.

[0146] (Experimental results) Figures 12, 13, and 14 show the calculation results for each evaluation index in the first, second, and third experimental examples. The numbers in parentheses in Figure 13 indicate the difference from the results of the first experimental example.

[0147] As shown in Figure 12, in the Hanoi domain of the first experimental example, R plan It was a little low. However, in each domain, R syntax and R plan The score was high. That is, it was able to generate syntactically correct problem descriptions. Also, in the Block World and Hanoi domains, R all Although the score was low, in the cooking domain, R part and R all The scores are both The results were high. Therefore, it was found that high reproducibility could be achieved in generating problem descriptions. From these results, it was found that effective problem descriptions can be generated according to this embodiment.

[0148] Furthermore, as shown in Figure 13, even when using the inference module of the second embodiment, it was possible to generate syntactically correct problem descriptions equivalent to those of the inference module of the first embodiment. Also, in the cooking and block world domains, R all Although the score decreased, in the Hanoi domain, R all The score improved. These results show that even when adopting an integrated configuration like the one shown in Figure 9 for the inference module, it is possible to generate a valid problem description.

[0149] Furthermore, as shown in Figure 14, by overlapping the Correction Re-prompt (CR) and Chain-of-Thought (CoT) prompts, R syntax , R plan and R all All scores increased. From the results, the Correction Re-prompt (CR) and Chain-of-Thought (CoT) prompts are appropriate. It was found to be effective in obtaining a title description.

[0150] This specification includes the following disclosures: [Note 1] A step of acquiring observational data of the environment in which the robot device operates, and instruction information regarding the objective of the task to be given to the robot device. A step of generating a problem description for the task from the acquired observation data and instruction information using an inference module, wherein the problem description includes a description of the initial state and target state of objects present in the environment, and Steps include outputting the generated problem description, A control unit configured to perform the following actions: Information processing device. [Note 2] The generated problem description follows a predetermined format. The information processing device described in Appendix 1. [Note 3] The generated problem description further includes a description of the object present in the environment. The information processing device described in Appendix 1 or 2. [Note 4] The aforementioned observation data is composed of sensing data from the sensor. The instruction information consists of language information that indicates the target in natural language. An information processing device described in any one of the appendices 1 through 3. [Note 5] The inference module is configured to include a pre-trained model for in-context learning, In the process of providing the generated problem description to the planner to generate an action plan for the robot device, if the planner outputs an error message, the control unit shall... The steps include obtaining the outputted error message, and Using the aforementioned inference module, a step of generating a new problem description from the problem description and the error message, It is configured to perform further actions. An information processing device described in any one of the appendices 1 through 4. [Note 6] In the acquisition step described above, the control unit is configured to further acquire environmental information relating to the environment. The generation of the problem description from the acquired observation data and instruction information is comprised of generating the problem description from the acquired observation data and instruction information and environmental information. An information processing device described in any one of the appendices 1 through 5. [Note 7] The generated problem description further includes a description of the object present in the environment, The inference module includes an object estimator, Using the inference module to generate the problem description includes using the object estimator to generate a description of the object present in the environment from the obtained observation data. An information processing device described in any one of the appendices 1 through 6. [Note 8] The object estimator comprises a pre-trained model that learns in context. The information processing device described in Appendix 7. [Note 9] In the acquisition step, the control unit is configured to further acquire attribute information of the object present in the environment. Generating a description of the object from the acquired observation data is comprised of generating a description of the object from the acquired observation data and attribute information. The information processing device described in Appendix 7 or Appendix 8. [Note 10] The inference module includes an initial state estimator, Using the inference module to generate the problem description includes using the initial state estimator to generate a description of the initial state of the object present in the environment. An information processing device described in any one of the appendices 1 through 9. [Note 11] The initial state estimator comprises a pre-trained model of in-context learning. The information processing device described in Appendix 10. [Note 12] The inference module includes a target estimator, Using the inference module to generate the problem description includes using the target estimator to generate a description of the target state of the object present in the environment. The information processing devices described in Appendix 1 to Appendix 11. [Note 13] The aforementioned target estimator comprises a pre-trained model of in-context learning. The information processing device described in Appendix 12. [Note 14] Computers A step of acquiring observational data of the environment in which the robot device operates, and instruction information regarding the objective of the task to be given to the robot device. A step of generating a problem description for the task from the acquired observation data and instruction information using an inference module, wherein the problem description includes a description of the initial state and target state of objects present in the environment, and Steps include outputting the generated problem description, Execute Information processing methods. [Note 15] On the computer, A step of acquiring observational data of the environment in which the robot device operates, and instruction information regarding the objective of the task to be given to the robot device. A step of generating a problem description for the task from the acquired observation data and instruction information using an inference module, wherein the problem description includes a description of the initial state and target state of objects present in the environment, and Steps include outputting the generated problem description, To execute program. [Explanation of Symbols]

[0151] 1… Information processing equipment, 11...Control unit, 12...Storage unit, 13...External interface, 14...Input device, 15...Output device, 16...Drive, 81...Program, 91...Storage medium, 111...acquisition section, 112...generation section, 113...output section, 3...Inference module, 300...Module data, 31...Object estimator, 33...Initial state estimator, 35...target estimator, 20... Observation data, 21... Instructional information, 22...Environmental information, 23...Domain description, 24...Domain information, 241...Attribute information, 5…Problem description, 50...Description of the object, 51...Description of the initial state, 52…Description of the target state

Claims

1. The inference module receives input data as prompts, including observational data of the environment in which the robot device operates and instructional information regarding the objective of the task to be given to the robot device, to generate a problem description of the task. A step of obtaining an error message indicating that the problem description is incorrect or that an appropriate action plan cannot be generated based on the problem description. The steps include: inputting a re-prompt including the error message and the problem description, or a re-prompt obtained by modifying the input data in response to the error message using a modification model, to the inference module to generate a new problem description; Steps include outputting the newly generated problem description, A control unit configured to perform the following actions: Information processing device.

2. The new problem description includes at least a description of the target state of the object in the environment, The information processing apparatus according to claim 1.

3. In the aforementioned new problem description, descriptions of objects that cannot be involved in the aforementioned operation are omitted. The information processing apparatus according to claim 1.

4. The inference module is a trained model that includes at least one of the following: a large-scale language model, a large-scale visual language model, or a large-scale audio model. The information processing apparatus according to claim 1.

5. The control unit is configured to perform a further step, before the output step, of determining whether the generated new problem description conforms to a predetermined format. The information processing apparatus according to claim 1.

6. The aforementioned error message indicates that at least the problem description is not suitable for action plan generation. This includes the inability to generate an action plan while avoiding prohibited items, the inability to generate an action plan with the given robotic device's skills, or the problem description not conforming to a prescribed format. The information processing apparatus according to claim 1.

7. The steps of obtaining the error message, generating the new problem description, and outputting the new problem description are repeated each time an error message is obtained. The information processing apparatus according to claim 1.

8. The maximum number of repetitions mentioned above is set. The information processing apparatus according to claim 7, wherein if the number of times exceeds the maximum number, the control unit stops executing the step of generating the new problem description.

9. The robotic device is one of the following: an industrial robot in a production line, an autonomous robot configured to operate autonomously, or a mobile body configured to move. The information processing apparatus according to claim 1.

10. The aforementioned industrial robot is one of the following: a vertical articulated robot, a horizontal articulated robot, a SCARA robot, a parallel link robot, or a Cartesian robot. The autonomous robot is one of the following: a humanoid robot, a guidance robot, an agricultural robot, a caregiving robot, a security robot, or a transport robot. The aforementioned mobile entity is one of the following: a cleaning robot, a mobile autonomous robot, a vehicle configured for autonomous driving, or an aircraft capable of autonomous flight. The information processing apparatus according to claim 9.

11. The aforementioned environment consists of at least one of a real environment and a virtual environment. The information processing apparatus according to claim 1.

12. The re-prompt, which includes the error message and the problem description, further includes a portion of the input data relating to the error message, The information processing apparatus according to claim 1.

13. The input data further includes environmental information relating to the environment, The information processing apparatus according to claim 12.

14. Computers The inference module receives input data as prompts, including observational data of the environment in which the robot device operates and instructional information regarding the objective of the task to be given to the robot device, to generate a problem description of the task. A step of obtaining an error message indicating that the problem description is incorrect or that an appropriate action plan cannot be generated based on the problem description. The steps include: inputting a re-prompt including the error message and the problem description, or a re-prompt obtained by modifying the input data in response to the error message using a modification model, to the inference module to generate a new problem description; Steps include outputting the newly generated problem description, Execute Information processing methods.

15. On the computer, The inference module receives input data as prompts, including observational data of the environment in which the robot device operates and instructional information regarding the objective of the task to be given to the robot device, to generate a problem description of the task. A step of obtaining an error message indicating that the problem description is incorrect or that an appropriate action plan cannot be generated based on the problem description. The steps include: inputting a re-prompt including the error message and the problem description, or a re-prompt obtained by modifying the input data in response to the error message using a modification model, to the inference module to generate a new problem description; Steps include outputting the newly generated problem description, To execute program.

Images