Information processing device, information processing method and program

Abstract

To provide a technology that obtains an output with a high explanatory property regarding a control on a robot apparatus.SOLUTION: An information processing device according to an aspect of the present invention obtains observation data on an environment where a robot apparatus is actuated, and instruction information relating to an aiming achievement for a task to be applied to the robot apparatus, generates, using an estimation module, a problem description in the task from the obtained observation data and instruction information, and outputs the generated problem description. The problem description is configured to involve descriptions for the initial state of an object present in the environment and for the aiming state thereof.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to an information processing device, an information processing method, and a program. [Background technology]

[0002] Interacting with robots using natural language is a promising method for non-experts to enable robots to solve complex and diverse tasks. In recent years, research using large-scale language models (LLMs) to output a sequence of actions for a robot in a workspace from linguistic instructions has been attracting attention.

[0003] For example, Non-Patent Document 1 proposes a system that generates an action sequence for a robot device from linguistic instructions. The system proposed in Non-Patent Document 1 accepts input of instructions in natural language (linguistic instructions). The system uses an LLM (Say module) to infer from the linguistic instructions the next action of the robot device that is likely to be executed. The system also uses a value function (Can module) to infer from observed data which action is likely to be executed. The system then integrates the two inference results and, depending on the integration result, determines the action sequence to be given to the robot device (an action sequence that is likely to be executed and is executable). [Prior art documents] [Non-patent literature]

[0004] [Non-Patent Document 1] Michael Ahn et al., “Do As I Can, Not As I Say: Grounding Language in Robotic Affordances”, [online], [Retrieved October 24, 2023], Internet<URL:https: / / arxiv.org / abs / 2204.01691> Summary of the Invention [Problem to be solved by the invention]

[0005] The inventors of the present invention found that the above-mentioned conventional system has the following problems. That is, in the conventional system, the generated action sequence is specialized for the trial environment and is not necessarily interpretable by humans. Since the action sequence is output as is, there is a problem that the interpretability of the obtained output regarding the control of the robot device is low.

[0006] In one aspect, the present invention has been made in view of the above circumstances, and an object of the present invention is to provide a technique for obtaining a highly explainable output regarding the control of a robot device. [Means for solving the problem]

[0007] In order to solve the above-mentioned problems, the present invention employs the following configurations. Note that the following configurations of the invention can be combined as appropriate.

[0008] That is, an information processing device according to one aspect of the present invention includes a control unit configured to execute the steps of acquiring observation data of an environment in which a robot device operates and instruction information regarding a goal of a task to be given to the robot device, generating a problem description of the task from the acquired observation data and the instruction information using an inference module, and outputting the generated problem description. The problem description includes descriptions of initial states and goal states of objects present in the environment.

[0009] The problem description, together with the domain description, is used by the planner to generate an action plan (i.e., obtain a sequence of actions). This problem description describes the initial and goal states of the objects in the environment to accomplish the task, and is highly descriptive (human-interpretable). Therefore, with this configuration, it is possible to obtain a highly explainable output for obtaining an action sequence (command) to be given to the robot device.

[0010] In the information processing device according to the above aspect, the generated problem description may be in accordance with a predetermined format. With this configuration, since the generated problem description conforms to the predetermined format, it is possible for a planner to easily generate an action plan from the problem description.

[0011] In the information processing device according to the above aspect, the generated problem description may further include a description of the objects present in the environment, where the description of the objects makes it easier to identify the objects present in the environment, thereby making it easier for a planner to generate an action plan from the problem description.

[0012] In the information processing device according to the above aspect, the observation data may be composed of sensing data from a sensor, and the instruction information may be composed of linguistic information instructing the target in a natural language. With this configuration, when an environment is observed by a sensor and a task target is given by linguistic instruction, a highly explainable output can be obtained regarding the control of the robot device.

[0013] In the information processing device according to the above aspect, the inference module may be configured to include a trained model for in-context learning. The control unit may be configured to further execute, when the planner outputs an error message in a process of providing the generated problem description to a planner and generating a behavior plan for the robot device, a step of acquiring the output error message and a step of generating a new problem description from the problem description and the error message using the inference module. According to this configuration, when an appropriate problem description is not obtained, the problem description can be automatically corrected. The trained model is a trained machine learning model.

[0014] The control unit of the information processing device according to the above aspect may be configured to further acquire environmental information related to the environment in the acquiring step. Generating the problem description from the acquired observation data and instruction information may be configured by generating the problem description from the acquired observation data, instruction information, and environmental information. According to this configuration, by further using environmental information as an explanatory variable in addition to the observation data and instruction information, the environment in which a task is performed can be more easily specified, and therefore it is expected that the accuracy of generating the problem description can be improved.

[0015] In the information processing device according to the above aspect, the generated problem description may further include a description of the object present in the environment. The inference module may include an object estimator. Generating the problem description using the inference module may include generating a description of the object present in the environment from the acquired observation data using the object estimator. With this configuration, it is possible to appropriately generate a description portion of the object in the problem description.

[0016] In the information processing device according to the above aspect, the object estimator may include a trained model for in-context learning. With this configuration, since the object estimator includes the trained model for in-context learning, it is possible to generically estimate objects existing in the environment, which is expected to enable generation of problem descriptions for various tasks.

[0017] The control unit of the information processing device according to the above aspect may be configured to further acquire attribute information of the object present in the environment in the acquiring step. Generating the description of the object from the observation data may be configured by generating the description of the object from the acquired observation data and the attribute information. With this configuration, since the attribute information makes it possible to more accurately identify objects existing in the environment, it is expected to improve the accuracy of generating the description portion of the object in the problem description.

[0018] In the information processing device according to the above aspect, the inference module may include an initial state estimator. Generating the problem description using the inference module may include generating a description of the initial state of the object present in the environment using the initial state estimator. With this configuration, it is possible to appropriately generate a description portion of the initial state in the problem description.

[0019] In the information processing device according to the above aspect, the initial state estimator may include a trained model for in-context learning. With this configuration, since the initial state estimator includes the trained model for in-context learning, the initial state of an object existing in the environment can be generally inferred, and thus it is expected that the problem description generation for various tasks can be handled.

[0020] In the information processing device according to the above aspect, the inference module may include a goal estimator. Generating the problem description using the inference module may include generating a description of the goal state of the object in the environment using the goal estimator. With this configuration, it is possible to appropriately generate the description portion of the goal state in the problem description.

[0021] In the information processing device according to the above aspect, the goal estimator may include a trained model for in-context learning. With this configuration, the goal estimator includes a trained model for in-context learning, so that the goal state of the object can be generically inferred, and thus it is expected that the problem description generation for various tasks can be handled.

[0022] The form of the present invention may not be limited to the above information processing device. As another aspect of the information processing device according to each of the above aspects, one aspect of the present invention may be an information processing method that realizes all or part of each of the above configurations, a program, or a storage medium that stores such a program and is readable by a machine such as a computer. A storage medium that is readable by a machine such as a computer is a medium that stores information such as a program by electrical, magnetic, optical, mechanical, or chemical action.

[0023] For example, an information processing method according to one aspect of the present invention may be an information processing method in which a computer executes the steps of acquiring observation data of an environment in which a robotic device operates and instruction information regarding a task goal to be given to the robotic device, generating a problem description of the task from the acquired observation data and the instruction information using an inference module, and outputting the generated problem description.

[0024] Also, for example, a program related to one aspect of the present invention may be a program for causing a computer to execute the steps of acquiring observation data of an environment in which a robotic device operates and instruction information regarding a task goal to be given to the robotic device, generating a problem description of the task from the acquired observation data and the instruction information using an inference module, and outputting the generated problem description. Effect of the Invention

[0025] According to the present invention, it is possible to obtain a highly explainable output regarding the control of a robot device. [Brief description of the drawings]

[0026] [Figure 1] FIG. 1 shows a schematic diagram of an example of a situation in which the present invention is applied. [Diagram 2] FIG. 2 is a schematic diagram showing an example of input and output of an inference module according to an embodiment. [Diagram 3]FIG. 3 illustrates an example of a problem description revision process according to an embodiment. [Figure 4] FIG. 4 illustrates an example of a configuration of an inference module according to an embodiment. [Figure 5A] FIG. 5A illustrates an example of an object estimator according to an embodiment. [Figure 5B] FIG. 5B illustrates an example of an initial state estimator according to an embodiment. [Figure 5C] FIG. 5C illustrates a schematic diagram of an example of a target estimator according to an embodiment. [Figure 6] FIG. 6 illustrates an example of a hardware configuration of an information processing device according to an embodiment. [Figure 7] FIG. 7 illustrates an example of a software configuration of the information processing device according to the embodiment. [Figure 8] FIG. 8 is a flowchart illustrating an example of a processing procedure of the information processing device according to the embodiment. [Figure 9] FIG. 9 is a schematic diagram showing an example of the configuration of an inference module according to another embodiment. [Figure 10] FIG. 10 shows each description included in the domain description for each of the prepared domains. [Figure 11A] FIG. 11A shows an example of observation data and instruction information given in the domain (Cooking). [Figure 11B] FIG. 11B shows an example of observation data and instruction information given in the domain (Blocksworld). [Figure 11C] FIG. 11C shows an example of observation data and instruction information given in the domain (Hanoi). [Figure 12] FIG. 12 shows the results of the first experimental example. [Figure 13] FIG. 13 shows the results of the second experimental example. [Figure 14] FIG. 14 shows the results of the third experimental example. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0027] An embodiment according to one aspect of the present invention (hereinafter, also referred to as "the present embodiment") will be described below with reference to the drawings. However, the present embodiment described below is merely an example of the present invention in every respect. Various improvements or modifications may be made without departing from the scope of the present invention. In carrying out the present invention, a specific configuration according to the embodiment may be appropriately adopted. Note that while the data appearing in this embodiment is described in natural language, more specifically, it is specified in pseudo-language, commands, parameters, machine language, etc. that can be recognized by a computer.

[0028] §1 Examples of application FIG. 1 shows a schematic diagram of an example of a situation in which the present invention is applied. An information processing device 1 according to this embodiment is one or more computers configured to generate a problem description 5 used by a planner to generate an action plan. Specifically, the information processing device 1 acquires observation data 20 of an environment in which the robot device R operates, and instruction information 21 regarding a goal of a task to be given to the robot device R. The information processing device 1 generates a problem description 5 of the task from the acquired observation data 20 and instruction information 21 using an inference module 3. The problem description 5 includes descriptions (51, 52) of an initial state and a goal state of one or more objects present in the environment. The information processing device 1 outputs the generated problem description 5.

[0029] The problem description 5 includes descriptions (51, 52) of the initial and goal states of one or more objects in the environment, so that the planner generates an action plan to reach the goal state from the initial state of the task. The initial state is the state before the task is performed. The description 51 may be referred to as a state. The goal state is a state after the task is performed and the purpose is achieved. The description 51 of the initial state describes the initial state of one or more objects present in the environment. On the other hand, the description 52 of the goal state describes the goal state of one or more objects to be reached by the completion of the task. That is, each description (51, 52) indicates each state before and after the performance of the task, and is interpretable by humans. In one example, each description (51, 52) indicates the state of an object, not the behavior of the robot device R, and is therefore easy for humans to interpret even without a special understanding of the robot device R (for example, without deciphering the program code). Therefore, the problem description 5 has high explanatory power. Therefore, according to this embodiment, a highly explanatory output can be obtained to obtain an action sequence (control command) to be given to the robot device R.

[0030] [Input / output data] In this embodiment, observed data 20 and instruction information 21 are provided to the inference module 3 as input data, and as a result of the execution of the arithmetic processing of the inference module 3, a problem description 5 is obtained as output data.

[0031] The observation data 20 may be any data that represents an initial state of an object before performing a task. The type of the observation data 20 is not particularly limited and may be appropriately selected according to the embodiment. The observation data 20 may be composed of data of one or more modalities. The environment may be observed by any method. The environment may include any events related to the situation in which the robot device R performs a task. The environment may be composed of at least a real environment and a virtual environment. For example, the environment may be composed of both a real environment and a virtual environment, such as AR (Augmented Reality) and MR (Mixed Reality). The environment may include VR (Virtual Reality) and the like. The number of objects present in the environment may be arbitrary. The observation data 20 may be acquired at any timing before starting to perform the task. When multiple tasks are given, the initial state may be a state before performing the multiple tasks, or a state before performing one task included in the multiple tasks (including an intermediate state between performing the multiple tasks). In other words, "initial" does not necessarily mean "beginning". A task may be any work (job) for transitioning from an initial state to a goal state. The state before the task is performed may be the initial state, and the state after the task is properly performed may be the goal state. The interval between tasks may be appropriately determined depending on the embodiment. Since the observation data 20 indicates the initial state of an object, the inference module 3 can infer the description 51 of the initial state in the problem description 5 from the observation data 20.

[0032] The instruction information 21 may be any information that can identify the goal of the task. The type of the instruction information 21 is not particularly limited and may be appropriately selected depending on the embodiment. The instruction information 21 may be composed of data of one or more modalities. The goal may be given appropriately. The instruction information 21 may be acquired at any timing before starting the performance of the task. The instruction information 21 may be acquired before or after the observation data 20. The instruction information 21 may also be acquired at least partially in parallel with the observation data 20. Since the goal of the task is expressed in the instruction information 21, the inference module 3 can infer the description 52 of the goal state in the problem description 5 from the instruction information 21.

[0033] The problem description 5 includes descriptions (51, 52) of the initial state and the goal state, thereby enabling the planner to generate an action plan for the robot device R, including the feasibility (whether or not the robot device R is executable), given the skills of the robot device R. As long as it is of this nature, the configuration of the problem description 5 is not particularly limited and may be appropriately selected depending on the embodiment. The problem description 5 is independent of the robot device and may therefore be used to generate an action plan for any robot device. The planner may be appropriately configured to generate an action plan from the problem description 5.

[0034] As shown in FIG. 1, in one example, the skills of the robot device R may be given by the domain description 23. Also, in one example, the planner may include a symbolic planner 61 and a motion planner 65. The symbolic planner 61 may be configured to generate an action sequence 63, which is a series of abstract actions, from the problem description 5 and the domain description 23. The abstract action is any set of actions that includes one or more actions of the robot device R and may be defined as a set of actions that can be expressed by a desire (e.g., words, etc.). The abstract action may be defined as a set of meaningful (i.e., understandable by humans) actions such as grasping an object, carrying it, positioning it, etc. For the symbolic planner 61, known symbolic planners such as Fast Downward of Reference 1 (“The Fast Downward Planning System”, [online], [searched on October 24, Reiwa 5], Internet <URL:https: / / planning.wiki / ref / planners / fd>), and LAMA planner of Reference 2 (Silvia Richter et al., “The LAMA planner: guiding cost-based anytime planning with landmarks”, [online], [searched on October 24, Reiwa 5], Internet <URL:https: / / dl.acm.org / doi / 10.5555 / 1946417.<1946420>) may be used.

[0035] The motion planner 65 may be configured to generate a sequence of control commands 67 for causing the robot device R to execute each behavior included in the behavior sequence 63 when the behavior sequence 63 is provided by the symbolic planner 61. The motion planner 65 may also be configured to determine whether or not the robot device R can execute each behavior included in the behavior sequence 63. The motion planner 65 may include, for example, a motion planner described in Reference 3 (James J. Kuffner, Jr., Steven M. LaValle, “RRT-Connect: An Efficient Approach to Single-Query Path Planning”, [online], [searched October 24, 2023], Internet)<URL:https: / / www.cs.cmu.edu / afs / cs / academic / class / 15494-s12 / readings / kuffner_icra2000.pdf> )'s RRT-connect, Reference 4 (Nathan Ratliff et al., "CHOMP: Gradient Optimization Techniques for Efficient Motion Planning", [online], [Retrieved October 24, 2023], Internet<URL:https: / / www.ri.cmu.edu / pub_files / 2009 / 5 / icra09-chomp.pdf> )'s CHOMP (Covariant Hamiltonian Optimization for Motion Planning), Reference 5 (Mrinal Kalakrishnan et al., "STOMP: Stochastic Trajectory Optimization for Motion Planning", [online], [Retrieved October 24, 2023], Internet <URL:http: / / ros.fei.edu.br / roswiki / attachments / Papers(2f)ICRA2011_Kalakrishnan / kalakrishnan_icra2011.pdf> A known motion planner such as STOMP (Synopsis of the Motion Picture Instrument) may be used. The generated control command 67 may be provided to the robot device R as appropriate.The robot device R may start performing the task at any timing according to the given control command 67. The configuration of the planner is not limited to the example in Fig. 1 and may be appropriately changed depending on the embodiment. In another example, the planner may be configured to generate an action plan by inferring the control command 67 directly from the problem description 5 and the domain description 23.

[0036] As long as the problem description 5 includes descriptions of the initial state and the goal state (51, 52) and can be used to generate an action plan, the format of the problem description 5 is not particularly limited and may be appropriately selected depending on the embodiment. In one example, the problem description 5 to be generated may be configured to conform to a predetermined format. The predetermined format may be, for example, PDDL (Planning Domain Definition Language), PDDLStream (Reference 6: “pddlstream”, [online], [searched October 24, 2023], Internet<URL:https: / / github.com / caelan / pddlstream> ) or other language for a planner (planning description language). In one example of the present embodiment, the problem description 5 to be generated conforms to a predetermined format, making it easier for a planner (e.g., a symbolic planner 61) to generate an action plan from the problem description 5.

[0037] In another example, the generated problem description 5 does not have to follow a predefined format. In this case, the generated problem statement 5 may be converted to conform to a predetermined format by an intermediate process such as a conversion program. Alternatively, the generated problem statement 5 may be used to generate an action plan without conforming to a predetermined format by a planner capable of accepting input in any format, such as a planner configured with a trained model for in-context learning.

[0038] The input / output data of the inference module 3 may be appropriately changed depending on the embodiment. The inference module 3 may further receive input of any data other than the observation data 20 and the instruction information 21. The inference module 3 may further output any data other than the problem description 5. The problem description 5 may further include any information other than the descriptions (51, 52) of the initial state and the goal state.

[0039] (Example of input / output data) Fig. 2 shows an example of input and output of the inference module 3 according to this embodiment. In the example of Fig. 2, the input data includes observation data 20, instruction information 21, and environment information 22. The problem description 5 (output data) includes descriptions (50, 51, 52) of the object, the initial state, and the goal state.

[0040] In one example, the environment may be observed by one or more sensors S, and the observation data 20 may accordingly be composed of sensing data of one or more sensors S. The type of the sensor S is not particularly limited and may be appropriately selected according to the embodiment. The sensor S may include, for example, a camera, a depth sensor, an infrared sensor, an optical sensor, a radar, a LiDAR (Light Detection And Ranging), a microphone, a position sensor, other measurement sensors, etc. The position sensor may be, for example, a GPS (Global Positioning System) sensor, a GNSS (Global Navigation Satellite System) sensor, etc. The sensing data may include, for example, image data, depth data, infrared data, measurement data of an optical sensor (such as a detection result of a marker), radar data, LiDAR data, sound data, position data, other measurement data, etc. The sensor S may also include one or more measurement sensors for measuring the state of the robot device R. The measurement sensor may include, for example, an encoder, a motion capture, a tactile sensor, a force sensor, an inertial measurement unit, etc. Accordingly, the sensing data may include measurement data of the robot device R (for example, joint angles, hand position, tactile data at the hand, force data at the hand, posture measurement data, etc.). The sensor S may be disposed outside the robot device R or may be disposed in the robot device R.

[0041] In addition, in one example, the instruction information 21 may be configured with language information that instructs the target in natural language. The instruction in natural language may be acquired by any method. The instruction in natural language may be acquired by, for example, text input, voice input, image input, other input, or other methods. The data format of the instruction information 21 (language information) may be appropriately selected according to the embodiment. The instruction information 21 may be configured with, for example, text data, voice data, image data, other types of data, or the like. The acquired instruction data in natural language may be used as the instruction information 21 as is, or may be used as the instruction information 21 after converting the data format. As an example of the latter, the instruction data may be converted by a conversion model, such as from voice to text, or from text to voice, and the converted instruction data may be used as the instruction information 21. The conversion may include any analysis process such as voice analysis. The conversion model may be included in the inference module 3, or may be prepared separately from the inference module 3. The conversion model may be configured with at least one of a trained model (trained machine learning model) and a rule-based model. The instructions in natural language may be given manually by an operator or automatically by computer processing. The instruction information 21 may be given appropriately depending on the task. In one example, the instruction information 21 is The instruction information 21 may be given each time, such as every time the robot apparatus R reaches a target, etc. In another example, when a target is specified in advance, such as the Tower of Hanoi, the instruction information 21 may be given. In one example of this embodiment, when the environment is observed by the sensor S and a task target is given by linguistic instructions, a highly explanatory output can be obtained regarding the control of the robot apparatus R.

[0042] The configurations of the observation data 20 and the instruction information 21 may be appropriately changed depending on the embodiment. The observation data 20 may include any data other than the sensing data, such as data provided manually or data generated by computer processing, together with the sensing data. In another example, the observation data 20 may not include the sensing data and may be composed of any data other than the sensing data. In another example, the instruction information 21 may be provided in a format other than a natural language. For example, the instruction information 21 may be provided in a text format using symbols other than a natural language. For example, the instruction information 21 may be composed of an image showing a target state of an object present in the environment. The image may be composed of at least one of a real image and a virtual image.

[0043] In addition, in one example, the information processing device 1 may be configured to further acquire environmental information 22 related to the environment. Generating the problem description 5 from the acquired observation data 20 and instruction information 21 may be configured by generating the problem description 5 from the acquired observation data 20, instruction information 21, and environmental information 22. According to one example of the present embodiment, the environmental information 22 restricts the environment in which the task is performed, thereby narrowing down the conditions for generating the problem description 5. That is, by further using the environmental information 22 as an explanatory variable in addition to the observation data 20 and instruction information 21, the environment in which the task is performed can be more easily specified. Therefore, it is expected that the accuracy of generating the problem description 5 can be improved.

[0044] It should be noted that the environment information 22 may include any information related to the relationships of the robot device R that may be involved in generating the problem description 5. In one example, the environment information 22 may include at least one of a domain description 23 and domain information 24.

[0045] The domain description 23 defines events common to all problems, including the skills of the robot device R. As described above, the domain description 23 may be used by the planner to generate an action plan together with the problem description 5. When the problem description 5 is given according to a predetermined format such as PDDL or PDDLStream, the domain description 23 may also be given according to the predetermined format. For example, when PDDL is adopted, the problem description 5 may be problem.pddl, and the domain description 23 may be domain.pddl. In one example, the domain description 23 may include a description (e.g., actions) that defines the skills of the robot device R, a description (e.g., predicates) that defines the state of a target object existing in the environment, a description (e.g., types) that defines the type of the target object, and a description (e.g., requirements) for checking compatibility with the planner. The state of the target object may include a state that the target object can take when the robot device R executes the defined skill (action).

[0046] The domain information 24 may include any information that limits the domain for generating the problem description 5. The domain information 24 may also be referred to as domain knowledge. The domain information 24 may be composed of any information related to the environment other than the domain description 23. The domain information 24 may supplement the conditions for generating the problem description 5 together with the domain description 23. In one example, the domain information 24 may include attribute information 241 of objects present in the environment. The attribute information 241 may include, for example, the name and characteristics of the object. The characteristics may include, for example, characteristics related to the appearance such as color, shape, size, etc. The characteristics related to the appearance are, for example, a cutting board that is round, a counter that is black, etc. This makes it possible to narrow down the characteristics of objects appearing in the observation data 20, and as a result, it is expected that the accuracy of inference for the observation data 20 will be improved. In one example, as described later, the inference module 3 is configured to include a trained model for in-context learning. When a trained model is generated, the domain information 24 may include one or more input / output samples 243 for the trained model. For example, an output sample may be a ground-truth sample of a portion of the problem statement 5 that is generated by the trained model. An input sample may be a sample of a portion of the observation data 20, instruction information 21, and environment information 22 that are provided to obtain a ground-truth sample of the problem statement 5 and that is input to the trained model.

[0047] The format of the input data (observation data 20, instruction information 21, and environmental information 22) provided to the inference module 3 is not particularly limited and may be appropriately determined according to the embodiment. In one example, the observation data 20, instruction information 21, and environmental information 22 may be input to the inference module 3 as is. In another example, preprocessing may be applied to at least one of the observation data 20, instruction information 21, and environmental information 22, and the preprocessed data may be input to the inference module 3. The preprocessing may include any arithmetic processing such as a process of analyzing information, a process of adding information, or a process of reducing information. The arithmetic model that performs the preprocessing may be included in the inference module 3, or may be prepared separately from the inference module 3. The arithmetic model may be configured by at least one of a trained model (trained machine learning model) and a rule-based model. For example, in a case where the observation data 20 is configured by image data, the inference module 3 may be configured to accept input of the image data. Alternatively, the image data may be analyzed by any method such as image processing, an analysis model, etc. This analysis processing is an example of preprocessing. The analysis result may include, for example, a bounding box detection result, an identification result of an object appearing in the image data, etc. The inference module 3 may be configured to accept an input of the analysis result for the image data.

[0048] In one example, the generated problem description 5 may further include a description 50 of an object present in the environment. The object description 50 may correspond to a list of objects present in the environment. The range of objects included in the description 50 may be appropriately determined depending on the embodiment. For example, the object description 50 may be configured with a list of all objects that can be observed in the target environment. Also, for example, the object description 50 may be configured with a list of some objects that can be observed. Some objects may be objects of interest, such as objects that may be involved in a task. In this case, objects of no interest (e.g., objects that may not be involved in a task) may be omitted in the object description 50. According to one example of the present embodiment, the object description 50 makes it easier to identify objects present in the environment, thereby making it easier for a planner (e.g., a symbolic planner 61) to generate an action plan from the problem description 5. Note that when the descriptions (51, 52) of the initial state and the goal state are obtained so as to conform to a predetermined format, the object description 50 may also be obtained so as to conform to a predetermined format. For example, when PDDL is adopted, the description of objects in problem.pddl is an example of a description of an object 50, the description of init (initial state) is an example of a description of an initial state 51, and the description of goal is an example of a description of a goal state 52.

[0049] The input / output form of the inference module 3 is not limited to the example in FIG. 2 and may be changed as appropriate depending on the embodiment. At least a part of the environmental information 22 may be omitted. In the problem description 5, the object description 50 may be omitted. In one example, when the robot device R operates in an environment where no objects other than the robot device R exist (for example, a drone flies in the air where there are no objects), the object description 50 may be omitted in the problem description 5.

[0050] [Problem description correction] In one example, the information processing device 1 may be configured to execute a process related to modifying the generated problem description 5. The process of modifying the problem description 5 may be automatically executed by a computer, or may be manually executed by a human being such as an operator.

[0051] As an example, the information processing device 1 may determine whether the generated problem description 5 is compatible with a planner (e.g., a symbolic planner 61). The determination may be performed using a trained model and The problem description 5 may be corrected by any method, such as using at least one of a rule-based model and a rule-based model. For example, when the problem description 5 is generated to follow a predetermined format, the information processing device 1 may determine whether the generated problem description 5 conforms to the predetermined format by evaluating whether the generated problem description 5 conforms to the predetermined format. If it is determined that the problem description 5 conforms to the planner, the generated problem description 5 may be appropriately given to the planner. On the other hand, if it is determined that the generated problem description 5 does not conform to the planner, the information processing device 1 may appropriately correct the generated problem description 5. For example, the information processing device 1 may correct the generated problem description 5 using a correction model. The correction model may be composed of at least one of a trained model and a rule-based model. Also, for example, the information processing device 1 may output the problem description 5 to an output device together with factors that do not conform to the planner (for example, indicating parts that do not conform to a predetermined format). The information processing device 1 may accept a correction to the problem description 5 by an operator via an input device. Then, the information processing device 1 may correct the problem description 5 according to the received content. As another example, the information processing device 1 may execute the above process of correcting the problem description 5 after omitting the process of determining whether or not it is suitable for the planner.

[0052] The timing of executing the process of correcting the problem description 5 is not particularly limited and may be appropriately determined depending on the embodiment. In one example, before the problem description 5 is used in the generation of an action plan by a planner (e.g., the symbolic planner 61), the information processing device 1 may execute a process of correcting the problem description 5. In another example, after the generated problem description 5 is given to the planner, the information processing device 1 may execute a process of correcting the problem description 5 in response to an error occurring in the attempt of the action plan by the planner, such as outputting an error message or being unable to generate a proper action plan. Note that the process of correcting the problem description 5 may be executed by an external computer other than the information processing device 1. Also, the process of correcting the problem description 5 may be omitted.

[0053] FIG. 3 is a schematic diagram showing an example of a problem statement 5 revision process according to the present embodiment. In one example, the inference module 3 may be configured to include a trained model 39 for in-context learning. In-context learning is, for example, acquiring the ability to make a specific inference through the context of an input (prompt), such as an input / output sample. In one example, the trained model 39 can acquire the ability of in-context learning by including a self-attention mechanism and an autoregressive model. The trained model 39 may be, for example, a large-scale language model (LLM), a large-scale visual language model (LVLM), or the like. The large-scale visual language model may include a visual question answering model, an open vocabulary object detection model, an open vocabulary object segmentation model, or the like. In addition, the trained model 39 may be, for example, a large-scale language model combined with one or more other modalities (such as sound), such as Audio Question Answering. Furthermore, the trained model 39 may be a large-scale model of one or more modalities other than language, such as a Large Audio Model. The data format of the input to the trained model 39 may be appropriately selected depending on the embodiment.

[0054] In the example of Fig. 3, by including a trained model 39 having the capability of in-context learning, the reasoning module 3 can adapt to an error when an error occurs and modify the problem statement 5 according to the error. Thus, in the example of Fig. 3, the information processing device 1 may generate the problem statement 5 by providing input data 200 to the reasoning module 3 and executing the calculation process of the reasoning module 3. The input data 200 includes observation data 20 and instruction information 21 of a target scene for which an action plan is to be generated. In one example, the input data 200 may further include environmental information 22.

[0055] The generated problem description 5 may be given to a planner. The planner An action plan may be generated using the problem description 5. When the planner has the configuration of FIG. 1, the generated problem description 5 may be provided to a symbolic planner 61, which may attempt to generate an action sequence 63 from the problem description 5. The process of generating an action plan by this planner may be executed by at least one of the information processing device 1 and an external computer other than the information processing device 1. As described above, the domain description 23 may be used together with the problem description 5 to generate the action plan.

[0056] If the problem description 5 is appropriate, the planner can generate an appropriate action plan from the problem description 5. In the example of FIG. 1, an appropriate motion sequence 63 is obtained, and the motion planner 65 can generate a sequence of control commands 67 accordingly. The generated action plan may be used as appropriate for motion control of the robot device R. On the other hand, if the problem description 5 is inappropriate, an error occurs in the generation of the action plan by the planner. The problem description 5 being inappropriate may include, for example, the problem description 5 not being suitable for the planner, the inability to generate an action plan while avoiding prohibited items, the inability to generate an action plan with the given skill of the robot device R, and the like. The problem description 5 not being suitable for the planner may include, for example, the problem description 5 not conforming to a predetermined format, and the like. In this case, an error message 615 is output from the planner.

[0057] In the process of providing the generated problem description 5 to the planner and generating a behavior plan for the robot device R, if the planner outputs an error message 615, the information processing device 1 may acquire the output error message 615. The configuration of the error message 615 is not particularly limited and may be appropriately determined according to the embodiment of the configuration of the planner, etc. Then, the information processing device 1 may provide the acquired error message 615 and problem description 5 to the inference module 3 and execute the calculation process of the inference module 3 again. That is, the information processing device 1 may use the problem description 5 and the error message 615 as a re-prompt and execute the process of generating the problem description 5 again. Thereby, the information processing device 1 may generate a new problem description 5 from the problem description 5 and the error message 615 using the inference module 3 (that is, may correct the problem description 5). According to one example of this embodiment, when an appropriate problem description 5 is not obtained, the problem description 5 can be automatically corrected.

[0058] The data provided to the inference module 3 at the time of re-prompting need not be limited to the problem description 5 and the error message 615. The configuration of the re-prompt may be appropriately changed depending on the embodiment. In another example, the information processing device 1 may provide at least a part of the input data 200 to the inference module 3 together with the problem description 5 and the error message 615. This is expected to improve the accuracy of correcting the problem description 5.

[0059] Furthermore, the information processing device 1 may recursively and repeatedly execute the process of correcting the problem description 5 by re-prompt. The number of times to be repeated is not particularly limited and may be appropriately determined depending on the embodiment. In one example, the information processing device 1 may repeatedly execute the process of correcting the problem description 5 by re-prompt until the error message 615 is no longer output. In another example, the number of times to execute the process of correcting the problem description 5 by re-prompt may be predefined. If the error message 615 is still output after the information processing device 1 has repeatedly executed the correction process by re-prompt a predetermined number of times, the information processing device 1 may stop the correction process by re-prompt and output the execution result up to that point (e.g., the generated problem description 5, etc.).

[0060] In addition, the method of re-prompting is not limited to the above example, and may be appropriately changed depending on the embodiment. In another example, the information processing device 1 may correct the problem statement 5 by using a prompt corrected in response to the error message 615 as a re-prompt, together with or instead of the problem statement 5 and the error message 615. For example, when the error message 615 is output, the information processing device 1 may correct the output error by using the corrected model. In response to the error message 615, the information processing device 1 may appropriately modify the prompt (input data 200) provided to the inference module 3. The modified model may be configured as appropriate using at least one of a trained model and a rule-based model. The information processing device 1 may then provide the modified prompt again to the inference module 3 and execute the computational processing of the inference module 3 to generate a new problem description 5.

[0061] [Inference module] The inference module 3 is configured to execute an inference process to generate a problem description 5 from input data including the observation data 20 and the instruction information 21. As long as such an inference process can be executed, the configuration of the inference module 3 is not particularly limited and may be appropriately determined depending on the embodiment. In one example, the inference module 3 may be configured by at least one of a rule-based model and a trained model (a trained machine learning model).

[0062] The rule-based model is configured to derive an inference result (in this embodiment, the generation result of problem statement 5) from a given input according to rules. The rules may be set as appropriate. The machine learning model is configured to have one or more calculation parameters that can be adjusted by machine learning. The one or more calculation parameters are used for the calculation of the desired inference (in this embodiment, the generation of problem statement 5). The machine learning model may be configured, for example, by a neural network, a regression model, a decision tree model, a support vector machine, or other functional formulas (calculation models). The machine learning method may be selected as appropriate depending on the machine learning model to be adopted (for example, backpropagation, etc.).

[0063] Machine learning is to adjust (optimize) values ​​of calculation parameters using training samples. Typically, a trained model may be generated by performing supervised learning using a plurality of learning data sets each of which is composed of a combination of an input sample (training sample) and an output sample (teacher signal, label). For example, the input sample may be a sample of input data (observation data 20, instruction information 21, etc.), and the output sample may be a sample of output data (problem description 5). In supervised learning, the values ​​of the calculation parameters of the machine learning model may be adjusted so that the output obtained from the machine learning model by providing an input sample matches the corresponding output sample. However, the method of generating a trained model is not limited to such an example, and may be appropriately changed depending on the embodiment. The learning data set is not limited to the above example, and may be appropriately selected depending on the embodiment. For example, when acquiring the ability of in-context learning, data other than the above may be used for the learning data set. In addition, the learning method is not limited to supervised learning, and other methods such as unsupervised learning (including self-supervised learning) and reinforcement learning may be used. The machine learning model may be trained at least online and offline. The machine learning model may be appropriately tuned by transfer learning, re-learning, additional learning, etc. Additional learning is described in, for example, Reference 7 ("LoRA", [online], [searched October 24, 2023], Internet<URL:https: / / github.com / microsoft / LoRA> )'s LoRA (Low-Rank Adaptation of Large Language Models), Reference 8 (Neil Houlsby et al., “Parameter-Efficient Transfer Learning for NLP”, [online], [Retrieved October 24, 2023], Internet<URL:http: / / proceedings.mlr.press / v97 / houlsby19a / houlsby19a.pdf> ) Adapter, Reference 9 (Brian Lester et al., “The Power of Scale for Parameter-Efficient Prompt Tuning”, [online], [Retrieved October 24, 2023], Internet<URL:https: / / arxiv.org / abs / 2104.08691> )'s Prompt Tuning, which adds a module to an existing trained model and adjusts the parameters of the additional module while leaving the parameters of the existing trained model unchanged. In addition, additional learning can be performed as described in Reference 10 (Kecheng Zheng et al., "Regularized Mask Tuning: Uncovering Hidden Knowledge in Pre-trained Vision-Language Models", [online], [Retrieved October 24, 2023], Internet <URL:https: / / openaccess.thecvf.com / content / ICCV2023 / papers / Zheng_Regularized_Mask_Tuning_Uncovering_Hidden_Knowledge_in_Pre-Trained_Vision-Language_Models_ICCV_2023_paper.pdf>) and other machine learning to adjust a mask that disables the use of a part of the trained model. When a trained model capable of in-context learning, such as the large-scale language model, is used for at least a part of the inference module 3, the trained model may be specialized for a specific problem, domain, or other situation by performing these additional learnings.

[0064] In one example, the inference module 3 may include a neural network. The structure of the neural network may not be particularly limited and may be appropriately determined according to the embodiment. The structure of the neural network may be specified by, for example, the number of layers from the input layer to the output layer, the type of each layer, the number of nodes (neurons) included in each layer, the connection relationship between the nodes in each layer, etc. In one example, the neural network may include any mechanism such as a recurrent structure, a self-attention mechanism, an autoregressive model, etc. In addition, the neural network may include any layer such as a fully connected layer, a convolutional layer, a pooling layer, a deconvolutional layer, an unpooling layer, a normalization layer, a dropout layer, an LSTM (Long short-term memory), etc. The neural network may include any type of model such as a diffusion model, a transformer model, a generative model, etc. The weight of the connection between each node included in the neural network and the threshold value of each node are examples of calculation parameters.

[0065] As described above, in one example, the inference module 3 may include a trained model 39 for in-context learning. In this case, the input data provided to the inference module 3 may include one or more input / output samples 243 for the trained model 39 as the domain information 24 of the environment information 22. The input / output samples 243 may be prepared appropriately according to the domain for which the problem description 5 is generated. In one example of the present embodiment, by providing the input / output samples 243 to the trained model 39, in-context learning can be performed and the trained model 39 can be adapted to the target domain. In other words, by preparing the input / output samples 243 for each domain, the inference module 3 can generically generate the problem description 5 without replacing the trained model 39. Therefore, according to one example of the present embodiment, it is expected that the problem description 5 can be generated for tasks in various domains.

[0066] Also, in one example, when the inference module 3 includes a trained model 39 for in-context learning, the domain description 23 may be prepared for each type of the robot device R. In addition, an output sample (a sample of the problem description 5) corresponding to the domain description 23 may be prepared for each type as the domain information 24. In response to this, the information processing device 1 may accept the designation of the type. The designation method is not particularly limited and may be appropriately selected according to the embodiment. Typically, the type may be designated by a manual method such as an operation by an operator via an input device, text input, or voice input. The designation of the type may be included in the instruction information 21. The information processing device 1 may provide the domain description 23 and the output sample (domain information 24) corresponding to the designated type to the inference module 3 (trained model 39) as the environment information 22. This makes it possible to generate the problem description 5 for generating an action plan for the robot device R having a skill of the designated type. That is, according to one example of the present embodiment, it is expected that the versatility of the inference module 3 is improved and the accuracy of generating the problem description 5 that matches the type is improved.

[0067] Furthermore, the form of input and output of the inference module 3 may be appropriately determined depending on the embodiment. As described above, in one example, the input data (observation data 20, instruction information 21, and environmental information 22) may be provided to the inference module 3 as is. In another example, at least a portion of the input data may be subjected to any pre-processing before being provided to the inference module 3. The pre-processing may include computational processing such as extraction of feature quantities, other analyses, and inferences. The inference module 3 may be provided with the input data after pre-processing. Also, in one example, the inference module The output of the reasoning module 3 may be configured to directly indicate the inference result (problem statement 5). In another example, the output of the reasoning module 3 may be configured to indirectly indicate the inference result. In this case, the inference result may be obtained by performing any information processing (interpretation processing) on ​​the output of the reasoning module 3. The generation of the problem statement 5 may be performed in real time on data available at the present time (which may include past data) or after the fact on data available in the past.

[0068] Furthermore, the inference module 3 may be configured as an integral unit, or may be configured as a set of multiple subelements. In one example, when the inference module 3 is configured as a set of multiple subelements, each subelement may be appropriately configured to generate a part (corresponding part) of the problem description 5. For example, one or more subelements may be prepared for each description (50, 51, 52) of the object, initial state, and goal state of the problem description 5. Each subelement may be configured by at least one of a machine learning model and a rule-based model.

[0069] (Example of inference module configuration) FIG. 4 is a schematic diagram showing an example of the configuration of the reasoning module 3 according to the present embodiment. In the example of FIG. 4, a scene is assumed in which the problem description 5 further includes a description 50 of an object present in the environment. In one example, the reasoning module 3 may include an object estimator 31, an initial state estimator 33, and a goal estimator 35. The object estimator 31 is an example of one or more subelements corresponding to the description 50 of the object. The initial state estimator 33 is an example of one or more subelements corresponding to the description 51 of the initial state. The goal estimator 35 is an example of one or more subelements corresponding to the description 52 of the goal state.

[0070] The object estimator 31 may be configured to generate a description 50 of one or more objects present in the environment from the acquired observation data 20. Generating the object description 50 may include detecting the objects. Generating the problem statement 5 using the reasoning module 3 may include generating, using the object estimator 31, a description 50 of the objects present in the environment from the acquired observation data 20. The observation data 20 contains information about the objects present in the environment. Therefore, according to one example of the present embodiment, the object description 50 in the problem statement 5 can be appropriately generated.

[0071] The configuration of the object estimator 31 is not particularly limited and may be appropriately selected depending on the embodiment as long as it is possible to generate an object description 50 from the observation data 20. The above description regarding the configuration of the inference module 3 may also be applied to the object estimator 31. The object estimator 31 may be configured by at least one of a rule-based model and a trained model.

[0072] A model capable of inferring the object description 50 only in a specific domain may be used for the object estimator 31. However, in order to be commonly usable for tasks in various domains, it is desirable to use a model capable of inferring the object description 50 generically as the object estimator 31, rather than a model capable of inferring the object description 50 only in such a specific domain. In one example, the object estimator 31 may be equipped with a trained model for in-context learning. By being equipped with a trained model capable of in-context learning, the versatility of the object estimator 31 can be increased. In other words, the description 50 of an object present in an environment can be inferred generically, and thus it can be expected to respond to the generation of object descriptions 50 for tasks in various domains.

[0073] The input / output form of the object estimator 31 may be appropriately determined depending on the embodiment. The object estimator 31 may further receive input of any data other than the observation data 20. The input data may be provided to the object estimator 31 as is, or may be provided to the object estimator 31 after pre-processing is applied. The object estimator 31 may update any data other than the object description 50. may be output to

[0074] In one example, the information processing device 1 may further acquire environmental information 22 related to the environment. The object estimator 31 may further use the acquired environmental information 22 to generate the object description 50. That is, generating the object description 50 from the acquired observation data 20 may be configured by generating the object description 50 from the acquired observation data 20 and the environmental information 22. According to one example of the present embodiment, by further using the environmental information 22, it is possible to expect improvement in the accuracy of generating the object description 50 in the problem description 5.

[0075] The environmental information 22 provided to the object estimator 31 may be appropriately selected according to the embodiment. The environmental information 22 may include at least one of the domain description 23 and the domain information 24. For example, the environmental information 22 may include at least a part of the domain description 23. For example, the environmental information 22 may include attribute information 241 of an object existing in the environment as the domain information 24. This makes it possible to more precisely identify an object existing in the environment by the attribute information 241, and therefore it is expected that the accuracy of generating the object description 50 in the problem description 5 can be improved. For example, when the object estimator 31 is equipped with a trained model for in-context learning, the environmental information 22 may include one or more input / output samples 243 as the domain information 24. The input sample may be composed of a sample of the observation data 20. When the environmental information 22 other than the input / output sample 243 is provided to the object estimator 31, the input sample may include a sample of the environmental information 22. The output sample may be composed of a correct answer sample of the object description 50 corresponding to the input sample.

[0076] The domain description 23 and the domain information 24 (attribute information 241, input / output sample 243) may be prepared as appropriate depending on the embodiment. In one example, the domain description 23 and the domain information 24 may be prepared in advance. The domain description 23 and the domain information 24 may be held in a memory resource in the information processing device 1, or may be provided to the information processing device 1 from the outside when used. In another example, at least a part of the domain description 23 and the domain information 24 may be generated as appropriate when used. For example, reference information including a list of objects and attribute information may be prepared in advance. The information processing device 1 may extract candidates for objects present in the environment from at least one of the instruction information 21 (e.g., language information) and the domain description 23 (e.g., a description defining the type of the target object). The information processing device 1 may generate attribute information 241 to be provided to the object estimator 31 by comparing the list of extracted object candidates with the list of reference information and extracting attribute information of objects matching the object candidates from the reference information. The process of generating this attribute information 241 may be executed as a pre-processing or may be executed as a process within the inference module 3.

[0077] The initial state estimator 33 may be configured to generate an initial state description 51 of one or more objects present in the environment from the acquired observation data 20. Generating the problem statement 5 using the reasoning module 3 may include generating an initial state description 51 of the objects present in the environment using the initial state estimator 33. The observation data 20 represents initial states of the objects present in the environment. Therefore, according to one example of the present embodiment, the initial state description 51 in the problem statement 5 can be appropriately generated.

[0078] As long as the description 51 of the initial state can be generated from the observation data 20, the configuration of the initial state estimator 33 is not particularly limited and may be appropriately selected depending on the embodiment. As with the object estimator 31, the above description regarding the configuration of the inference module 3 may also be applied to the initial state estimator 33. The initial state estimator 33 may be configured by at least one of a rule-based model and a trained model.

[0079] The initial state estimator 33 has a model that can infer the description of the initial state 51 only in a specific domain. However, in order to be commonly usable for tasks in various domains, it is desirable to use a model capable of inferring the description 51 of the initial state in a general manner as the initial state estimator 33, rather than a model capable of inferring the description 51 of the initial state only in such a specific domain. In one example, the initial state estimator 33 may include a trained model for in-context learning. By including a trained model capable of in-context learning, the versatility of the initial state estimator 33 can be increased. In other words, the description 51 of the initial state of an object present in the environment can be inferred in a general manner, and this can be expected to support the generation of the description 51 of the initial state for tasks in various domains.

[0080] The input / output form of the initial state estimator 33 may be appropriately determined according to the embodiment, similar to the object estimator 31. The initial state estimator 33 may further receive an input of any data other than the observation data 20. The input data may be provided to the initial state estimator 33 as is, or may be provided to the initial state estimator 33 after preprocessing is applied. The input data may be provided to the object estimator 31 first, and the calculation result (including intermediate calculation result) of the object estimator 31 on the input data may be provided to the initial state estimator 33. For example, providing the observation data 20 to the initial state estimator 33 may include providing the observation data 20 to the initial state estimator 33 as is, providing the preprocessing result of the observation data 20 to the initial state estimator 33, and providing the calculation result of the object estimator 31 on the observation data 20 to the initial state estimator 33. The preprocessing result and the calculation result of the object estimator 31 may be, for example, a detection result of an object in the observation data 20. However, the order in which the input data is provided is not limited to such an example. The input data may be provided to the initial state estimator 33 first, and the result of the calculation of the initial state estimator 33 on the input data may be provided to the object estimator 31. For example, providing the observation data 20 to the object estimator 31 may include providing the observation data 20 directly to the object estimator 31, providing the result of pre-processing the observation data 20 to the object estimator 31, and providing the result of the calculation of the initial state estimator 33 on the observation data 20 to the object estimator 31. In addition, the initial state estimator 33 may further output any data other than the description 51 of the initial state of the object.

[0081] In one example, similar to the object estimator 31, the initial state estimator 33 may further use the environment information 22 to generate the description 51 of the initial state. That is, the generation of the description 51 of the initial state from the observation data 20 may be configured by generating the description 51 of the initial state from the observation data 20 and the environment information 22. The environment information 22 provided to the initial state estimator 33 may be appropriately selected according to the embodiment. The environment information 22 may include at least one of the domain description 23 and the domain information 24. For example, the environment information 22 may include at least a part of the domain description 23. Also, for example, the environment information 22 may include attribute information 241 as the domain information 24. Also, for example, when the initial state estimator 33 has a trained model of in-context learning, the environment information 22 may include one or more input / output samples 243 as the domain information 24. The input sample may be configured by a sample of the observation data 20 (including the preprocessing result and the calculation result of the object estimator 31). When environmental information 22 other than the input / output samples 243 is provided to the initial state estimator 33, the input samples may include samples of the environmental information 22. The output samples may be composed of correct answer samples of the initial state description 51 corresponding to the input samples. The environmental information 22 (domain description 23, domain information 24) may be prepared as appropriate depending on the embodiment. By further using the environmental information 22, it is expected that the accuracy of generating the initial state description 51 in the problem description 5 can be improved.

[0082] In addition, in one example, the calculation process of the object estimator 31 may be performed to completion before the initial state estimator 33. In this case, the object description 50 generated by the object estimator 31 may be provided to the initial state estimator 33 as input data for generating the initial state description 51. That is, the initial state estimator 33 may receive the object description 50 generated by the object estimator 31 as input data for generating the initial state description 51. 0 may be further used to generate the description 51 of the initial state. When the initial state estimator 33 has a trained model of in-context learning and the environmental information 22 provided to the initial state estimator 33 includes the input / output sample 243, the input sample may include a sample of the description 50 of the object. However, the order of operations in the inference module 3 may not be limited to this example. The calculation process of the initial state estimator 33 may be executed at least partially in parallel with the object estimator 31, or may be executed to completion before the object estimator 31. When the calculation process of the initial state estimator 33 is executed to completion before the object estimator 31, the description 51 of the initial state generated by the initial state estimator 33 may be provided to the object estimator 31. When the object estimator 31 has a trained model of in-context learning and the environmental information 22 provided to the object estimator 31 includes the input / output sample 243, the input sample may include a sample of the description 51 of the initial state.

[0083] The goal estimator 35 may be configured to generate a goal state description 52 of one or more objects in the environment from the acquired instruction information 21. Generating the problem description 5 using the reasoning module 3 may include generating a goal state description 52 of the objects in the environment using the goal estimator 35. The instruction information 21 indicates information related to the goal of the task. Therefore, according to one example of the present embodiment, the goal state description 52 in the problem description 5 can be appropriately generated.

[0084] As long as the description 52 of the goal state can be generated from the instruction information 21, the configuration of the goal estimator 35 is not particularly limited and may be appropriately selected depending on the embodiment. As with the object estimator 31 and the like, the above description regarding the configuration of the inference module 3 may also be applied to the goal estimator 35. The goal estimator 35 may be configured by at least one of a rule-based model and a trained model.

[0085] The goal estimator 35 may use a model capable of inferring the goal state description 52 only in a specific domain. However, in order to be commonly usable for tasks in various domains, it is desirable to use a model capable of inferring the goal state description 52 generically as the goal estimator 35, rather than a model capable of inferring the goal state description 52 only in such a specific domain. In one example, the goal estimator 35 may include a trained model for in-context learning. By including a trained model capable of in-context learning, the versatility of the goal estimator 35 can be increased. In other words, the goal state description 52 of an object present in the environment can be generically inferred, and this can be expected to support the generation of goal state descriptions 52 for tasks in various domains.

[0086] The input / output form of the target estimator 35 may be appropriately determined according to the embodiment, similarly to the object estimator 31 and the like. The target estimator 35 may further receive an input of any data other than the instruction information 21. The input data may be provided to the target estimator 35 as is, or may be provided to the target estimator 35 after preprocessing is applied. As with the relationship between the object estimator 31 and the initial state estimator 33, the calculation result (including intermediate calculation result) of at least one of the object estimator 31 and the initial state estimator 33 may be provided to the target estimator 35. Alternatively, the calculation result (including intermediate calculation result) of the target estimator 35 may be provided to at least one of the object estimator 31 and the initial state estimator 33. The target estimator 35 may further output any data other than the description 52 of the target state of the object.

[0087] In one example, similar to the object estimator 31, the target estimator 35 may further use the environment information 22 to generate the target state description 52. That is, generating the target state description 52 from the instruction information 21 may be configured by generating the target state description 52 from the instruction information 21 and the environment information 22. The environment information 22 to be provided to the target estimator 35 may be appropriately selected depending on the embodiment. The environment information 22 may be generated by using the domain description 23 and the domain information 24. For example, the environmental information 22 may include at least one of the domain description 23. For example, the environmental information 22 may include attribute information 241 as the domain information 24. For example, when the goal estimator 35 has a trained model for in-context learning, the environmental information 22 may include one or more input / output samples 243 as the domain information 24. The input sample may be composed of a sample of the instruction information 21. When the environmental information 22 other than the input / output sample 243 is provided to the goal estimator 35, the input sample may include a sample of the environmental information 22. The output sample may be composed of a correct answer sample of the description 52 of the goal state corresponding to the input sample. The environmental information 22 (domain description 23, domain information 24) may be appropriately prepared according to the embodiment. By further using the environmental information 22, it is expected to improve the accuracy of generating the description 52 of the goal state in the problem description 5. The environmental information 22 provided to the object estimator 31, the initial state estimator 33, and the target estimator 35 may or may not overlap at least partially.

[0088] In addition, in one example, the calculation process of the object estimator 31 may be completed before the target estimator 35. In this case, the object description 50 generated by the object estimator 31 may be provided to the target estimator 35 as input data for generating the target state description 52. Similarly, the calculation process of the initial state estimator 33 may be completed before the target estimator 35. In this case, the initial state description 51 generated by the initial state estimator 33 may be provided to the target estimator 35 as input data for generating the target state description 52. That is, the target estimator 35 may further use at least one of the object description 50 and the initial state description 51 generated by the object estimator 31 and the initial state estimator 33 to generate the target state description 52. When the goal estimator 35 has a trained model for in-context learning and the environmental information 22 provided to the goal estimator 35 includes an input / output sample 243, the input sample may include at least one of the object description 50 and the initial state description 51. However, the order of operations in the inference module 3 may not be limited to this example. The operation process of the target estimator 35 may be executed at least partially in parallel with the object estimator 31, or may be executed to completion before the object estimator 31. When the operation process of the target estimator 35 is executed to completion before the object estimator 31, the description 52 of the target state generated by the target estimator 35 may be provided to the object estimator 31. Similarly, the operation process of the target estimator 35 may be executed at least partially in parallel with the initial state estimator 33, or may be executed to completion before the initial state estimator 33. When the operation process of the target estimator 35 is executed to completion before the initial state estimator 33, the description 52 of the target state generated by the target estimator 35 may be provided to the initial state estimator 33. When at least one of the object estimator 31 and the initial state estimator 33 has a trained model for in-context learning, and the provided environment information 22 includes an input / output sample 243, the input sample may include a sample of the description 52 of the target state.

[0089] The configuration of the inference module 3 is not limited to this example, and may be appropriately changed depending on the embodiment. In another example, at least any combination of the object estimator 31, the initial state estimator 33, and the goal estimator 35 may be integrally configured. At least any one of the object estimator 31, the initial state estimator 33, and the goal estimator 35 may be omitted as appropriate. In the example of FIG. 4, the inference module 3 includes the trained model 39 for in-context learning, and at least any one of the object estimator 31, the initial state estimator 33, and the goal estimator 35 may be configured by having a trained model for in-context learning. When the inference module 3 includes the trained model 39 for in-context learning and the environmental information 22 provided to the inference module 3 includes the input / output sample 243, the information processing device 1 may use the input / output sample 243 as few-shot prompting. For example, the information processing device 1 may provide only the input / output sample 243 to the trained model 39 before generating the problem description 5, or may provide the input / output sample 243 together with other input data when generating the problem description 5. The information processing device 1 may provide the trained model 39 with the problem description 5. Then, the information processing device 1 may execute calculation processing of the trained model 39. In this way, the information processing device 1 may execute in-context learning in the trained model 39 and adapt the trained model 39 to the domain in which the problem description 5 is generated.

[0090] (Example of an object estimator) 5A is a schematic diagram illustrating an example of the object estimator 31 according to the present embodiment. In one example, the object estimator 31 may include a trained model 311 and a rule-based model 313. When the observation data 20 and the environmental information 22 are input, the trained model 311 may be configured to detect an object appearing in the observation data 20 using the environmental information 22 as a clue, and output an object detection result 201. The trained model 311 may be a trained model capable of in-context learning.

[0091] For example, the sensor S may be a camera, and the observation data 20 may be image data. The environmental information 22 may be a list of attribute information 241 that indicates the attributes of an object in linguistic expressions, such as “white bowl” and “round cutting board”. The trained model 311 is described in Reference 11 (Shilong Liu et al., “Grounding DINO: Marrying DINO with grounded pre-training for open-set object detection”, [online], [Retrieved October 24, 2023], Internet<URL:https: / / arxiv.org / abs / 2303.05499> ), Reference 12 (Alexander Kirillov et al., “Segment Anything”, [online], [Retrieved October 24, 2020], Internet<URL:https: / / arxiv.org / abs / 2304.02643> ) or the like. The detection result 201 may include at least one of a result of detecting an area in which an object is captured as a bounding box and a result of identifying the object. The rule-based model 313 may be configured to generate an object description 50 from the object detection result 201 by the trained model 311 according to a rule. The rule may be set as appropriate.

[0092] Note that the configuration of FIG. 5A is merely one example of the object estimator 31. The configuration of the object estimator 31 is not limited to the example of FIG. 5A, and may be appropriately changed according to the embodiment. In another example, the trained model 311 may be provided with environmental information 22 other than the attribute information 241. The observation data 20 may further include data in a format other than image data, or may be composed only of data in a format other than image data. The trained model 311 and the rule-based model 313 may be integrally configured. The rule-based model 313 may be replaced by the trained model. The trained model 311 may be configured to generate a description 50 of an object. In response to this, the rule-based model 313 may be omitted.

[0093] (Example of an initial state estimator) FIG. 5B is a schematic diagram showing an example of the initial state estimator 33 according to the present embodiment. In one example, the initial state estimator 33 may include a detector 331 and a trained model 333. The information processing device 1 may extract partial data 205 corresponding to each object from the observation data 20 according to the object detection result 201. The detector 331 may be configured to generate a caption 206 for each object from the extracted partial data 205 for each object. The caption 206 for each object may be configured to include information about the object appearing in the partial data 205. The trained model 333 may be configured to generate a description 51 of the initial state of the object from the object detection result 201 and the caption 206. The trained model 333 using the object detection result 201 is an example of the initial state estimator 33 using the calculation result of the object estimator 31 as the observation data 20. The detector 331 and the trained model 333 may each be a trained model capable of performing in-context learning.

[0094] For example, the part data 205 may be a part extracted by a bounding box of each object. The detector 331 may be image data. The detector 331 may be a detector based on the method described in Reference 13 (Junnan Li et al., “BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models”, [online], [Retrieved October 24, 2023], Internet<URL:https: / / arxiv.org / abs / 2301.12597> ) or the like. A question may be appropriately provided to the detector 331 together with the partial data 205. In one example, the question may be configured with predetermined text data (such as “Q: what does this object describe? A: .”) that asks about an object shown in the partial image. The trained model 333 may be a visual question answering model proposed in Reference 14 (OpenAI, “GPT-4 Technical Report”, [online], [searched October 24, 2023], Internet<URL:https: / / arxiv.org / pdf / 2303.08774.pdf> ) or the like. Accordingly, the caption 206 may be text data describing the partial data 205 (partial image data). The object detection result 201 may be appropriately converted into text data 202, and the obtained text data 202 may be provided to the trained model 333 as the detection result 201 (observation data 20). In one example, the text data 202 may be composed of text indicating the name (identification result) of the detected object and the coordinate range of the bounding box of the object. The trained model 333 may further be provided with the object description 50 generated by the object estimator 31. In addition, the input / output sample 243 may further be provided to the trained model 333 as the environment information 22 (domain information 24). In one example, the input sample may be composed of samples of the text data 202, the caption 206, and the object description 50, and the output sample may be composed of a correct answer sample of the corresponding initial state description 51. The input / output samples 243 may be used for few-shot prompting of the trained model 333.

[0095] Note that the configuration of FIG. 5B is merely one example of the initial state estimator 33. The configuration of the initial state estimator 33 is not limited to the example of FIG. 5B, and may be appropriately changed according to the embodiment. In another example, the trained model 333 may be provided with environmental information 22 other than the input / output sample 243. In another example, the trained model 333 may include an input portion that accepts input of data of other modalities than text, such as a large-scale visual language model. In this way, the trained model 333 may be configured to be able to accept input of data of other modalities. In response to this, the observation data 20 or the detection result 201 may be provided to the trained model 333 as is. The caption 206 may be composed of data in a format other than text data. In addition, the partial data 205 (observation data 20) may further include data in a format other than image data, or may be composed only of data in a format other than image data. The data format of the detection result 201 is not particularly limited, and may be appropriately selected according to the embodiment. The detector 331 may be omitted.

[0096] (Example of a target estimator) 5C is a schematic diagram of an example of a target estimator 35 according to the present embodiment. In one example, the target estimator 35 may include a trained model 351. The trained model 351 may be configured to generate a target state description 52 of an object from the instruction information 21. The trained model 351 may be capable of in-context learning.

[0097] For example, the trained model 351 may be a large-scale language model (LLM) proposed in the above-mentioned Reference 14 or the like. Accordingly, the instruction information 21 may be composed of text data instructing a target in a natural language. The trained model 351 may further be provided with at least one of an object description 50 generated by the object estimator 31 and an initial state description 51 generated by the initial state estimator 33. Also, input / output samples 243 may be further provided to the trained model 351 as the environment information 22 (domain information 24). In one example, the input samples may be composed of samples of the instruction information 21, the object description 50 and the initial state description 51, and the output samples may be composed of correct samples of the corresponding target state description 52. The input / output sample 243 may be used for few-shot prompting of the trained model 351. The trained model 351 may be prepared in common with the trained model 333 of the initial state estimator 33, or may be prepared separately.

[0098] Note that the configuration of FIG. 5C is merely one example of the target estimator 35. The configuration of the target estimator 35 is not limited to the example of FIG. 5C, and may be appropriately changed according to the embodiment. In another example, the trained model 351 may be provided with environmental information 22 other than the input / output sample 243. In another example, the trained model 351 may be configured to be able to accept input of data of other modalities, similar to the trained model 333. Accordingly, the trained model 351 may be provided with data in a format other than text data. The instruction information 21 may further include data in a format other than text data, or may be composed of only data in a format other than text data.

[0099] [Robot device] The robot device R is not particularly limited and may be appropriately selected according to the embodiment. The robot device R may be, for example, an industrial robot in a production line, an autonomous robot configured to be able to operate autonomously, a mobile body configured to be able to move, etc. The industrial robot may be, for example, a vertical articulated robot, a horizontal articulated robot (SCARA robot), a parallel link robot, an orthogonal robot, etc. The autonomous robot may be, for example, a humanoid robot, a guide robot, an agricultural robot, a care robot, a security robot, a transport robot, etc. The content of the autonomous processing may be appropriately selected according to the embodiment. The mobile body may include, for example, a cleaning robot, the above-mentioned autonomous robot (including a mobile robot) configured to be able to move, a vehicle configured to be able to be driven automatically, an aircraft (drone, etc.) capable of flying automatically, etc. The robot device R may exist in a real space or a virtual space. The action plan using the generated problem description 5 may be performed for the control of the robot device R in a real space, or may be performed for the simulation of the robot device R in a virtual space.

[0100] The generation of the problem description 5 according to this embodiment may be applied to any task that the robot device R can accomplish. The task may be, for example, work, movement, etc. The work may be, for example, assembly, cooking, cleaning, chemical experiments, etc. When the robot device R is equipped with a communication device, the sensing data (observation data 20) may include data obtained by communication. For example, in a scene where a behavior plan (movement plan) of a moving object is generated, the sensing data may include data obtained by road-to-vehicle communication, vehicle-to-vehicle communication, etc.

[0101] The robot device R may be operated in a situation where it coexists with humans. In this situation, the instruction information 21 may be given by a human (operator). The generated problem description 5 may be output in a manner perceptible to humans (for example, may be output to a display). The information processing device 1 may accept human intervention (manual correction) on the generated problem description 5. In one example, the driving conditions of the robot device R, such as the amount of movement, angle, attitude, and swing, may be corrected by human intervention. The problem description 5 may be generated through dialogue-style interaction between the information processing device 1 and the operator by using a trained model capable of in-context learning in the inference module 3 (at least one of the object estimator 31, the initial state estimator 33, and the target estimator 35). The input by the human may be given directly to the information processing device 1, or indirectly to the information processing device 1 via the robot device R. The input by the human may be given by any method, such as operating an input device (including text input), recording, or capturing an image. The planner may be configured to generate a plan of action from the problem statement 5, taking into account the presence of humans.

[0102] [Output problem description] The output format of the problem description 5 is not particularly limited and may be appropriately selected depending on the embodiment. In a simple example, the generated problem description 5 may be output as is. Outputting the problem description 5 may include outputting the problem description 5 to at least one of a memory resource of the information processing device 1, an output device connected to the information processing device 1, and an external computer. The output method may be appropriately selected depending on the embodiment. For example, the problem description 5 may be output by text, an image, a sound, or a combination of these.

[0103] In another example, outputting the problem description 5 may include at least a part of providing the problem description 5 to a planner and generating an action plan by the planner. When including generating an action plan by the planner, outputting the problem description 5 may further include controlling an operation of the robotic device R (i.e., driving the robotic device R) in accordance with the generated action plan.

[0104] The generation of the action plan by the planner may be executed by at least one of the information processing device 1 and one or more external computers. When the planner includes a symbolic planner 61 and a motion planner 65, the information processing device 1 may execute the arithmetic processing of one of the symbolic planner 61 and the motion planner 65, and the external computer may execute the arithmetic processing of the other. The arithmetic processing of both the symbolic planner 61 and the motion planner 65 may be executed by either the information processing device 1 or one or more external computers. When the arithmetic processing of both the symbolic planner 61 and the motion planner 65 are executed by one or more external computers, the external computer that executes the arithmetic processing of the symbolic planner 61 and the external computer that executes the arithmetic processing of the motion planner 65 may be the same or different.

[0105] Furthermore, the control of the robot apparatus R according to the generated behavior plan may be executed by at least one of the information processing device 1 and an external computer. When at least a part of the generation of the behavior plan by the planner and the control of the robot apparatus R are executed by an external computer, the external computer that executes the generation of the behavior plan by the planner and the external computer that controls the robot apparatus R may be the same or different. One or more external computers may execute at least a part of the generation of the behavior plan, and the information processing device 1 may execute the control of the robot apparatus R according to the generated behavior plan. The information processing device 1 and the external computer may appropriately exchange data by any method such as data communication.

[0106] §2 Configuration Example [Hardware configuration] Fig. 6 illustrates a schematic example of a hardware configuration of the information processing device 1 according to the present embodiment. In the example of Fig. 6, the information processing device 1 according to the present embodiment is a computer to which a control unit 11, a storage unit 12, an external interface 13, an input device 14, an output device 15, and a drive 16 are electrically connected.

[0107] The control unit 11 includes a hardware processor such as a CPU (Central Processing Unit), a RAM (Random Access Memory), and a ROM (Read Only Memory), and is configured to execute information processing based on programs and various data. The control unit 11 (CPU) is an example of a processor resource. The storage unit 12 may be configured, for example, as a hard disk drive, a solid state drive, or the like. The storage unit 12, the RAM, and the ROM are examples of memory resources. In this embodiment, the storage unit 12 stores various information such as a program 81, module data 300, and the like.

[0108] The program 81 is a program for causing the information processing device 1 to execute information processing (FIG. 8 described later) related to the generation of the problem statement 5. The program 81 includes a series of instructions for the information processing. The module data 300 indicates information related to the inference module 3. As long as the inference module 3 can be reproduced when generating the module data 300, the configuration of the module data 300 is not particularly limited and may be appropriately determined according to the embodiment. In one example, when the module data 300 includes a machine learning model, the module data 300 may include information indicating values ​​of calculation parameters adjusted by machine learning. In some cases, the module data 300 may further include information indicating the configuration of the machine learning model (e.g., the structure of a neural network, etc.). When the module data 300 includes a rule-based model, the module data 300 may include information indicating a rule. When the inference module 3 has the configuration of FIG. 4, information regarding the object estimator 31, the initial state estimator 33, and the target estimator 35 may be held as separate data (files), or at least any combination may be held as the same data. The module data 300 may be incorporated into the program 81.

[0109] The external interface 13 is configured to connect to an external device in a wired or wireless manner. The external interface 13 may be, for example, a Universal Serial Bus (USB) port, a communication port, a dedicated port, or the like. When the external interface 13 includes a communication port, the communication standard of the communication port may be selected arbitrarily. In this embodiment, the information processing device 1 may be connected to an external device (for example, a sensor S, a robot device R, an external computer, or the like) via the external interface 13.

[0110] The input device 14 is a device for performing input, such as a mouse, a keyboard, etc. The output device 15 is a device for performing output, such as a display, a speaker, etc. An operator can operate the information processing device 1 by using the input device 14 and the output device 15. The input device 14 and the output device 15 may be connected via an external interface 13. The input device 14 and the output device 15 may be integrally configured, for example, by a touch panel display, etc.

[0111] The drive 16 is a device for reading various information such as a program stored in a storage medium 91. At least one of the program 81 and the module data 300 may be stored in the storage medium 91 instead of or together with the storage unit 12. The storage medium 91 is configured to accumulate various information (stored programs, etc.) by electrical, magnetic, optical, mechanical, or chemical action so that a machine such as a computer can read the information. The information processing device 1 may acquire at least one of the program 81 and the module data 300 from the storage medium 91. The storage medium 91 may be a disk-type storage medium such as a CD or a DVD, or may be a storage medium other than a disk-type storage medium such as a semiconductor memory (for example, a flash memory). The type of the drive 16 may be appropriately selected according to the type of the storage medium 91. The drive 16 may be connected via an external interface 13.

[0112] Regarding the specific hardware configuration of the information processing device 1, components can be omitted, replaced, or added as appropriate depending on the embodiment. For example, the control unit 11 may include multiple hardware processors. The hardware processor may be configured with a microprocessor, a field-programmable gate array (FPGA), a digital signal processor (DSP), a graphics processing unit (GPU), an application specific integrated circuit (ASIC), or the like. The storage unit 12 may be configured with a RAM and a ROM included in the control unit 11. At least one of the external interface 13, the input device 14, the output device 15, and the drive 16 may be omitted. The information processing device 1 may be configured with multiple computers. In this case, the hardware configurations of the computers may be the same or different. Furthermore, the information processing device 1 may be an information processing device designed specifically for the service provided, as well as a general-purpose server device, a general-purpose personal computer (PC), a tablet PC, a terminal device, or the like. When an external computer is used, the hardware configuration of the external computer may be the same as that of the information processing device 1. The external computer may be an information processing device designed specifically for the service to be provided, or may be a general-purpose server device, a general-purpose PC, a tablet PC, a terminal device, or the like.

[0113] [Software configuration] 7 illustrates a schematic example of a software configuration of the information processing device 1 according to this embodiment. The control unit 11 of the information processing device 1 loads a program 81 stored in the storage unit 12 into the RAM, and executes instructions included in the program 81 using the CPU. This causes the information processing device 1 to operate as a computer including an acquisition unit 111, a generation unit 112, and an output unit 113 as software modules. That is, in this embodiment, each software module of the information processing device 1 is realized by the control unit 11 (CPU).

[0114] The acquisition unit 111 is configured to acquire observation data 20 of the environment in which the robot device R operates, and instruction information 21 regarding a goal of the task to be given to the robot device R. In one example, the acquisition unit 111 may be configured to further acquire environmental information 22 regarding the environment. With the information processing device 1 holding the module data 300, the generation unit 112 includes an inference module 3. The generation unit 112 is configured to generate a problem description 5 of the task from the acquired observation data 20 and instruction information 21, using the inference module 3. In one example, generating the problem description 5 of the task from the observation data 20 and instruction information 21 may be configured by generating the problem description 5 of the task from the observation data 20, the instruction information 21, and the environmental information 22. The output unit 113 is configured to output the generated problem description 5.

[0115] In this embodiment, an example is described in which each software module of the information processing device 1 is realized by a general-purpose CPU. However, some or all of the above software modules may be realized by one or more dedicated processors or chipsets. Each of the above modules may be realized as a hardware module. With regard to the software configuration of the information processing device 1, modules may be omitted, replaced, or added as appropriate depending on the embodiment.

[0116] §3 Example of operation 8 is a flowchart showing an example of a processing procedure of the information processing device 1 according to this embodiment. The following processing procedure is an example of an information processing method executed by a computer. However, the following processing procedure of the information processing device 1 is merely an example, and each step may be changed as much as possible. In addition, steps may be omitted, replaced, or added to the following processing procedure as appropriate depending on the embodiment.

[0117] (Step S101) In step S101, the control unit 11 operates as the acquisition unit 111 and acquires the observation data 20 and the instruction information 21. In one example, the observation data 20 may be composed of sensing data of one or more sensors S, and the instruction information 21 may be composed of language information that instructs a target in a natural language. In another example, the control unit 11 may further acquire the environment information 22. The environment information 22 may include at least one of a domain description 23 and domain information 24. The domain information 24 may include attribute information 241. If the inference module 3 includes a trained model 39 for in-context learning, the domain information 24 may include an input / output sample 243. After acquiring the observation data 20 and the instruction information 21, the control unit 11 proceeds to the next step S102.

[0118] (Step S102) In step S102, the control unit 11 operates as the generation unit 112 and generates a problem description 5 of the task from the acquired observation data 20 and instruction information 21 using the inference module 3. In one example, when environmental information 22 is further acquired, the control unit 11 uses the inference module 3 to In this case, the problem description 5 may be generated from the observation data 20, the instruction information 21, and the environment information 22.

[0119] The generated problem description 5 includes descriptions (51, 52) of initial and goal states of one or more objects present in the environment. In one example, the generated problem description 5 may follow a predefined format. The generated problem description 5 may further include descriptions 50 of objects present in the environment.

[0120] In one example, generating the problem statement 5 using the reasoning module 3 may include at least one of generating an object statement 50 by the object estimator 31, generating an initial state statement 51 by the initial state estimator 33, and generating a goal state statement 52 by the goal estimator 35. When the reasoning module 3 includes a trained model 39 for in-context learning and the domain information 24 includes input / output samples 243, the control unit 11 may adapt the trained model 39 to the domain for generating the problem statement 5 using the input / output samples 243 as few-shot prompting. After generating the problem statement 5, the control unit 11 proceeds to the next step S103.

[0121] (Step S103) In step S103, the control unit 11 operates as the output unit 113 and outputs the generated problem description 5. In one example, the control unit 11 may output the problem description 5 to at least one of the memory resources of the information processing device 1, the output device 15, and an external computer. In another example, the control unit 11 may provide the problem description 5 to a planner and execute at least a part of the process of generating an action plan by the planner as the process of step S103. When the information processing device 1 executes the calculation process of the planner, the control unit 11 may appropriately acquire information about the planner. For example, the information about the planner may be stored in at least one of the memory resources (e.g., RAM, storage unit 12) of the information processing device 1 and the external computer, and the control unit 11 may acquire the information about the planner from either of them. When the process of generating an action plan by the planner is executed by the external computer, the control unit 11 may provide the generated problem description 5 to the external computer as the process of step S103. In another example, when a behavior plan is generated by the information processing device 1 or an external computer, the control unit 11 may control the operation of the robot device R according to the generated behavior plan as the processing of step S103. Controlling the operation of the robot device R may include directly controlling the robot device R and indirectly controlling the robot device R by giving an instruction to a controller of the robot device R. At least one of the generation of the behavior plan by the planner and the control of the robot device R may be executed separately from the processing of step S103. The control of the robot device R may be omitted. When the control of the robot device R is omitted, the generation of the behavior plan by the planner may also be omitted. When the output of the problem description 5 is completed, the control unit 11 ends the processing procedure of the information processing device 1 according to this operation example.

[0122] In one example, in the process of generating an action plan by the planner, if the planner outputs an error message 615, the control unit 11 may execute a process related to the correction of the problem description 5. For example, as shown in FIG. 3, when the inference module 3 includes a trained model 39 for in-context learning, the control unit 11 may acquire the output error message 615. Then, the control unit 11 may provide the acquired error message 615 and the problem description 5 to the inference module 3 and execute the calculation process of the inference module 3 again. At the time of this re-prompt, the control unit 11 may further provide at least a part of the input data 200 to the inference module 3. As a result, the control unit 11 may generate a new corrected problem description 5. The control unit 11 may recursively repeat the process of correcting the problem description 5 by this re-prompt.

[0123] [Features] In this embodiment, the problem description 5 generated in step S102 includes descriptions (51, 52) of the initial state and the goal state of each of one or more objects present in the environment, so that the planner generates an action plan to reach the goal state from the initial state of the task. The description 51 of the initial state describes the initial state of each of one or more objects present in the environment. On the other hand, the description 52 of the goal state describes the goal state of each of one or more objects to be reached by accomplishing the task. In other words, each description (51, 52) indicates each state before and after the performance of the task, and is interpretable by humans. Therefore, the problem description 5 has high explanatory power. Therefore, according to this embodiment, it is possible to obtain a highly explanatory output for obtaining an action sequence (control command) to be given to the robot device R.

[0124] § 4 Variations Although the embodiment of the present invention has been described above in detail, the above description is merely an example of the present invention in every respect. Various improvements or modifications may be made to the above embodiment as appropriate. For example, the following modifications are possible. In the following, the same reference numerals are used for components similar to those in the above embodiment, and the description of the same points as those in the above embodiment is omitted as appropriate. The following modifications can be combined as appropriate.

[0125] <4.1> In the above embodiment, the sensor S may be installed so that the observation direction can be changed by being attached to a driving device or including a driving device. For example, if the sensor S includes a camera, the camera may be installed so that the shooting direction can be changed by being attached to an electric pan head. The electric pan head is an example of a driving device. The information processing device 1 may be configured to be able to directly or indirectly drive the driving device. In this way, the information processing device 1 may appropriately change the observation direction of the sensor S.

[0126] In addition, the environment information 22 may include a designation list of objects to be observed. In the above step S101, the control unit 11 of the information processing device 1 may acquire the environment information 22 including the designation list. The attribute information 241 may also serve as the designation list of objects. In the above step S102, the control unit 11 may determine whether or not all objects included in the designation list have been detected from the observation data 20. In the example of FIG. 4, the control unit 11 may determine whether or not all objects included in the designation list have been detected from the observation data 20 in the calculation process of the object estimator 31 or the initial state estimator 33. In one example, when an object detector is provided, such as the trained model 311 of FIG. 5A, the control unit 11 may determine whether or not all objects included in the designation list have been detected according to the object detection result by the detector for the observation data 20. In another example, the control unit 11 may determine whether or not all objects included in the designation list have been detected according to whether or not all objects included in the designation list are included in at least one of the object description 50 and the initial state description 51. If at least some of the objects included in the designation list are not detected from the observation data 20, the control unit 11 may drive the driving device of the sensor S to appropriately change the observation direction of the sensor S. The direction and amount of change may be determined by any method, such as randomly or according to a predetermined rule. Then, the control unit 11 may obtain the observation data 20 again and execute the process of step S102 again. The control unit 11 may repeatedly execute these processes until all of the objects included in the designation list are detected. On the other hand, if all of the objects included in the designation list are detected from the observation data 20, the control unit 11 may proceed with the process of generating the problem description 5. According to this modification, the observation direction of the sensor S is corrected to an appropriate direction, and it is expected that the accuracy of generating the problem description 5 will be improved.

[0127] The above-described confirmation process of the detected object may be appropriately modified depending on the embodiment. For example, the designation list may further include objects (recommended objects) that may or may not be observed. If only the recommended object is not detected, the control unit 11 may output an alert and then proceed with the process of generating the problem description 5. Alternatively, the control unit 11 may stop the process of generating the problem description 5 and inquire of the operator whether or not to continue the process of generating the problem description 5.

[0128] <4.2> The domain description 23 may include a part that depends on the robot device R (e.g., a description that defines the skills of the robot device R) and a part that is related to the environment (e.g., a description that defines the type of target object). Thus, similar to the method of generating the problem description 5 in the above embodiment, the information processing device 1 may use a calculation module to generate at least a part of the part related to the environment of the domain description 23 from at least a part of the observation data 20, the instruction information 21, and the environmental information 22. The calculation module may be configured of at least one of a trained model and a rule-based model. According to this modification, it is expected that the effort required to prepare the domain description 23 can be reduced.

[0129] <4.3> In the example of Fig. 4 above, the reasoning module 3 is configured to individually generate each description (50, 51, 52) of the object, the initial state, and the goal state in the problem description 5. However, the configuration of the reasoning module 3 is not limited to this example. In another example, the reasoning module 3 may be configured to integrally generate at least any combination of the descriptions (50, 51, 52) of the object, the initial state, and the goal state in the problem description 5.

[0130] Fig. 9 shows a schematic diagram of an example of the configuration of an inference module 3A according to another embodiment. The inference module 3A includes an object detector 371, a caption model 373, and a large-scale language model 375. In the example of Fig. 9, the observation data 20 may be composed of image data, and the instruction information 21 may be composed of text data instructing a target in a natural language. The object detector 371 includes an open vocabulary object recognition (OPO) model proposed in the above-mentioned References 11 and 12, etc. For the caption model 373, a visual question answering model proposed in the above-mentioned reference 13 or the like may be used. For the large-scale language model 375, a model proposed in the above-mentioned reference 14 or the like may be used.

[0131] First, the control unit 11 of the information processing device 1 may detect an object from the observation data 20 using the object detector 371. The control unit 11 may extract partial data (partial image data) of each object from the observation data 20 according to the object detection result by the object detector 371, and provide the extracted partial data to the caption model 373. The control unit 11 may generate a caption for each object from the extracted partial data of each object using the caption model 373. Then, the control unit 11 may generate each description (50, 51, 52) of the problem description 5 from the text data of the object detection result by the object detector 371, the text data of the caption by the caption model 373, and the instruction information 21 using the large-scale language model 375. According to this modification, it is possible to simplify the calculation process.

[0132] <4.4> In the above embodiment, the re-prompting method in FIG. 3 may be appropriately improved depending on the embodiment. In one example, the re-prompting method in FIG. 3 is described in Reference 15 (Jason Wei et al., “Chain-of-Thought Prompting Elicits Reasoning in Large Language Models”, [online], [searched October 24, 2023], Internet<URL:https: / / arxiv.org / pdf / 2201.11903.pdf> ), Reference 16 (Takeshi Kojima et al., “Large Language Models are Zero-Shot Reasoners”, [online], [Retrieved October 24, 2020], Internet<URL:https: / / arxiv.org / abs / 2205.11916> ), Reference 17 (Denny Zh ou et al., “Least-to-Most Prompting Enables Complex Reasoning in Large Language Models”, [online], [Retrieved October 24, 2023], Internet<URL:https: / / arxiv.org / abs / 2205.10625> ) or the like may be used. The Chain-of-Thought prompt is a method of obtaining a final inference result (the result of generating the problem description 5 in this embodiment) by repeating stepwise inferences through intermediate inference steps. For example, the control unit 11 of the information processing device 1 may provide the inference module 3 with template data for asking a question about an error, such as "What part of the PDDL problem do you think is causing this error?", together with the problem description 5 and the error message 615, to cause the inference module 3 to generate an explanation for the error. Thereafter, the control unit 11 may provide the inference module 3 with the problem description 5 and the error message 615 to cause the inference module 3 to generate a new problem description 5.

[0133] §5 Experimental Examples The following experiments were carried out to verify the effectiveness of the above-described embodiment, however, the present invention is not limited to the following examples.

[0134] (Inference module) The inference module according to the first embodiment has the configuration shown in FIG. 4. The object detector has the configuration shown in FIG. 5A. The object detector's trained model uses the model proposed in Reference 11. The initial state estimator has the configuration shown in FIG. 5B. The initial state estimator's detector uses the model proposed in Reference 13. The initial state estimator's trained model uses the model proposed in Reference 14. The target estimator has the configuration shown in FIG. 5C. The target estimator's trained model uses the model proposed in Reference 14.

[0135] In addition, image data was used as the observation data. Text data instructing the target in natural language was used as the instruction information. The trained model of the object estimator was given attribute information indicating the attributes of the object in linguistic expressions along with the observation data (image data). The trained model of the initial state estimator was given text data indicating the object detection results, text data of the caption by the detector, a description of the object generated by the object estimator, and input / output samples used for few-shot prompting. The trained model of the target estimator was given instruction information (text data), a description of the object generated by the object estimator, a description of the initial state generated by the initial state estimator, and input / output samples used for few-shot prompting. The number of input / output samples was three.

[0136] As the inference module according to the second embodiment, the configuration of the inference module shown in Fig. 9 was adopted. For the object detector of the inference module, the model proposed in Reference 11 was adopted. For the caption model, the model proposed in Reference 13 was adopted. For the large-scale language model, the model proposed in Reference 14 was adopted. To the large-scale language model, instruction information (text data), text data of the detection results by the object detector, text data of the caption by the caption model, and input / output samples used for few-shot prompting were provided.

[0137] PDDL was used as the problem description format. A symbolic planner was prepared to confirm whether an action plan can be generated using the generated problem description. Fast Downward and VAL (Reference 18: “KCL-Planning / VAL”, [online], [Retrieved October 24, 2023], Internet) were used as the environment to execute the symbolic planner.<URL:https: / / github.com / KCL-Planning / VAL> In the reasoning modules of the first and second embodiments, a correction mechanism by re-prompt as shown in FIG. 3 was prepared. The re-prompt was composed of the input data, the generated problem description, and an error message. The method used was the Chain-of-Thought prompt described above. The maximum number of times the problem statement could be revised by re-prompting was set to two.

[0138] (Dataset) Fig. 10 shows the types (Object types), predicates, and actions of the domain description in each prepared domain. As shown in Fig. 10, datasets were prepared in three domains: Cooking, Blocksworld, and Tower of Hanoi. The cooking task was assumed to be slicing vegetables and putting them in a bowl. Two robot arms placed on the left and right were assumed as the robot device R. The state and location of the vegetables were specified as the goal state.

[0139] Blocks-World is based on the reference 19 (Naresh Gupta et al., “On the Complexity of Blocks-World Planning”, [online], [Retrieved October 24, 2023], Internet <URL:https: / / www.semanticscholar.org / paper / On-the-Complexity-of-Blocks-World-Planning-Gupta-Nau / db01349fd0d29c9443e37b0b0aa4ddb948ace5ce> ) is a classic domain used in the past. Seven colored blocks with no overlaps were used for each problem. The robot arm was set up so that it was not always holding something at the beginning. The goal state specified the relationship between the blocks.

[0140] Hanoi is a member of the 20th edition of the IEEE International Conference on Computer Vision and Pattern Recognition (CVPR) (Ron Alford et al., “Translating HTNs to PDDL: A Small Amount of Domain Knowledge Can Go a Long Way”, [online], [Retrieved October 24, 2023], Internet<URL:https: / / www.cs.umd.edu / ~nau / papers / alford2009translating.pdf> ) is a classic domain used in other games. Three pegs and 10 disks of six colors were set. Disks of the same color were numbered in order of width. The three pegs were numbered from left to right. The initial and goal states were specified by the positions of the disks.

[0141] Figures 11A, 11B, and 11C show examples of observation data and instruction information given in the Cooking, Block World, and Hanoi domains. For each domain, one domain description was prepared. Also, a problem description of the correct answer sample was prepared for each problem. For Hanoi, the same instruction information was used for all problems.

[0142] (Evaluation Index) To evaluate whether the problem description was properly generated, four indices (R syntax , R plan , R part , R all ) was prepared. R syntax R is defined as the percentage of generated problem statements that conform to the syntax of PDDL. If VAL returns no warnings or exit codes, the generated problem statement is considered syntactically correct. plan R is defined as the ratio of times that the symbolic planner did not output an error message when generating an action plan from the generated problem description. An action plan was attempted to be generated by the symbolic planner, and if VAL returned no error messages, the action plan was considered to have been generated validly. part and R all To evaluate reproducibility, we defined R as the ratio of generated problem descriptions that contain all descriptions in the correct answer sample. part In R, we calculated the descriptions of the object, the initial state, and the goal state separately. all Then, we performed the calculations on the entire generated problem description.

[0143] (Experimental Example) In the first experimental example, problem descriptions were generated in each of the three domains using the inference module of the first embodiment. The problem description was set to be modified by re-prompting up to two times. 100 problems were prepared for each domain. Ten problem descriptions were generated for each problem by changing the combination of input and output samples. The generated problem descriptions were evaluated using the above four evaluation indices (R syntax , R plan , R part , R all ) was calculated.

[0144] In the second experimental example, problem descriptions were generated in each of the three domains using the inference module of the second embodiment. Three evaluation indices (R syntax , R plan , R all The other conditions were set to be the same as those in the first experimental example.

[0145] In the third experimental example, using the reasoning module of the first embodiment, problem descriptions were generated in each of the three domains under four conditions: (A) with revision reprompts (CR) and Chain-of-Thought (CoT) prompts (up to two revisions), (B) with revision reprompts (CR) and Chain-of-Thought (CoT) prompts (up to one revision), (C) with revision reprompts (CR) and no Chain-of-Thought (CoT) prompts (up to one revision), and (D) without revision reprompts (CR) and no Chain-of-Thought (CoT) prompts. Three evaluation indices (R syntax , R plan , R all ) was calculated. The other conditions were set to the same as in the first experimental example. The other conditions were set to the same as in the first experimental example.

[0146] (Experimental Results) 12, 13, and 14 show the calculation results of the evaluation indexes in Experimental Example 1, Experimental Example 2, and Experimental Example 3. The values ​​in parentheses in Fig. 13 show the difference from the result of Experimental Example 1.

[0147] As shown in Fig. 12, in the Hanoi domain of the first experimental example, R plan Although the results were somewhat low, R syntax and R plan The scores of R were high, i.e., it was able to generate syntactically correct problem statements. all Although the score was low in the cooking domain, R part and R all The scores for both were high. Therefore, it was found that high reproducibility was possible in generating problem descriptions. From these results, it was found that effective problem descriptions could be generated according to this embodiment.

[0148] As shown in FIG. 13, the inference module of the second embodiment was able to generate syntactically correct problem descriptions, equivalent to the inference module of the first embodiment. all Although the score of R dropped, in the Hanoi domain, all From these results, we found that it is possible to generate effective problem descriptions even if the inference module is structured as an integrated structure as shown in Figure 9.

[0149] In addition, as shown in Figure 14, by overlapping the correction re-prompt (CR) and Chain-of-Thought (CoT) prompts, R syntax , R plan and R all The results showed that the correction re-prompt (CR) and chain-of-thought (CoT) prompts were effective in obtaining appropriate problem statements.

[0150] This specification includes the following disclosure. [Appendix 1] Obtaining observation data of an environment in which the robot device operates and instruction information regarding a goal of a task to be given to the robot device; generating a problem description of the task from the acquired observation data and the instruction information using a reasoning module, the problem description including descriptions of initial and goal states of objects in the environment; and outputting the generated problem description; A control unit configured to execute Information processing device. [Appendix 2] the problem description generated follows a predefined format; 2. An information processing device according to claim 1. [Appendix 3] the generated problem description further comprises a description of the objects present in the environment; 3. The information processing device according to claim 1 or 2. [Appendix 4] The observation data is composed of sensing data from a sensor, The instruction information is composed of linguistic information that indicates the goal in a natural language. 4. An information processing device according to any one of claims 1 to 3. [Appendix 5] The inference module is configured to include a trained model for in-context learning; The control unit provides the generated problem description to a planner and, in a process of generating a behavior plan for the robot device, when the planner outputs an error message, acquiring the error message that is output; and generating a new problem description from the problem description and the error messages using the reasoning module; and further configured to perform 5. An information processing device according to any one of claims 1 to 4. [Appendix 6] In the acquiring step, the control unit is configured to further acquire environmental information related to the environment, generating the problem description from the acquired observation data and the instruction information comprises generating the problem description from the acquired observation data, the instruction information, and the environment information; An information processing device according to any one of claims 1 to 5. [Appendix 7] the generated problem description further comprises a description of the objects present in the environment; the reasoning module includes an object estimator; generating the problem statement using the reasoning module includes generating, using the object estimator, a description of the objects present in the environment from the acquired observation data. 7. An information processing device according to any one of claims 1 to 6. [Appendix 8] The object estimator comprises a trained model for in-context learning. 8. The information processing device according to claim 7. [Appendix 9] In the acquiring step, the control unit is configured to further acquire attribute information of the object present in the environment, generating a description of the object from the acquired observation data comprises generating a description of the object from the acquired observation data and the attribute information; 9. The information processing device according to claim 7 or 8. [Appendix 10] the inference module includes an initial state estimator; generating the problem statement using the reasoning module includes generating a description of the initial states of the objects in the environment using the initial state estimator. 10. An information processing device according to any one of claims 1 to 9. [Appendix 11] the initial state estimator comprises a trained model for in-context learning; 11. The information processing device according to claim 10. [Appendix 12] the inference module includes a goal estimator; generating the problem statement using the reasoning module includes generating a description of the goal state of the objects in the environment using the goal estimator. 12. An information processing device according to claim 1. [Appendix 13] the goal estimator comprises a trained model for in-context learning; 13. The information processing device according to claim 12. [Appendix 14] The computer Obtaining observation data of an environment in which the robot device operates and instruction information regarding a goal of a task to be given to the robot device; generating a problem description of the task from the acquired observation data and the instruction information using a reasoning module, the problem description including descriptions of initial and goal states of objects in the environment; and outputting the generated problem description; Execute Information processing methods. [Appendix 15] On the computer, Obtaining observation data of an environment in which the robot device operates and instruction information regarding a goal of a task to be given to the robot device; generating a problem description of the task from the acquired observation data and the instruction information using a reasoning module, the problem description including descriptions of initial and goal states of objects in the environment; and outputting the generated problem description; In order to execute program. [Explanation of symbols]

[0151] 1...information processing device, 11: control unit, 12: storage unit, 13: external interface, 14...input device, 15...output device, 16...drive, 81... program, 91... storage medium, 111...acquisition section, 112...generation section, 113...output section, 3...inference module, 300...module data, 31... object estimator, 33... initial state estimator, 35...target estimator, 20...observation data, 21...indication information, 22...Environmental information, 23...Domain description, 24...Domain information, 241...Attribute information, 5…Problem description, 50...Description of object, 51...Description of initial state, 52...Description of the target state

Claims

1. Obtaining observation data of an environment in which the robotic device operates and instruction information regarding a goal of a task to be given to the robotic device; generating a problem description for the task from the acquired observation data and the instruction information using a reasoning module, the problem description including a description of a goal state of objects in the environment; and outputting the generated problem description; a control unit configured to perform Information processing device.

2. the problem description generated follows a predetermined format; The information processing device according to claim 1 .

3. the generated problem description further includes a description of the objects present in the environment; The information processing device according to claim 1 .

4. the observation data is composed of sensing data from a sensor; the instruction information is composed of linguistic information that indicates the target in natural language; The information processing device according to claim 1 .

5. the inference module is configured to include a trained model for in-context learning; the control unit obtaining an error message for the generated problem description; and generating a new problem description from the problem description and the error message using the reasoning module; and further configured to perform The information processing device according to claim 1 .

6. In the acquiring step, the control unit is configured to further acquire environmental information related to the environment; Generating the problem description from the acquired observation data and the instruction information comprises generating the problem description from the acquired observation data, the instruction information, and the environment information. It consists of The information processing device according to claim 1 .

7. the generated problem description further comprises a description of the objects present in the environment; the inference module includes an object estimator; generating the problem statement using the reasoning module includes generating descriptions of the objects present in the environment from the acquired observation data using the object estimator; The information processing device according to claim 1 .

8. the object estimator comprises a trained model for in-context learning; The information processing device according to claim 7 .

9. In the acquiring step, the control unit is configured to further acquire attribute information of the object present in the environment; generating a description of the object from the acquired observation data comprises generating a description of the object from the acquired observation data and the attribute information; 9. The information processing device according to claim 7 or 8.

10. the inference module includes an initial state estimator; generating the problem statement using the reasoning module includes generating a description of the initial states of the objects in the environment using the initial state estimator. The information processing device according to claim 1 .

11. the initial state estimator comprises a trained model for in-context learning; The information processing device according to claim 10.

12. the inference module includes a target estimator; generating the problem statement using the reasoning module includes generating a description of the goal state of the objects in the environment using the goal estimator. The information processing device according to claim 1 .

13. the goal estimator comprises a trained model for in-context learning; The information processing device according to claim 12.

14. The computer Obtaining observation data of an environment in which the robotic device operates and instruction information regarding a goal of a task to be given to the robotic device; generating a problem description for the task from the acquired observation data and the instruction information using a reasoning module, the problem description including a description of a goal state of objects in the environment; and outputting the generated problem description; To execute Information processing methods.

15. On the computer, Obtaining observation data of an environment in which the robotic device operates and instruction information regarding a goal of a task to be given to the robotic device; generating a problem description of the task from the acquired observation data and the instruction information using an inference module, the problem description comprising: Steps, including a description of the goal state, and outputting the generated problem description; In order to execute program.

16. The instruction information in the natural language includes at least one of text input, voice input, and image input. The information processing device according to claim 4 .

17. The sensing data includes image data, depth data, infrared data, measurement data from an optical sensor, radar data, LiDAR data, sound data, position data, and measurement data from a robot device. and data obtained by communication, The measurement data of the robot device includes at least one of joint angles, hand positions, tactile data at the hand, force data at the hand, and posture measurement data. The information processing device according to claim 4 .

18. The description of the target state includes position information of a target of the object. The information processing device according to claim 1 .

19. The problem description further includes an initial state of objects in the environment. The information processing device according to claim 1 .

20. The error message is an error message output by a planner in a process of generating a behavior plan for the robotic device by providing the generated problem description to the planner. The information processing device according to claim 5 .

Images