Information processing apparatus, information processing method, and program

Abstract

To provide a technique for obtaining highly explanatory output regarding control of a robot device.SOLUTION: An information processing apparatus according to an aspect of the present invention acquires observation data of an environment in which a robot apparatus operates and instruction information related to a target of a task to be given to the robot apparatus, generates a problem description of the task from the acquired observation data and the instruction information using an inference module, and outputs the generated problem description. The problem description is configured to include descriptions of initial states and target states of objects present in the environment.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to an information processing device, an information processing method, and a program. [Background technology]

[0002] Interacting with robots using natural language is a promising way for non-experts to enable them to solve complex and diverse tasks. In recent years, research has been gaining attention on using large-scale language models (LLMs) to output sequences of robotic actions in a workspace from linguistic instructions.

[0003] For example, Non-Patent Document 1 proposes a system that generates an action sequence for a robot device from a linguistic instruction. The system proposed in Non-Patent Document 1 accepts input of instructions in natural language (linguistic instruction). The system uses an LLM (Say module) to infer from the linguistic instruction the next action of the robot device that is likely to be executed. The system also uses a value function (Can module) to infer from observation data the action that is likely to be executed. The system then integrates the results of the two inferences and, depending on the integration result, determines the action sequence to be given to the robot device (an action sequence that is likely to be executed and is feasible). [Prior art documents] [Non-patent literature]

[0004] [Non-Patent Document 1] Michael Ahn et al., “Do As I Can, Not As I Say: Grounding Language in Robotic Affordances”, [online], [Retrieved October 24, 2023], Internet<URL:https: / / arxiv.org / abs / 2204.01691> Summary of the Invention [Problem to be solved by the invention]

[0005] The present inventors have found that the above-mentioned conventional systems have the following problems. In other words, the action sequences generated by the conventional systems are specific to the trial environment and are not necessarily interpretable by humans. Because the action sequences are output as is, there is a problem in that the interpretability of the obtained output regarding the control of the robot device is low.

[0006] In one aspect, the present invention has been made in view of the above circumstances, and an object of the present invention is to provide a technique for obtaining a highly explainable output regarding the control of a robot device. [Means for solving the problem]

[0007] In order to solve the above-mentioned problems, the present invention employs the following configurations. Note that the following configurations of the invention can be combined as appropriate.

[0008] That is, an information processing device according to one aspect of the present invention includes a control unit configured to execute the steps of acquiring observation data of an environment in which a robotic device operates and instruction information regarding a goal of a task to be given to the robotic device, generating a problem description of the task from the acquired observation data and the instruction information using an inference module, and outputting the generated problem description, wherein the problem description includes descriptions of initial states and goal states of objects in the environment.

[0009] The problem description, together with the domain description, is used by the planner to generate an action plan (i.e., obtain a sequence of actions). This problem description describes the initial and goal states of the objects in the environment to accomplish the task, and is highly descriptive (human-interpretable). Therefore, with this configuration, it is possible to obtain a highly descriptive output for obtaining an action sequence (command) to be given to the robot device.

[0010] In the information processing device according to the above aspect, the generated problem description may conform to a predetermined format. According to this configuration, since the generated problem description conforms to the predetermined format, it is possible for a planner to easily generate an action plan from the problem description.

[0011] In the information processing device according to the above aspect, the generated problem description may further include a description of the objects present in the environment. With this configuration, the object description makes it easier to identify the objects present in the environment, thereby making it easier for a planner to generate an action plan from the problem description.

[0012] In the information processing device according to the above aspect, the observation data may be composed of sensing data from a sensor, and the instruction information may be composed of linguistic information specifying the goal in natural language. With this configuration, when an environment is observed by a sensor and a task goal is given in linguistic instructions, a highly descriptive output can be obtained regarding control of the robot device.

[0013] In the information processing device according to the above aspect, the inference module may be configured to include a trained model for in-context learning. The system may be further configured to, if the planner outputs an error message during the process of providing the generated problem description to a planner to generate a behavior plan for the robotic device, acquire the output error message and generate a new problem description from the problem description and the error message using the inference module. With this configuration, if an appropriate problem description cannot be obtained, the problem description can be automatically corrected. Note that the trained model is a trained machine learning model.

[0014] The control unit of the information processing device according to the above aspect may be configured to further acquire environmental information related to the environment in the acquiring step. Generating the problem description from the acquired observation data and instruction information may be configured by generating the problem description from the acquired observation data, instruction information, and environmental information. With this configuration, by further using environmental information as an explanatory variable in addition to the observation data and instruction information, it becomes possible to more accurately identify the environment in which a task is to be performed, and this can be expected to improve the accuracy of generating the problem description.

[0015] In the information processing device according to the above aspect, the generated problem description may further include a description of the object present in the environment. The inference module may include an object estimator. Generating the problem description using the inference module may include generating a description of the object present in the environment from the acquired observation data using the object estimator. With this configuration, it is possible to appropriately generate the description portion of the object in the problem description.

[0016] In the information processing device according to the above aspect, the object estimator may include a trained model for in-context learning. With this configuration, the object estimator includes a trained model for in-context learning, which enables general-purpose estimation of objects present in the environment, and is expected to be able to handle the generation of problem descriptions for various tasks.

[0017] The control unit of the information processing device according to the above aspect may be configured to further acquire attribute information of the object present in the environment in the acquiring step. Generating the object description from the observation data may be configured by generating the object description from the acquired observation data and the attribute information. With this configuration, it is possible to more accurately identify objects present in the environment using the attribute information, which can be expected to improve the accuracy of generating the object description portion of the problem description.

[0018] In the information processing device according to the above aspect, the inference module may include an initial state estimator. Generating the problem description using the inference module may include generating a description of the initial state of the object present in the environment using the initial state estimator. With this configuration, it is possible to appropriately generate the description portion of the initial state in the problem description.

[0019] In the information processing device according to the above aspect, the initial state estimator may include a trained model for in-context learning. With this configuration, the initial state estimator includes a trained model for in-context learning, which enables general-purpose inference of the initial states of objects present in the environment, and is expected to be able to respond to the generation of problem descriptions for various tasks.

[0020] In the information processing device according to the above aspect, the inference module may include a goal estimator. Generating the problem description using the inference module may include generating a description of the goal state of the object in the environment using the goal estimator. This configuration allows the description portion of the goal state in the problem description to be appropriately generated.

[0021] In the information processing device according to the above aspect, the goal estimator may include a trained model for in-context learning. With this configuration, the goal estimator includes a trained model for in-context learning, which enables general-purpose inference of the goal state of an object, and is expected to be able to respond to the generation of problem descriptions for various tasks.

[0022] The present invention is not limited to the information processing device. As another aspect of the information processing device according to each of the above aspects, one aspect of the present invention may be an information processing method that realizes all or part of each of the above configurations, a program, or a storage medium that stores such a program and is readable by a machine such as a computer. A storage medium that is readable by a machine such as a computer is a medium that stores information such as a program by electrical, magnetic, optical, mechanical, or chemical action.

[0023] For example, an information processing method according to one aspect of the present invention may be an information processing method in which a computer executes the steps of acquiring observation data of an environment in which a robotic device operates and instruction information regarding a task goal to be given to the robotic device, generating a problem description of the task from the acquired observation data and instruction information using an inference module, and outputting the generated problem description.

[0024] Furthermore, for example, a program according to one aspect of the present invention may be a program for causing a computer to execute the steps of acquiring observation data of an environment in which a robotic device operates and instruction information regarding a task goal to be given to the robotic device, generating a problem description for the task from the acquired observation data and instruction information using an inference module, and outputting the generated problem description. [Effects of the Invention]

[0025] According to the present invention, highly descriptive output can be obtained regarding the control of a robot device. [Brief explanation of the drawings]

[0026] [Figure 1] FIG. 1 shows a schematic diagram of an example of a situation in which the present invention is applied. [Figure 2] FIG. 2 is a diagram illustrating an example of input and output of an inference module according to an embodiment. [Figure 3]FIG. 3 is a diagram illustrating an example of a process for correcting a problem description according to an embodiment. [Figure 4] FIG. 4 is a diagram illustrating an example of the configuration of an inference module according to an embodiment. [Figure 5A] FIG. 5A illustrates an example of an object estimator according to an embodiment. [Figure 5B] FIG. 5B schematically illustrates an example of an initial state estimator according to an embodiment. [Figure 5C] FIG. 5C illustrates a schematic diagram of an example of a target estimator according to an embodiment. [Figure 6] FIG. 6 is a diagram illustrating an example of a hardware configuration of an information processing device according to an embodiment. [Figure 7] FIG. 7 is a diagram illustrating an example of a software configuration of the information processing device according to the embodiment. [Figure 8] FIG. 8 is a flowchart illustrating an example of a processing procedure of the information processing device according to the embodiment. [Figure 9] FIG. 9 is a diagram illustrating an example of the configuration of an inference module according to another embodiment. [Figure 10] FIG. 10 shows each description included in the domain description for each prepared domain. [Figure 11A] FIG. 11A shows an example of observation data and instruction information given in the domain (Cooking). [Figure 11B] FIG. 11B shows an example of observation data and instruction information given in the domain (Blocksworld). [Figure 11C] FIG. 11C shows an example of observation data and instruction information given in the domain (Hanoi). [Figure 12] FIG. 12 shows the results of the first experimental example. [Figure 13] FIG. 13 shows the results of the second experimental example. [Figure 14] FIG. 14 shows the results of the third experimental example. DETAILED DESCRIPTION OF THE INVENTION

[0027] An embodiment according to one aspect of the present invention (hereinafter also referred to as "the present embodiment") will be described below with reference to the drawings. However, the present embodiment described below is merely an example of the present invention in all respects. Various improvements or modifications may be made without departing from the scope of the present invention. In implementing the present invention, a specific configuration according to the embodiment may be appropriately adopted. Note that while data appearing in the present embodiment is described in natural language, more specifically, it is specified using computer-recognizable pseudo-language, commands, parameters, machine language, etc.

[0028] §1 Application Examples FIG. 1 schematically illustrates an example of a scenario in which the present invention is applied. An information processing device 1 according to this embodiment is one or more computers configured to generate a problem description 5 used by a planner to generate an action plan. Specifically, the information processing device 1 acquires observation data 20 of an environment in which the robot device R operates and instruction information 21 regarding a task goal to be given to the robot device R. The information processing device 1 generates the task problem description 5 from the acquired observation data 20 and instruction information 21 using an inference module 3. The problem description 5 includes descriptions (51, 52) of initial and goal states of one or more objects present in the environment. The information processing device 1 outputs the generated problem description 5.

[0029] The problem description 5 includes descriptions (51, 52) of the initial and goal states of one or more objects in the environment, so that the planner can generate an action plan that will reach the goal state from the initial state of the task. The initial state is the state before the task is performed. The problem description 5 may also be referred to as a state. The goal state is the state after the task is performed and the purpose is achieved. The initial state description 51 describes the initial state of each of one or more objects in the environment. On the other hand, the goal state description 52 describes the goal state of each of one or more objects to be reached by completing the task. That is, each description (51, 52) indicates the state before and after the task is performed and is interpretable by humans. In one example, each description (51, 52) indicates the state of an object rather than the behavior of the robot device R, and is therefore easy for humans to interpret even without a special understanding of the robot device R (for example, without deciphering the program code). Therefore, the problem description 5 has high interpretability. Therefore, according to this embodiment, a highly interpretable output can be obtained to obtain an action sequence (control command) to be given to the robot device R.

[0030] [Input / output data] In this embodiment, the observation data 20 and instruction information 21 are given to the inference module 3 as input data, and as a result of the calculation process of the inference module 3, the problem description 5 is obtained as output data.

[0031] The observation data 20 may be any data that represents the initial state of an object before a task is performed. The type of observation data 20 is not particularly limited and may be selected appropriately depending on the embodiment. The observation data 20 may be composed of data of one or more modalities. The environment may be observed by any method. The environment may include all events related to the situation in which the robot device R performs a task. The environment may be composed of at least a real environment and a virtual environment. For example, the environment may be composed of both a real environment and a virtual environment, such as AR (Augmented Reality) and MR (Mixed Reality). The environment may include VR (Virtual Reality), etc. The number of objects present in the environment may be arbitrary. The observation data The observation data 20 may be acquired at any timing before starting to perform a task. When multiple tasks are given, the initial state may be a state before the multiple tasks are performed, or a state before one of the multiple tasks is performed (including an intermediate state between the multiple tasks). In other words, "initial" does not necessarily mean "beginning." A task may be any work (job) for transitioning from the initial state to a goal state. The state before performing this task may be the initial state, and the state after the task is properly performed may be the goal state. The interval between tasks may be determined appropriately depending on the embodiment. Because the initial state of the object is reflected in the observation data 20, the inference module 3 can infer the description 51 of the initial state in the problem description 5 from the observation data 20.

[0032] The instruction information 21 may be any information that can identify the goal of the task. The type of instruction information 21 is not particularly limited and may be selected appropriately depending on the embodiment. The instruction information 21 may be composed of data of one or more modalities. The goal may be given as appropriate. The instruction information 21 may be acquired at any timing before starting to perform the task. The instruction information 21 may be acquired before or after the observation data 20. Furthermore, the instruction information 21 may be acquired at least partially in parallel with the observation data 20. Because the goal of the task is expressed in the instruction information 21, the inference module 3 can infer the description 52 of the goal state in the problem description 5 from the instruction information 21.

[0033] The problem description 5 includes descriptions (51, 52) of the initial state and the goal state, thereby enabling the planner to generate an action plan for the robot device R, including the feasibility (whether or not the robot device R is feasible) of the robot device R, given the skills of the robot device R. As long as the problem description 5 is of this nature, the configuration of the problem description 5 is not particularly limited and may be selected appropriately depending on the embodiment. The problem description 5 is independent of the robot device and may therefore be used to generate an action plan for any robot device. The planner may be configured appropriately to generate an action plan from the problem description 5.

[0034] As shown in FIG. 1, in one example, the skills of the robot device R may be given by the domain description 23. Also, in one example, the planner may include a symbolic planner 61 and a motion planner 65. The symbolic planner 61 may be configured to generate an action sequence 63, which is a series of abstract actions, from the problem description 5 and the domain description 23. An abstract action is any collection of actions that includes one or more actions of the robot device R and may be defined as a collection of actions that can be expressed by a desire (e.g., words, etc.). The abstract actions may be defined, for example, as a collection of meaningful (i.e., understandable by humans) actions such as grasping an object, carrying it, positioning it, etc. The symbolic planner 61 may use, for example, known symbolic planners such as Fast Downward of Reference 1 (“The Fast Downward Planning System”, [online], [searched on October 24, 2023], Internet <URL:https: / / planning.wiki / ref / planners / fd>) and LAMA planner of Reference 2 (Silvia Richter et al., “The LAMA planner: guiding cost-based anytime planning with landmarks”, [online], [searched on October 24, 2023], Internet <URL:https: / / dl.acm.org / doi / 10.5555 / 1946417.<1946420>).

[0035] The motion planner 65 may be configured to generate a sequence of control commands 67 for causing the robot device R to execute each action included in the action sequence 63 when the action sequence 63 is provided by the symbolic planner 61. The motion planner 65 may also be configured to determine whether the robot device R can execute each action included in the action sequence 63. The motion planner 65 may include, for example, a control command sequence 67 described in Reference 3 (James J. Kuffner, Jr., Steven M. LaValle, "RRT-Connect: An Efficient Approach to Single-Query Path Planning", [online], [searched October 24, 2023], Internet)<URL:https: / / www.cs.cmu.edu / afs / cs / academic / class / 15494-s12 / readings / kuffner_icra2000.pdf> ) RRT-connect, Reference 4 (Nathan Ratliff et al., “CHOMP: Gradient Optimization Techniques for Efficient Motion Planning”, [online], [Retrieved October 24, 2023], In Internet<URL:https: / / www.ri.cmu.edu / pub_files / 2009 / 5 / icra09-chomp.pdf> ) CHOMP (Covariant Hamiltonian Optimization for Motion Planning), Reference 5 (Mrinal Kalakrishnan et al., “STOMP: Stochastic Trajectory Optimization for Motion Planning”, [online], [Retrieved October 24, 2023], Internet <URL:http: / / ros.fei.edu.br / roswiki / attachments / Papers(2f)ICRA2011_Kalakrishnan / kalakrishnan_icra2011.pdf> ) may be used. The generated control command 67 may be given to the robot device R as appropriate. The robot device R may start performing the task at any timing in accordance with the given control command 67. Note that the configuration of the planner is not limited to the example of FIG. 1 and may be changed as appropriate depending on the embodiment. In another example, the planner may be configured to generate an action plan by inferring the control command 67 directly from the problem description 5 and the domain description 23.

[0036] The format of the problem description 5 is not particularly limited and may be appropriately selected depending on the embodiment, as long as it includes descriptions of the initial state and the goal state (51, 52) and can be used to generate an action plan. In one example, the generated problem description 5 may be configured to follow a predetermined format. The predetermined format may be, for example, PDDL (Planning Domain Definition Language), PDDLStream (Reference 6: "pddlstream", [online], [searched October 24, 2023], Internet<URL:https: / / github.com / caelan / pddlstream> ) etc. In one example of this embodiment, the problem description 5 to be generated conforms to a predetermined format, which makes it easier for a planner (e.g., a symbolic planner 61) to generate an action plan from the problem description 5.

[0037] In another example, the generated problem statement 5 does not have to follow a predetermined format. In this case, the generated problem statement 5 may be converted to conform to a predetermined format by an intermediate process such as a conversion program. Alternatively, the generated problem statement 5 may be used to generate an action plan without conforming to a predetermined format by a planner that can accept input in any format, such as a planner configured with a trained model for in-context learning.

[0038] The input and output data of the inference module 3 may be changed as appropriate depending on the embodiment. The inference module 3 may further receive input of any data other than the observation data 20 and the instruction information 21. The inference module 3 may further output any data other than the problem description 5. The problem description 5 may further include any information other than the descriptions (51, 52) of the initial state and the goal state.

[0039] (Example of input / output data) Fig. 2 shows an example of input and output of the inference module 3 according to this embodiment. In the example of Fig. 2, the input data includes observation data 20, instruction information 21, and environment information 22. The problem description 5 (output data) includes descriptions of the object, the initial state, and the goal state (50, 51, 52).

[0040] In one example, the environment may be observed by one or more sensors S, and accordingly, the observation data 20 may be configured by sensing data from one or more sensors S. The type of sensor S is not particularly limited and may be appropriately selected depending on the embodiment. The sensor S may include, for example, a camera, a depth sensor, an infrared sensor, an optical sensor, a radar, a LiDAR (Light Detection And Ranging), a microphone, a position sensor, and other measurement sensors. The position sensor may be, for example, a GPS (Global Positioning System) sensor, a GNSS (Global Navigation Satellite System) sensor, etc. The sensing data may be, for example, This includes image data, depth data, infrared data, optical sensor measurement data (such as marker detection results), radar data, LiDAR data, sound data, position data, and other measurement data. The sensor S may also include one or more measurement sensors for measuring the state of the robotic device R. The measurement sensors may include, for example, an encoder, a motion capture, a tactile sensor, a force sensor, an inertial measurement unit, etc. Accordingly, the sensing data may include measurement data of the robot device R (for example, joint angles, hand positions, tactile data at the hand, force data at the hand, posture measurement data, etc.). The sensor S may be arranged outside the robot device R or may be arranged in the robot device R.

[0041] In one example, the instruction information 21 may be composed of linguistic information instructing a target in natural language. The natural language instruction may be acquired by any method. The natural language instruction may be acquired by, for example, text input, voice input, image input, or other input methods. The data format of the instruction information 21 (linguistic information) may be selected appropriately depending on the embodiment. The instruction information 21 may be composed of, for example, text data, voice data, image data, or other types of data. The acquired natural language instruction data may be used as the instruction information 21 as is, or may be used as the instruction information 21 after converting its data format. As an example of the latter, the instruction data may be converted using a conversion model, such as from voice to text or from text to voice, and the converted instruction data may be used as the instruction information 21. The conversion may include any analysis process, such as voice analysis. The conversion model may be included in the inference module 3 or may be prepared separately from the inference module 3. The conversion model may be composed of at least one of a trained model (a trained machine learning model) and a rule-based model. The instructions in natural language may be given manually by an operator or automatically by computer processing. The instruction information 21 may be given appropriately depending on the task. In one example, the instruction information 21 is The instruction information 21 may be given each time, such as every time the robot R reaches a target. In another example, when a target is specified in advance, such as the Tower of Hanoi, the instruction information 21 may be given. In one example of this embodiment, when the environment is observed by the sensor S and a task target is given by verbal instruction, a highly descriptive output can be obtained regarding the control of the robot device R.

[0042] The configurations of the observation data 20 and the instruction information 21 may be modified as appropriate depending on the embodiment. The observation data 20 may include, in addition to sensing data, any data other than sensing data, such as manually provided data or data generated by computer processing. In another example, the observation data 20 may not include sensing data and may instead be composed of any data other than sensing data. In another example, the instruction information 21 may be provided in a format other than natural language. For example, the instruction information 21 may be provided in a text format using symbols other than natural language. In another example, the instruction information 21 may be composed of an image indicating a target state of an object present in the environment. The image may be composed of at least one of a real image and a virtual image.

[0043] In addition, in one example, the information processing device 1 may be configured to further acquire environmental information 22 related to the environment. Generating the problem description 5 from the acquired observation data 20 and instruction information 21 may be configured by generating the problem description 5 from the acquired observation data 20, instruction information 21, and environmental information 22. According to one example of the present embodiment, the environmental information 22 imposes constraints on the environment in which the task is performed, thereby narrowing down the conditions for generating the problem description 5. In other words, by further using the environmental information 22 as an explanatory variable in addition to the observation data 20 and instruction information 21, the environment in which the task is performed can be more easily identified. Therefore, an improvement in the accuracy of generating the problem description 5 can be expected.

[0044] It should be noted that the environment information 22 may include any information related to the relationships of the robotic device R that may be involved in generating the problem description 5. In one example, the environment information 22 may include at least one of a domain description 23 and domain information 24.

[0045] The domain description 23 defines phenomena common to all problems, including the skills of the robot device R. As described above, the domain description 23 may be used together with the problem description 5 to generate an action plan by the planner. When the problem description 5 is given in accordance with a predetermined format such as PDDL or PDDLStream, the domain description 23 may also be given in accordance with the predetermined format. For example, when PDDL is adopted, the problem description 5 may be problem.pddl, and the domain description 23 may be domain.pddl. Also, in one example, the domain description The statement 23 is a statement that defines the skills of the robot device R (e.g., actions), the actions that exist in the environment, The target object state may include descriptions (e.g., predicates) that define the state of the target object to be executed, descriptions (e.g., types) that define the type of the target object, and descriptions (e.g., requirements) that confirm compatibility with the planner. The state of the target object is used to execute the defined skills (behaviors) on the robotic device R. The state information may include the states that the target object can take when executed.

[0046] The domain information 24 may include any information that limits the domain for generating the problem statement 5. The domain information 24 may also be referred to as domain knowledge. The domain information 24 may be composed of any information related to the environment other than the domain description 23. The domain information 24 may supplement the conditions for generating the problem statement 5 together with the domain description 23. In one example, the domain information 24 may include attribute information 241 of objects present in the environment. The attribute information 241 may include, for example, the name and characteristics of the object. The characteristics may include, for example, characteristics related to appearance such as color, shape, size, etc. Examples of characteristics related to appearance include a cutting board being round and a counter being black. This makes it possible to narrow down the characteristics of objects appearing in the observation data 20, and as a result, it is expected that the accuracy of inference for the observation data 20 will be improved. In one example, as will be described later, the inference module 3 is configured to include a trained model for in-context learning. When a trained model is generated, the domain information 24 may include one or more input / output samples 243 for the trained model. For example, the output sample may be a ground truth sample of a portion of the problem statement 5 generated by the trained model. The input sample may be a sample of a portion of the observation data 20, instruction information 21, and environment information 22 provided to the trained model to obtain a ground truth sample of the problem statement 5.

[0047] The format of the input data (observation data 20, instruction information 21, and environmental information 22) provided to the inference module 3 is not particularly limited and may be determined appropriately depending on the embodiment. In one example, the observation data 20, instruction information 21, and environmental information 22 may be input to the inference module 3 as is. In another example, preprocessing may be applied to at least one of the observation data 20, instruction information 21, and environmental information 22, and the preprocessed data may be input to the inference module 3. The preprocessing may include any computational process, such as a process of analyzing information, a process of adding information, or a process of reducing information. A computational model that performs the preprocessing may be included in the inference module 3 or may be prepared separately from the inference module 3. The computational model may be configured as at least one of a trained model (a trained machine learning model) and a rule-based model. For example, in a case where the observation data 20 is composed of image data, the inference module 3 may be configured to accept input of the image data. Alternatively, the image data may be analyzed by any method, such as image processing or an analytical model. This analysis process is an example of preprocessing. The analysis result may include, for example, a bounding box detection result, an identification result of an object appearing in the image data, etc. The inference module 3 may be configured to accept an input of the analysis result for the image data.

[0048] In one example, the generated problem description 5 may further include a description 50 of objects present in the environment. The object description 50 may correspond to a list of objects present in the environment. The range of objects included in the description 50 may be determined appropriately depending on the embodiment. For example, the object description 50 may be a list of all objects that can be observed in the target environment. Alternatively, for example, the object description 50 may be a list of a portion of objects that can be observed. The portion of objects may be objects of interest, such as objects that can be involved in the task. In this case, objects of no interest (e.g., objects that cannot be involved in the task) may be omitted from the object description 50. According to one example of the present embodiment, the object description 50 makes it easier to identify objects present in the environment, thereby making it easier for a planner (e.g., a symbolic planner 61) to generate an action plan from the problem description 5. Note that when the descriptions (51, 52) of the initial state and the goal state are obtained in a predetermined format, the object description 50 may also be obtained in a predetermined format. For example, when PDDL is adopted, the description of objects in problem.pddl is an example of a description of an object 50, the description of init (initial state) is an example of a description of an initial state 51, and the description of goal is an example of a description of a target state 52.

[0049] The input / output form of the inference module 3 is not limited to the example in FIG. 2 and may be changed as appropriate depending on the embodiment. At least a part of the environment information 22 may be omitted. The object description 50 may be omitted in the problem statement 5. In one example, when the robot device R operates in an environment where no other objects exist other than the robot device R (for example, when a drone flies in the air where there is nothing), the object description 50 may be omitted in the problem statement 5.

[0050] [Problem description correction] In one example, the information processing device 1 may be configured to execute a process related to modifying the generated problem description 5. The process of modifying the problem description 5 may be executed automatically by a computer or manually by an operator or the like.

[0051] As an example, the information processing device 1 may determine whether the generated problem description 5 is compatible with a planner (e.g., a symbolic planner 61). The problem description 5 may be generated by any method, such as using at least one of a rule-based model and a method for generating a problem. For example, if the problem description 5 is generated to conform to a predetermined format, the information processing device 1 may determine whether the generated problem description 5 conforms to the predetermined format by evaluating whether the generated problem description 5 conforms to the predetermined format. If the problem description 5 is determined to conform to the planner, the generated problem description 5 may be provided to the planner as appropriate. On the other hand, if the problem description 5 is determined to not conform to the planner, the information processing device 1 may modify the generated problem description 5 as appropriate. For example, the information processing device 1 may modify the generated problem description 5 using a modification model. The modification model may be configured using at least one of a trained model and a rule-based model. Furthermore, for example, the information processing device 1 may output the problem description 5 to an output device together with factors that cause the problem description 5 to not conform to the planner (e.g., indicating parts that do not conform to the predetermined format). The information processing device 1 may accept modifications to the problem description 5 by an operator via an input device. The information processing device 1 may then modify the problem description 5 according to the received content. As another example, the information processing device 1 may execute the above process of correcting the problem description 5 after omitting the process of determining whether or not it is suitable for the planner.

[0052] The timing of executing the process of correcting the problem description 5 is not particularly limited and may be determined appropriately depending on the embodiment. In one example, the information processing device 1 may execute the process of correcting the problem description 5 before the problem description 5 is used to generate an action plan by a planner (e.g., the symbolic planner 61). In another example, after the generated problem description 5 is provided to the planner, the information processing device 1 may execute the process of correcting the problem description 5 in response to an error occurring in the planner's attempt to generate an action plan, such as outputting an error message or being unable to generate a proper action plan. Note that the process of correcting the problem description 5 may be executed by an external computer other than the information processing device 1. Furthermore, the process of correcting the problem description 5 may be omitted.

[0053] FIG. 3 schematically illustrates an example of a problem statement 5 revision process according to the present embodiment. In one example, the inference module 3 may be configured to include a trained model 39 for in-context learning. In-context learning refers to acquiring the ability to make specific inferences through the context of inputs (prompts), such as input / output samples. In one example, the trained model 39 can acquire the ability of in-context learning by including a self-attention mechanism and an autoregressive model. The trained model 39 may be, for example, a large-scale language model (LLM), a large-scale visual language model (LVLM), etc. The large-scale visual language model may include a visual question answering model, an open vocabulary object detection model, an open vocabulary object segmentation model, etc. Alternatively, the trained model 39 may be a large-scale language model combined with one or more other modalities (e.g., sound), such as audio question answering. Furthermore, the trained model 39 may be, for example, a Large Audio Model. It may also be a large-scale model of one or more modalities other than language. The data format of the input to the trained model 39 may be selected appropriately depending on the embodiment.

[0054] In the example of Fig. 3, the inference module 3 includes a trained model 39 capable of in-context learning, so that when an error occurs, the inference module 3 can adapt to the error and modify the problem statement 5 in response to the error. Thus, in the example of Fig. 3, the information processing device 1 may generate the problem statement 5 by providing input data 200 to the inference module 3 and executing the calculation process of the inference module 3. The input data 200 includes observation data 20 and instruction information 21 of a target scene for which an action plan is to be generated. In one example, the input data 200 may further include environmental information 22.

[0055] The generated problem description 5 may be given to a planner as appropriate. An action plan may be generated using the problem description 5. When the planner has the configuration of FIG. 1, the generated problem description 5 may be provided to a symbolic planner 61, which may then attempt to generate an action sequence 63 from the problem description 5. The process of generating an action plan using this planner may be executed by at least one of the information processing device 1 and an external computer other than the information processing device 1. As described above, the domain description 23 may be used together with the problem description 5 to generate the action plan.

[0056] If the problem description 5 is appropriate, the planner can generate an appropriate action plan from the problem description 5. In the example of FIG. 1, an appropriate motion sequence 63 is obtained, and the motion planner 65 can generate a sequence of control commands 67 accordingly. The generated action plan may be used as appropriate to control the motion of the robot device R. On the other hand, if the problem description 5 is inappropriate, an error occurs in the generation of the action plan by the planner. An inappropriate problem description 5 may include, for example, the problem description 5 not being compatible with the planner, the inability to generate an action plan that avoids prohibited items, or the inability to generate an action plan with the given skills of the robot device R. An inappropriate problem description 5 may include, for example, the problem description 5 not conforming to a predetermined format. In this case, an error message 615 is output from the planner.

[0057] In the process of providing the generated problem description 5 to the planner and generating a behavior plan for the robot device R, if the planner outputs an error message 615, the information processing device 1 may acquire the output error message 615. The configuration of the error message 615 is not particularly limited and may be determined appropriately depending on the embodiment, such as the configuration of the planner. Then, the information processing device 1 may provide the acquired error message 615 and problem description 5 to the inference module 3 and re-execute the calculation process of the inference module 3. That is, the information processing device 1 may use the problem description 5 and the error message 615 as a re-prompt to re-execute the process of generating the problem description 5. As a result, the information processing device 1 may generate a new problem description 5 from the problem description 5 and the error message 615 using the inference module 3 (that is, may modify the problem description 5). According to an example of this embodiment, if an appropriate problem description 5 cannot be obtained, the problem description 5 can be automatically modified.

[0058] The data provided to the inference module 3 when re-prompting does not have to be limited to the problem description 5 and the error message 615. The configuration of the re-prompt may be changed as appropriate depending on the embodiment. In another example, the information processing device 1 may provide at least a part of the input data 200 to the inference module 3 together with the problem description 5 and the error message 615. This can be expected to improve the accuracy of correcting the problem description 5.

[0059] Furthermore, the information processing device 1 may recursively and repeatedly execute the process of correcting the problem description 5 by re-prompting. The number of repetitions is not particularly limited and may be determined appropriately depending on the embodiment. In one example, the information processing device 1 may repeatedly execute the process of correcting the problem description 5 by re-prompting until the error message 615 is no longer output. In another example, the number of times the process of correcting the problem description 5 by re-prompting is executed may be specified in advance. If the error message 615 is still output after the information processing device 1 has repeatedly executed the correction process by re-prompting a predetermined number of times, the information processing device 1 may stop the correction process by re-prompting and output the execution results up to that point (e.g., the generated problem description 5, etc.).

[0060] Furthermore, the re-prompting method is not limited to the above example, and may be changed as appropriate depending on the embodiment. In another example, the information processing device 1 may correct the problem statement 5 by using a prompt corrected in accordance with the error message 615 as a re-prompt, together with or instead of the problem statement 5 and the error message 615. For example, when the error message 615 is output, the information processing device 1 may correct the output error by using the corrected model. In response to the error message 615, the information processing device 1 may appropriately modify the prompt (input data 200) provided to the inference module 3. The modified model may be configured as appropriate using at least one of a trained model and a rule-based model. The information processing device 1 may then provide the modified prompt to the inference module 3 again and execute the calculation process of the inference module 3 to generate a new problem description 5.

[0061] [Inference module] The inference module 3 is configured to execute an inference process to generate a problem description 5 from input data including observation data 20 and instruction information 21. As long as the inference module 3 is capable of executing such an inference process, the configuration of the inference module 3 is not particularly limited and may be determined appropriately depending on the embodiment. In one example, the inference module 3 may be configured by at least one of a rule-based model and a trained model (a trained machine learning model).

[0062] The rule-based model is configured to derive an inference result (in this embodiment, the generation result of problem statement 5) from a given input in accordance with rules. The rules may be set as appropriate. The machine learning model is configured to have one or more calculation parameters that can be adjusted by machine learning. The one or more calculation parameters are used to calculate the desired inference (in this embodiment, the generation of problem statement 5). The machine learning model may be configured, for example, by a neural network, a regression model, a decision tree model, a support vector machine, or other functional formulas (calculation models). The machine learning method may be selected as appropriate depending on the machine learning model employed (for example, backpropagation, etc.).

[0063] Machine learning involves adjusting (optimizing) values ​​of computational parameters using training samples. Typically, a trained model may be generated by supervised learning using multiple training datasets, each of which is composed of a combination of input samples (training samples) and output samples (teacher signals, labels). For example, the input samples may be samples of input data (such as observation data 20 and instruction information 21), and the output samples may be samples of output data (such as problem descriptions 5). In supervised learning, values ​​of computational parameters of a machine learning model may be adjusted so that, when an input sample is provided, the output obtained from the machine learning model matches the corresponding output sample. However, the method of generating a trained model is not limited to this example and may be appropriately changed depending on the embodiment. The training dataset is not limited to the above example and may be selected appropriately depending on the embodiment. For example, when acquiring in-context learning capabilities, data other than those described above may be used for the training dataset. Furthermore, the learning method is not limited to supervised learning; for example, other methods such as unsupervised learning (including self-supervised learning) and reinforcement learning may be used. The machine learning model may be trained online or offline. The machine learning model may be tuned as needed through transfer learning, re-learning, additional learning, etc. Additional learning is described in, for example, Reference 7 (“LoRA”, [online], [searched October 24, 2023], Internet<URL:https: / / github.com / microsoft / LoRA> )'s LoRA (Low-Rank Adaptation of Large Language Models), Reference Reference 8 (Neil Houlsby et al., “Parameter-Efficient Transfer Learning for NLP”, [online], [Retrieved October 24, 2023], Internet<URL:http: / / proceedings.mlr.press / v97 / houlsby19a / houlsby19a.pdf> ) Adapter, Reference 9 (Brian Lester et al., “The Power of Scale for Parameter-Efficient Prompt Tuning”, [online], [ Searched on October 24, 2023], Internet<URL:https: / / arxiv.org / abs / 2104.08691> ) and other machine learning techniques that add a module to an existing trained model and adjust the parameters of the additional module while leaving the parameters of the existing trained model unchanged. Additional learning can also be performed using machine learning techniques such as Prompt Tuning, which adds a module to an existing trained model and adjusts the parameters of the additional module while leaving the parameters of the existing trained model unchanged. Reference 10 (Kecheng Zheng et al., “Regularized Mask Tuning: Uncovering Hidden Knowledge in Pre-trained Vision-Language Models”, [online], [Retrieved October 24, 2023], Internet <URL:https: / / openaccess.thecvf.com / content / ICCV2023 / papers / Zheng_Regularized_Mask_Tuning_Uncovering_Hidden_Knowledge_in_Pre-Trained_Vision-Language_Models_ICCV_2023_paper.pdf), etc. When a trained model capable of in-context learning, such as the large-scale language model described above, is used for at least a part of inference module 3, the trained model may be specialized for a specific problem, domain, or other situation by performing this additional learning.

[0064] In one example, the inference module 3 may include a neural network. The structure of the neural network is not particularly limited and may be determined appropriately depending on the embodiment. The structure of the neural network may be specified, for example, by the number of layers from the input layer to the output layer, the type of each layer, the number of nodes (neurons) included in each layer, the connection relationships between the nodes in each layer, etc. In one example, the neural network may include any mechanism such as a recurrent structure, a self-attention mechanism, or an autoregressive model. Furthermore, the neural network may include any layer such as a fully connected layer, a convolutional layer, a pooling layer, a deconvolutional layer, an unpooling layer, a normalization layer, a dropout layer, or a long short-term memory (LSTM). The neural network may include any type of model such as a diffusion model, a transformer model, or a generative model. The connection weights between each node included in the neural network and the threshold value of each node are examples of calculation parameters.

[0065] As described above, in one example, the inference module 3 may include a trained model 39 for in-context learning. In this case, input data provided to the inference module 3 may include one or more input / output samples 243 for the trained model 39 as domain information 24 of the environment information 22. The input / output samples 243 may be prepared appropriately depending on the domain for which the problem description 5 is to be generated. In one example of this embodiment, by providing the input / output samples 243 to the trained model 39, in-context learning can be performed, and the trained model 39 can be adapted to the target domain. In other words, by preparing the input / output samples 243 for each domain, the inference module 3 can generate a general-purpose problem description 5 without replacing the trained model 39. Therefore, according to one example of this embodiment, it is expected that problem descriptions 5 can be generated for tasks in various domains.

[0066] In one example, when the inference module 3 includes a trained model 39 for in-context learning, a domain description 23 may be prepared for each type of robot device R. In addition, an output sample (a sample of the problem description 5) corresponding to the domain description 23 may be prepared for each type as domain information 24. In response to this, the information processing device 1 may accept a designation of the type. The designation method is not particularly limited and may be selected appropriately depending on the embodiment. Typically, the type may be designated manually, such as by an operator operating an input device, entering text, or entering voice. The designation of the type may be included in the instruction information 21. The information processing device 1 may provide the domain description 23 and the output sample (domain information 24) corresponding to the designated type as the environment information 22 to the inference module 3 (trained model 39). This makes it possible to generate a problem description 5 for generating an action plan for the robot device R having a skill of the designated type. That is, according to this example of the present embodiment, it is possible to expect an improvement in the versatility of the inference module 3 and an improvement in the accuracy of generating a problem description 5 that matches the type.

[0067] Furthermore, the form of input and output of the inference module 3 may be determined appropriately depending on the embodiment. As described above, in one example, the input data (observation data 20, instruction information 21, and environmental information 22) may be provided to the inference module 3 as is. In another example, at least a portion of the input data may be subjected to any pre-processing before being provided to the inference module 3. The pre-processing may include extraction of feature quantities, other analysis, inference, and other arithmetic processing. The inference module 3 may be provided with the input data after pre-processing. Furthermore, in one example, the inference module The output of the reasoning module 3 may be configured to directly indicate the inference result (problem statement 5). In another example, the output of the reasoning module 3 may be configured to indirectly indicate the inference result. In this case, the inference result may be obtained by performing any information processing (interpretation processing) on ​​the output of the reasoning module 3. The generation of the problem statement 5 may be performed in real time using data available at the present time (which may include past data) or may be performed retrospectively using data obtained in the past.

[0068] Furthermore, the inference module 3 may be configured as an integrated module, or may be configured as a set of multiple subelements. In one example, when the inference module 3 is configured as a set of multiple subelements, each subelement may be appropriately configured to generate a part (corresponding part) of the problem description 5. For example, one or more subelements may be prepared for each description (50, 51, 52) of the object, initial state, and goal state of the problem description 5. Each subelement may be configured as at least one of a machine learning model and a rule-based model.

[0069] (Example of inference module configuration) FIG. 4 schematically illustrates an example of the configuration of the inference module 3 according to this embodiment. In the example of FIG. 4, a scenario is assumed in which the problem description 5 further includes descriptions 50 of objects present in the environment. In one example, the inference module 3 may include an object estimator 31, an initial state estimator 33, and a goal estimator 35. The object estimator 31 is an example of one or more subelements corresponding to the object descriptions 50. The initial state estimator 33 is an example of one or more subelements corresponding to the initial state descriptions 51. The goal estimator 35 is an example of one or more subelements corresponding to the goal state descriptions 52.

[0070] The object estimator 31 may be configured to generate descriptions 50 of one or more objects present in the environment from the acquired observation data 20. Generating the object descriptions 50 may include detecting the objects. Generating the problem statement 5 using the inference module 3 may include generating, using the object estimator 31, descriptions 50 of objects present in the environment from the acquired observation data 20. The observation data 20 contains information about objects present in the environment. Therefore, according to one example of the present embodiment, the object descriptions 50 in the problem statement 5 can be appropriately generated.

[0071] The configuration of the object estimator 31 is not particularly limited and may be selected appropriately depending on the embodiment, as long as it is capable of generating an object description 50 from the observation data 20. The above description regarding the configuration of the inference module 3 may also be applied to the object estimator 31. The object estimator 31 may be configured using at least one of a rule-based model and a pre-trained model.

[0072] The object estimator 31 may use a model capable of inferring object descriptions 50 only in a specific domain. However, in order to be commonly usable for tasks in various domains, it is desirable to use a model capable of inferring object descriptions 50 generically as the object estimator 31, rather than a model capable of inferring object descriptions 50 only in such a specific domain. In one example, the object estimator 31 may be equipped with a trained model for in-context learning. By providing a trained model capable of in-context learning, the versatility of the object estimator 31 can be increased. In other words, the object estimator 31 can generically infer descriptions 50 of objects present in the environment, which can be expected to enable the generation of object descriptions 50 for tasks in various domains.

[0073] The input / output form of the object estimator 31 may be determined appropriately depending on the embodiment. The object estimator 31 may further receive input of any data other than the observation data 20. The input data may be provided to the object estimator 31 as is, or may be provided to the object estimator 31 after preprocessing has been applied. The object estimator 31 may update any data other than the object description 50. You can also output to

[0074] In one example, the information processing device 1 may further acquire environmental information 22 related to the environment. The object estimator 31 may further use the acquired environmental information 22 to generate the object description 50. That is, generating the object description 50 from the acquired observation data 20 may be configured by generating the object description 50 from the acquired observation data 20 and the environmental information 22. According to one example of the present embodiment, by further using the environmental information 22, it is possible to expect improvement in the accuracy of generating the object description 50 in the problem description 5.

[0075] The environmental information 22 provided to the object estimator 31 may be selected appropriately depending on the embodiment. The environmental information 22 may include at least one of the domain description 23 and the domain information 24. For example, the environmental information 22 may include at least a portion of the domain description 23. Furthermore, for example, the environmental information 22 may include attribute information 241 of objects present in the environment as the domain information 24. This allows the attribute information 241 to more accurately identify objects present in the environment, thereby improving the accuracy of generating the object description 50 in the problem description 5. Furthermore, for example, when the object estimator 31 includes a trained model for in-context learning, the environmental information 22 may include one or more input / output samples 243 as the domain information 24. The input samples may be composed of samples of the observation data 20. When environmental information 22 other than the input / output samples 243 is provided to the object estimator 31, the input samples may include samples of the environmental information 22. The output samples may be composed of correct samples of the object description 50 corresponding to the input samples.

[0076] The domain description 23 and the domain information 24 (attribute information 241, input / output samples 243) may be prepared as appropriate depending on the embodiment. In one example, the domain description 23 and the domain information 24 may be prepared in advance. The domain description 23 and the domain information 24 may be stored in memory resources within the information processing device 1, or may be provided to the information processing device 1 from an external device when used. In another example, at least a portion of the domain description 23 and the domain information 24 may be generated as appropriate when used. For example, reference information including a list of objects and attribute information may be prepared in advance. The information processing device 1 may extract object candidates present in the environment from at least one of the instruction information 21 (e.g., linguistic information) and the domain description 23 (e.g., a description defining the type of target object). The information processing device 1 may compare the list of extracted object candidates with the list of reference information and extract attribute information of objects matching the object candidates from the reference information, thereby generating attribute information 241 to be provided to the object estimator 31. The process of generating this attribute information 241 may be executed as a pre-processing or may be executed as a process within the inference module 3.

[0077] The initial state estimator 33 may be configured to generate an initial state description 51 of one or more objects present in the environment from the acquired observation data 20. Generating the problem statement 5 using the inference module 3 may include generating the initial state description 51 of the objects present in the environment using the initial state estimator 33. The observation data 20 represents the initial states of the objects present in the environment. Therefore, according to one example of the present embodiment, the initial state description 51 in the problem statement 5 can be appropriately generated.

[0078] As long as the description 51 of the initial state can be generated from the observation data 20, the configuration of the initial state estimator 33 is not particularly limited and may be appropriately selected depending on the embodiment. As with the object estimator 31, the above description regarding the configuration of the inference module 3 may also be applied to the initial state estimator 33. The initial state estimator 33 may be configured using at least one of a rule-based model and a trained model.

[0079] The initial state estimator 33 has a model that can infer the initial state description 51 only in a specific domain. However, in order to be commonly usable for tasks in various domains, it is desirable to use a model capable of inferring the initial state description 51 generically as the initial state estimator 33, rather than a model capable of inferring the initial state description 51 only in a specific domain. In one example, the initial state estimator 33 may include a pre-trained model for in-context learning. By including a pre-trained model capable of in-context learning, the versatility of the initial state estimator 33 can be increased. In other words, the initial state description 51 of an object existing in the environment can be inferred generically, which can be expected to enable the generation of initial state descriptions 51 for tasks in various domains.

[0080] The input / output form of the initial state estimator 33 may be determined appropriately depending on the embodiment, similar to the object estimator 31. The initial state estimator 33 may further receive input of any data other than the observation data 20. The input data may be provided to the initial state estimator 33 as is, or may be provided to the initial state estimator 33 after preprocessing has been applied. The input data may be provided to the object estimator 31 first, and the calculation results (including intermediate calculation results) of the object estimator 31 on the input data may be provided to the initial state estimator 33. For example, providing the observation data 20 to the initial state estimator 33 may include providing the observation data 20 as is to the initial state estimator 33, providing the preprocessing results of the observation data 20 to the initial state estimator 33 to the initial state estimator 33, and providing the calculation results of the object estimator 31 on the observation data 20 to the initial state estimator 33. The preprocessing results and the calculation results of the object estimator 31 may be, for example, the detection results of an object in the observation data 20. However, the order in which the input data is provided is not limited to this example. The input data may be provided to the initial state estimator 33 first, and the result of the calculation by the initial state estimator 33 on the input data may be provided to the object estimator 31. For example, providing the observation data 20 to the object estimator 31 may include providing the observation data 20 directly to the object estimator 31, providing the result of preprocessing the observation data 20 to the object estimator 31, and providing the result of the calculation by the initial state estimator 33 on the observation data 20 to the object estimator 31. In addition, the initial state estimator 33 may further output any data other than the description 51 of the initial state of the object.

[0081] In one example, similar to the object estimator 31, the initial state estimator 33 may further use the environment information 22 to generate the initial state description 51. That is, generating the initial state description 51 from the observation data 20 may be configured by generating the initial state description 51 from the observation data 20 and the environment information 22. The environment information 22 provided to the initial state estimator 33 may be selected appropriately depending on the embodiment. The environment information 22 may include at least one of the domain description 23 and the domain information 24. For example, the environment information 22 may include at least a portion of the domain description 23. Furthermore, for example, the environment information 22 may include attribute information 241 as the domain information 24. Furthermore, for example, when the initial state estimator 33 includes a trained model for in-context learning, the environment information 22 may include one or more input / output samples 243 as the domain information 24. The input sample may be configured by a sample of the observation data 20 (including a preprocessing result and a calculation result of the object estimator 31). When environmental information 22 other than the input / output samples 243 is provided to the initial state estimator 33, the input samples may include samples of the environmental information 22. The output samples may be composed of correct samples of the initial state description 51 corresponding to the input samples. The environmental information 22 (domain description 23, domain information 24) may be prepared as appropriate depending on the embodiment. By further using the environmental information 22, it is possible to expect an improvement in the accuracy of generating the initial state description 51 in the problem description 5.

[0082] In one example, the calculation process of the object estimator 31 may be completed before the calculation process of the initial state estimator 33. In this case, the object description 50 generated by the object estimator 31 may be provided to the initial state estimator 33 as input data for generating the initial state description 51. That is, the initial state estimator 33 may use the object description 50 generated by the object estimator 31 as input data for generating the initial state description 51. 0 may further be used to generate the initial state description 51. When the initial state estimator 33 comprises a trained model for in-context learning and the environmental information 22 provided to the initial state estimator 33 includes input / output samples 243, the input samples may include samples of the object description 50. However, the order of operations within the inference module 3 is not limited to this example. The operation process of the initial state estimator 33 may be executed at least partially in parallel with the object estimator 31, or may be executed to completion before the object estimator 31. When the operation process of the initial state estimator 33 is executed to completion before the object estimator 31, the initial state description 51 generated by the initial state estimator 33 may be provided to the object estimator 31. When the object estimator 31 comprises a trained model for in-context learning and the environmental information 22 provided to the object estimator 31 includes input / output samples 243, the input samples may include samples of the initial state description 51.

[0083] The goal estimator 35 may be configured to generate goal state descriptions 52 of one or more objects in the environment from the acquired instruction information 21. Generating the problem description 5 using the reasoning module 3 may include generating goal state descriptions 52 of objects in the environment using the goal estimator 35. The instruction information 21 contains information related to the goal of the task. Therefore, according to one example of the present embodiment, the goal state descriptions 52 in the problem description 5 can be appropriately generated.

[0084] The configuration of the goal estimator 35 is not particularly limited and may be selected appropriately depending on the embodiment, as long as it is capable of generating the description 52 of the goal state from the instruction information 21. As with the object estimator 31 and the like, the above description regarding the configuration of the inference module 3 may also be applied to the goal estimator 35. The goal estimator 35 may be configured using at least one of a rule-based model and a trained model.

[0085] The goal estimator 35 may use a model capable of inferring goal state descriptions 52 only in a specific domain. However, in order to be commonly applicable to tasks in various domains, it is desirable to use a model capable of inferring goal state descriptions 52 generically, rather than a model capable of inferring goal state descriptions 52 only in a specific domain, as the goal estimator 35. In one example, the goal estimator 35 may include a trained model for in-context learning. Having a trained model capable of in-context learning can increase the versatility of the goal estimator 35. In other words, the goal estimator 35 can generically infer goal state descriptions 52 of objects in the environment, which can be expected to enable the generation of goal state descriptions 52 for tasks in various domains.

[0086] The input / output form of the target estimator 35 may be determined appropriately depending on the embodiment, similarly to the object estimator 31, etc. The target estimator 35 may further receive input of any data other than the instruction information 21. The input data may be provided to the target estimator 35 as is, or may be provided to the target estimator 35 after preprocessing has been applied. Similar to the relationship between the object estimator 31 and the initial state estimator 33, the calculation results (including intermediate calculation results) of at least one of the object estimator 31 and the initial state estimator 33 may be provided to the target estimator 35. Alternatively, the calculation results (including intermediate calculation results) of the target estimator 35 may be provided to at least one of the object estimator 31 and the initial state estimator 33. The target estimator 35 may further output any data other than the description 52 of the target state of the object.

[0087] In one example, similar to the object estimator 31, the target estimator 35 may further use the environment information 22 to generate the target state description 52. That is, generating the target state description 52 from the instruction information 21 may be configured by generating the target state description 52 from the instruction information 21 and the environment information 22. The environment information 22 to be provided to the target estimator 35 may be appropriately selected depending on the embodiment. The environment information 22 may be generated by generating the target state description 52 from the instruction information 21 and the environment information 22. The environmental information 22 may include at least one of the above. For example, the environmental information 22 may include at least a portion of the domain description 23. For example, the environmental information 22 may include attribute information 241 as the domain information 24. For example, if the goal estimator 35 has a trained model for in-context learning, the environmental information 22 may include one or more input / output samples 243 as the domain information 24. The input samples may be composed of samples of the instruction information 21. If environmental information 22 other than the input / output samples 243 is provided to the goal estimator 35, the input samples may include samples of the environmental information 22. The output samples may be composed of correct samples of the goal state description 52 corresponding to the input samples. The environmental information 22 (domain description 23, domain information 24) may be prepared as appropriate depending on the embodiment. By further using the environmental information 22, it is expected that the accuracy of generating the goal state description 52 in the problem description 5 can be improved. The environmental information 22 provided to the object estimator 31, the initial state estimator 33, and the target estimator 35 may or may not overlap at least partially.

[0088] In one example, the calculation process of the object estimator 31 may be completed before the target estimator 35. In this case, the object description 50 generated by the object estimator 31 may be provided to the target estimator 35 as input data for generating the goal state description 52. Similarly, the calculation process of the initial state estimator 33 may be completed before the target estimator 35. In this case, the initial state description 51 generated by the initial state estimator 33 may be provided to the target estimator 35 as input data for generating the goal state description 52. That is, the target estimator 35 may further use at least one of the object description 50 and the initial state description 51 generated by the object estimator 31 and the initial state estimator 33 to generate the goal state description 52. When the goal estimator 35 includes a trained model for in-context learning and the environment information 22 provided to the target estimator 35 includes input / output samples 243, the input samples may include at least one of the object description 50 and the initial state description 51. However, the order of operations within the inference module 3 is not limited to this example. The calculation process of the target estimator 35 may be executed at least partially in parallel with the object estimator 31, or may be executed to completion before the object estimator 31. If the calculation process of the target estimator 35 is executed to completion before the object estimator 31, the description of the goal state 52 generated by the target estimator 35 may be provided to the object estimator 31. Similarly, the calculation process of the target estimator 35 may be executed at least partially in parallel with the initial state estimator 33, or may be executed to completion before the initial state estimator 33. If the calculation process of the target estimator 35 is executed to completion before the initial state estimator 33, the description of the goal state 52 generated by the target estimator 35 may be provided to the initial state estimator 33. If at least one of the object estimator 31 and the initial state estimator 33 has a trained model for in-context learning, and the provided environment information 22 includes input / output samples 243, the input samples may include samples of the description of the goal state 52.

[0089] The configuration of the inference module 3 is not limited to this example and may be modified as appropriate depending on the embodiment. In another example, at least one combination of the object estimator 31, the initial state estimator 33, and the goal estimator 35 may be integrated. At least one of the object estimator 31, the initial state estimator 33, and the goal estimator 35 may be omitted as appropriate. Furthermore, in the example of FIG. 4 , the inference module 3 includes a trained model 39 for in-context learning, which may be configured by at least one of the object estimator 31, the initial state estimator 33, and the goal estimator 35 being equipped with a trained model for in-context learning. When the inference module 3 includes the trained model 39 for in-context learning and the environmental information 22 provided to the inference module 3 includes input / output samples 243, the information processing device 1 may use the input / output samples 243 as few-shot prompting. For example, the information processing device 1 may provide only the input / output samples 243 to the trained model 39 before generating the problem statement 5, or may provide the input / output samples 243 together with other input data when generating the problem statement 5. The information processing device 1 may provide the trained model 39 with the problem description 5. Then, the information processing device 1 may execute calculation processing of the trained model 39. In this way, the information processing device 1 may execute in-context learning in the trained model 39 and adapt the trained model 39 to the domain in which the problem description 5 is generated.

[0090] (Example of an object estimator) 5A schematically illustrates an example of an object estimator 31 according to this embodiment. In one example, the object estimator 31 may include a trained model 311 and a rule-based model 313. When observation data 20 and environmental information 22 are input, the trained model 311 may be configured to detect an object appearing in the observation data 20 using the environmental information 22 as a clue, and output an object detection result 201. The trained model 311 may be a trained model capable of in-context learning.

[0091] For example, the sensor S may be a camera, and the observation data 20 may be image data. The environmental information 22 may be expressed in linguistic terms such as "white bowl" and "round cutting board." The attribute information 241 may be a list of attribute information indicating the attributes of the object. The trained model 311 is based on the method described in Reference 11 (Shilong Liu et al., "Grounding DINO: Marrying DINO with grounded pre-training for open-set object detection", [online], [Retrieved October 24, 2023], internet<URL:https: / / arxiv.org / abs / 2303.05499> ), Reference 12 (Alexander Kirillov et al., “Segment Anything”, [online], [Retrieved October 24, 2020], Internet<URL:https: / / arxiv.org / abs / 2304.02643> ) or the like. The detection result 201 may include at least one of a result of detecting the area in which the object is captured as a bounding box and a result of identifying the object. The rule-based model 313 may be configured to generate an object description 50 from the object detection result 201 by the trained model 311 in accordance with rules. The rules may be set as appropriate.

[0092] Note that the configuration of FIG. 5A is merely one example of the object estimator 31. The configuration of the object estimator 31 is not limited to the example of FIG. 5A and may be modified as appropriate depending on the embodiment. In another example, the trained model 311 may be provided with environmental information 22 other than the attribute information 241. The observation data 20 may further include data in a format other than image data, or may be composed solely of data in a format other than image data. The trained model 311 and the rule-based model 313 may be configured integrally. The rule-based model 313 may be replaced by the trained model. The trained model 311 may be configured to generate an object description 50. Accordingly, the rule-based model 313 may be omitted.

[0093] (Example of an initial state estimator) FIG. 5B schematically illustrates an example of an initial state estimator 33 according to this embodiment. In one example, the initial state estimator 33 may include a detector 331 and a trained model 333. The information processing device 1 may extract partial data 205 corresponding to each object from the observation data 20 in accordance with the object detection results 201. The detector 331 may be configured to generate a caption 206 for each object from the extracted partial data 205 for each object. The caption 206 for each object may be configured to include information about the object appearing in the partial data 205. The trained model 333 may be configured to generate an initial state description 51 for the object from the object detection results 201 and the caption 206. The trained model 333's use of the object detection results 201 is an example of the initial state estimator 33 using the calculation results of the object estimator 31 as the observation data 20. The detector 331 and the trained model 333 may each be a trained model capable of in-context learning.

[0094] For example, the part data 205 is a part extracted by a bounding box of each object. The detector 331 may be image data. The detector 331 may be implemented using the method described in Reference 13 (Junnan Li et al., "BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large "Language Models", [online], [Retrieved October 24, 2023], Internet<URL:https: / / arxiv.org / abs / 2301.12597> The detector 331 is provided with a question sentence along with the partial data 205. In one example, the question may be composed of predetermined text data (such as "Q: what does this object describe? A: .") that asks about an object shown in a partial image. Model 333 is based on the reference 14 (OpenAI, “GPT-4 Technical Report”, [online], [Rei]). Searched on October 24, 1985], Internet<URL:https: / / arxiv.org / pdf / 2303.08774.pdf> ) or the like. Accordingly, the caption 206 may be text data describing the partial data 205 (partial image data). The object detection result 201 may be appropriately converted into text data 202, and the obtained text data 202 may be provided to the trained model 333 as the detection result 201 (observation data 20). In one example, the text data 202 may be composed of text indicating the name of the detected object (identification result) and the coordinate range of the object's bounding box. The trained model 333 may further be provided with an object description 50 generated by the object estimator 31. In addition, input / output samples 243 may further be provided to the trained model 333 as environment information 22 (domain information 24). In one example, the input sample may be composed of samples of the text data 202, the caption 206, and the object description 50, and the output sample may be composed of a correct answer sample of the corresponding initial state description 51. The input / output samples 243 may be used for few-shot prompting of the trained model 333.

[0095] Note that the configuration of FIG. 5B is merely one example of the initial state estimator 33. The configuration of the initial state estimator 33 is not limited to the example of FIG. 5B and may be modified as appropriate depending on the embodiment. In another example, the trained model 333 may be provided with environment information 22 other than the input / output samples 243. In another example, the trained model 333 may include an input section that accepts input of data of modalities other than text, such as a large-scale visual language model. As a result, the trained model 333 may be configured to be able to accept input of data of other modalities. Accordingly, the observation data 20 or the detection result 201 may be provided to the trained model 333 as is. The caption 206 may be composed of data in a format other than text data. Furthermore, the partial data 205 (observation data 20) may further include data in a format other than image data, or may be composed solely of data in a format other than image data. The data format of the detection result 201 is not particularly limited and may be selected as appropriate depending on the embodiment. The detector 331 may be omitted.

[0096] (Example of a target estimator) 5C is a schematic diagram illustrating an example of a target estimator 35 according to the present embodiment. In one example, the target estimator 35 may include a trained model 351. The trained model 351 may be configured to generate a target state description 52 of the object from the instruction information 21. The trained model 351 may be capable of in-context learning.

[0097] For example, the trained model 351 may be a large-scale language model (LLM) proposed in the above-mentioned reference 14, etc. Accordingly, the instruction information 21 may be a text that indicates the target in natural language. The trained model 351 may be further provided with at least one of an object description 50 generated by the object estimator 31 and an initial state description 51 generated by the initial state estimator 33. Input / output samples 243 may also be provided to the trained model 351 as the environment information 22 (domain information 24). In one example, the input samples may be composed of samples of the instruction information 21, the object description 50, and the initial state description 51, and the output samples may be composed of ground truth samples of the corresponding target state description 52. The input / output samples 243 may be used for few-shot prompting of the trained model 351. The trained model 351 may be prepared in common with the trained model 333 of the initial state estimator 33, or may be prepared separately.

[0098] Note that the configuration of FIG. 5C is merely one example of the target estimator 35. The configuration of the target estimator 35 is not limited to the example of FIG. 5C and may be modified as appropriate depending on the embodiment. In another example, the trained model 351 may be provided with environmental information 22 other than the input / output sample 243. In another example, the trained model 351 may be configured to be able to accept input of data of other modalities, similar to the trained model 333. Accordingly, the trained model 351 may be provided with data in a format other than text data. The instruction information 21 may further include data in a format other than text data, or may be composed solely of data in a format other than text data.

[0099] [Robot device] The robot device R is not particularly limited and may be appropriately selected depending on the embodiment. The robot device R may be, for example, an industrial robot used in a production line, an autonomous robot configured to operate autonomously, or a mobile body configured to move. The industrial robot may be, for example, a vertical articulated robot, a horizontal articulated robot (SCARA robot), a parallel link robot, or an orthogonal robot. The autonomous robot may be, for example, a humanoid robot, a guide robot, an agricultural robot, a care robot, a security robot, or a transport robot. The content of the autonomous processing may be appropriately selected depending on the embodiment. The mobile body may include, for example, a cleaning robot, the above-mentioned autonomous robot configured to move (including a mobile robot), a vehicle configured to be capable of self-driving, or an air vehicle capable of self-flying (such as a drone). The robot device R may exist in real space or virtual space. The action plan generated using the problem description 5 may be performed for controlling the robot device R in real space, or for simulating the robot device R in virtual space.

[0100] The generation of the problem description 5 according to this embodiment may be applied to any task that can be accomplished by the robot device R. The task may be, for example, work, movement, etc. The work may be, for example, assembly, cooking, cleaning, chemical experiments, etc. If the robot device R is equipped with a communication device, the sensing data (observation data 20) may include data obtained through communication. For example, in a situation where a behavior plan (movement plan) for a moving object is generated, the sensing data may include data obtained through road-to-vehicle communication, vehicle-to-vehicle communication, etc.

[0101] The robot device R may also be used in a situation where it coexists with humans. In this situation, the instruction information 21 may be provided by a human (operator). The generated problem description 5 may be output in a manner perceptible to humans (for example, output on a display). The information processing device 1 may accept human intervention (manual correction) of the generated problem description 5. In one example, the human intervention may correct the driving characteristics of the robot device R, such as the amount of movement, angle, posture, and swing. The problem description 5 may be generated through interactive interaction between the information processing device 1 and an operator by using a trained model capable of in-context learning in the inference module 3 (at least one of the object estimator 31, the initial state estimator 33, and the target estimator 35). Human input may be provided directly to the information processing device 1 or indirectly to the information processing device 1 via the robot device R. Human input may be provided by any method, such as operating an input device (including text input), recording, or capturing an image. The planner may be configured to generate an action plan from the problem statement 5 taking into account the presence of humans.

[0102] [Problem description output] The output format of the problem description 5 is not particularly limited and may be selected appropriately depending on the embodiment. In a simple example, the generated problem description 5 may be output as is. Outputting the problem description 5 may include outputting the problem description 5 to at least one of a memory resource of the information processing device 1, an output device connected to the information processing device 1, and an external computer. The output method may be appropriately selected depending on the embodiment. For example, the problem description 5 may be output as text, an image, audio, or a combination of these.

[0103] In another example, outputting the problem description 5 may include at least a part of providing the problem description 5 to a planner and generating an action plan by the planner. If generating an action plan by the planner is included, outputting the problem description 5 may further include controlling the operation of the robotic device R (i.e., driving the robotic device R) in accordance with the generated action plan.

[0104] The generation of an action plan by the planner may be executed by at least one of the information processing device 1 and one or more external computers. When the planner includes a symbolic planner 61 and a motion planner 65, the information processing device 1 may execute the calculation processing of one of the symbolic planner 61 and the motion planner 65, and the external computer may execute the calculation processing of the other. The calculation processing of both the symbolic planner 61 and the motion planner 65 may be executed by either the information processing device 1 or one or more external computers. When the calculation processing of both the symbolic planner 61 and the motion planner 65 is executed by one or more external computers, the external computer that executes the calculation processing of the symbolic planner 61 and the external computer that executes the calculation processing of the motion planner 65 may be the same or different.

[0105] Furthermore, the control of the robot device R according to the generated behavior plan may be executed by at least one of the information processing device 1 and an external computer. When at least a part of the generation of the behavior plan by the planner and the control of the robot device R are executed by an external computer, the external computer that executes the generation of the behavior plan by the planner and the external computer that controls the robot device R may be the same or different. One or more external computers may execute at least a part of the generation of the behavior plan, and the information processing device 1 may execute the control of the robot device R according to the generated behavior plan. The information processing device 1 and the external computer may exchange data as appropriate by any method such as data communication.

[0106] §2 Configuration example [Hardware configuration] Fig. 6 schematically illustrates an example of the hardware configuration of the information processing device 1 according to this embodiment. In the example of Fig. 6, the information processing device 1 according to this embodiment is a computer to which a control unit 11, a storage unit 12, an external interface 13, an input device 14, an output device 15, and a drive 16 are electrically connected.

[0107] The control unit 11 includes a CPU (Central Processing Unit) which is a hardware processor, The memory 12 includes RAM (Random Access Memory), ROM (Read Only Memory), etc., and is configured to execute information processing based on programs and various data. The control unit 11 (CPU) is an example of a processor resource. The storage unit 12 may be configured, for example, with a hard disk drive, a solid state drive, etc. The storage unit 12, RAM, and ROM are examples of memory resources. In this embodiment, the storage unit 12 stores various information such as the program 81 and module data 300.

[0108] The program 81 is a program for causing the information processing device 1 to execute information processing (see FIG. 8 described later) related to the generation of the problem statement 5. The program 81 includes a series of instructions for the information processing. The module data 300 indicates information related to the inference module 3. As long as the inference module 3 can be reproduced when generating the module data 300, the configuration of the module data 300 is not particularly limited and may be determined appropriately depending on the embodiment. For example, if the module data 300 includes a machine learning model, the module data 300 may include information indicating values ​​of calculation parameters adjusted by machine learning. In some cases, the module data 300 may further include information indicating the configuration of the machine learning model (e.g., the structure of a neural network). If the module data 300 includes a rule-based model, the module data 300 may include information indicating rules. If the inference module 3 has the configuration shown in FIG. 4, information regarding the object estimator 31, the initial state estimator 33, and the target estimator 35 may be stored as separate data (files), or at least any combination of information regarding the object estimator 31, the initial state estimator 33, and the target estimator 35 may be stored as the same data. The module data 300 may be incorporated into the program 81.

[0109] The external interface 13 is configured to connect to an external device via a wired or wireless connection. The external interface 13 may be, for example, a USB (Universal Serial Bus) port, a communication port, a dedicated port, or the like. When the external interface 13 includes a communication port, the communication standard of the communication port may be selected arbitrarily. In this embodiment, the information processing device 1 may be connected to an external device (for example, a sensor S, a robot device R, an external computer, or the like) via the external interface 13.

[0110] The input device 14 is a device for inputting, for example, a mouse, a keyboard, etc. The output device 15 is a device for outputting, for example, a display, a speaker, etc. An operator can operate the information processing device 1 by using the input device 14 and the output device 15. The input device 14 and the output device 15 may be connected via an external interface 13. The input device 14 and the output device 15 may be integrally configured, for example, by a touch panel display, etc.

[0111] The drive 16 is a device for reading various information such as programs stored in a storage medium 91. At least one of the program 81 and the module data 300 may be stored in the storage medium 91 instead of or together with the storage unit 12. The storage medium 91 is configured to store various information (such as stored programs) by electrical, magnetic, optical, mechanical, or chemical action so that a machine such as a computer can read the information. The information processing device 1 may acquire at least one of the program 81 and the module data 300 from the storage medium 91. The storage medium 91 may be a disk-type storage medium such as a CD or DVD, or may be a non-disk-type storage medium such as a semiconductor memory (e.g., a flash memory). The type of the drive 16 may be selected appropriately depending on the type of the storage medium 91. The drive 16 may be connected via an external interface 13.

[0112] It should be noted that, with regard to the specific hardware configuration of the information processing device 1, components may be omitted, replaced, or added as appropriate depending on the embodiment. For example, the control unit 11 may include multiple hardware processors. The hardware processors may be a microprocessor, a field-programmable gate array (FPGA), a digital signal processor (DSP), a GP The storage unit 12 may be configured by a U (Graphics Processing Unit), an ASIC (Application Specific Integrated Circuit), etc. At least one of the external interface 13, the input device 14, the output device 15, and the drive 16 may be omitted. The information processing device 1 may be composed of a plurality of computers. In this case, the hardware configuration of each computer may or may not be the same. Furthermore, the information processing device 1 may be an information processing device designed specifically for the service to be provided, as well as a general-purpose server device, a general-purpose PC (Personal Computer), a tablet PC, a terminal device, etc. When an external computer is used, In this case, the hardware configuration of the external computer may be the same as that of the information processing device 1. The external computer may be an information processing device designed specifically for the service to be provided, or may be a general-purpose server device, a general-purpose PC, a tablet PC, a terminal device, or the like.

[0113] [Software configuration] 7 schematically illustrates an example of the software configuration of the information processing device 1 according to this embodiment. The control unit 11 of the information processing device 1 loads a program 81 stored in the storage unit 12 into RAM, and executes instructions included in the program 81 using the CPU. As a result, the information processing device 1 operates as a computer including an acquisition unit 111, a generation unit 112, and an output unit 113 as software modules. That is, in this embodiment, each software module of the information processing device 1 is realized by the control unit 11 (CPU).

[0114] The acquisition unit 111 is configured to acquire observation data 20 of the environment in which the robot device R operates, and instruction information 21 related to a goal of a task to be given to the robot device R. In one example, the acquisition unit 111 may be configured to further acquire environmental information 22 related to the environment. Since the information processing device 1 holds module data 300, the generation unit 112 includes an inference module 3. The generation unit 112 is configured to generate a task problem description 5 from the acquired observation data 20 and instruction information 21, using the inference module 3. In one example, generating the task problem description 5 from the observation data 20 and instruction information 21 may be configured by generating the task problem description 5 from the observation data 20, the instruction information 21, and the environment information 22. The output unit 113 is configured to output the generated problem description 5.

[0115] In this embodiment, an example is described in which each software module of the information processing device 1 is implemented by a general-purpose CPU. However, some or all of the software modules may be implemented by one or more dedicated processors or chipsets. Each module may be implemented as a hardware module. Regarding the software configuration of the information processing device 1, modules may be omitted, replaced, or added as appropriate depending on the embodiment.

[0116] §3 Example of operation 8 is a flowchart showing an example of a processing procedure of the information processing device 1 according to this embodiment. The following processing procedure is an example of an information processing method executed by a computer. However, the following processing procedure of the information processing device 1 is merely an example, and each step may be changed as much as possible. Furthermore, steps in the following processing procedure may be omitted, replaced, or added as appropriate depending on the embodiment.

[0117] (Step S101) In step S101, the control unit 11 operates as the acquisition unit 111 and acquires observation data 20 and instruction information 21. In one example, the observation data 20 may be composed of sensing data from one or more sensors S, and the instruction information 21 may be composed of linguistic information that instructs a target in natural language. In another example, the control unit 11 may further acquire environment information 22. The environment information 22 may include at least one of a domain description 23 and domain information 24. The domain information 24 may include attribute information 241. If the inference module 3 includes a trained model 39 for in-context learning, the domain information 24 may include input / output samples 243. After acquiring the observation data 20 and the instruction information 21, the control unit 11 proceeds to the next step S102.

[0118] (Step S102) In step S102, the control unit 11 operates as the generation unit 112 and generates a problem description 5 of the task from the acquired observation data 20 and instruction information 21 using the inference module 3. In one example, when environmental information 22 is further acquired, the control unit 11 uses the inference module 3 to In this case, the problem description 5 may be generated from the observation data 20, the instruction information 21, and the environment information 22.

[0119] The generated problem description 5 includes descriptions (51, 52) of initial and goal states of one or more objects present in the environment. In one example, the generated problem description 5 may follow a predetermined format. The generated problem description 5 may also include descriptions 50 of objects present in the environment.

[0120] Also, in one example, generating the problem statement 5 using the reasoning module 3 may include at least one of generating an object statement 50 using the object estimator 31, generating an initial state statement 51 using the initial state estimator 33, and generating a goal state statement 52 using the goal estimator 35. When the reasoning module 3 includes a trained model 39 for in-context learning and the domain information 24 includes input / output samples 243, the control unit 11 may use the input / output samples 243 as few-shot prompting to adapt the trained model 39 to the domain for generating the problem statement 5. After generating the problem statement 5, the control unit 11 proceeds to the next step S103.

[0121] (Step S103) In step S103, the control unit 11 operates as the output unit 113 and outputs the generated problem description 5. In one example, the control unit 11 may output the problem description 5 to at least one of the memory resources of the information processing device 1, the output device 15, and an external computer. In another example, the control unit 11 may provide the problem description 5 to a planner and execute at least a part of the process of generating an action plan by the planner as the process of step S103. When the information processing device 1 executes the calculation process of the planner, the control unit 11 may appropriately acquire information about the planner. For example, the information about the planner may be stored in at least one of the memory resources (e.g., RAM, storage unit 12) of the information processing device 1 and the external computer, and the control unit 11 may acquire the information about the planner from either of them. When the process of generating an action plan by the planner is executed by the external computer, the control unit 11 may provide the generated problem description 5 to the external computer as the process of step S103. In another example, when a behavior plan is generated by the information processing device 1 or an external computer, the control unit 11 may control the operation of the robot device R in accordance with the generated behavior plan as the processing of step S103. Controlling the operation of the robot device R may include directly controlling the robot device R and indirectly controlling the robot device R by giving instructions to a controller of the robot device R. Note that at least one of the generation of the behavior plan by the planner and the control of the robot device R may be executed separately from the processing of step S103. The control of the robot device R may be omitted. When the control of the robot device R is omitted, the generation of the behavior plan by the planner may also be omitted. When the output of the problem description 5 is completed, the control unit 11 ends the processing procedure of the information processing device 1 according to this operation example.

[0122] In one example, when the planner outputs an error message 615 during the process of generating an action plan using the planner, the control unit 11 may execute a process for correcting the problem statement 5. For example, as shown in FIG. 3 , when the inference module 3 includes a trained model 39 for in-context learning, the control unit 11 may acquire the output error message 615. Then, the control unit 11 may provide the acquired error message 615 and the problem statement 5 to the inference module 3, and execute the calculation process of the inference module 3 again. During this re-prompt, the control unit 11 may further provide at least a portion of the input data 200 to the inference module 3. As a result, the control unit 11 may generate a new corrected problem statement 5. The control unit 11 may recursively execute the process of correcting the problem statement 5 using this re-prompt.

[0123] [Features] In this embodiment, the problem description 5 generated in step S102 includes descriptions (51, 52) of the initial state and the goal state of each of one or more objects present in the environment, so that the planner generates an action plan that reaches the goal state from the initial state of the task. The initial state description 51 describes the initial state of each of one or more objects present in the environment. On the other hand, the goal state description 52 describes the goal state of each of one or more objects that will be reached by completing the task. In other words, each description (51, 52) indicates the state before and after the task is performed and is human-interpretable. Therefore, the problem description 5 has high interpretability. Therefore, according to this embodiment, a highly interpretable output can be obtained to obtain an action sequence (control command) to be given to the robot device R.

[0124] §4 Variations Although the embodiments of the present invention have been described above in detail, the above description is merely an example of the present invention in every respect. Various improvements or modifications may be made to the above embodiments as appropriate. For example, the following modifications are possible. Note that, in the following, the same reference numerals are used for components similar to those in the above embodiments, and descriptions of the same points as those in the above embodiments are omitted as appropriate. The following modifications can be combined as appropriate.

[0125] <4.1> In the above embodiment, the sensor S may be attached to or include a driving device, thereby enabling the observation direction to be changed. For example, if the sensor S includes a camera, the camera may be attached to an electric pan head, thereby enabling the shooting direction to be changed. The electric pan head is an example of a driving device. The information processing device 1 may be configured to be able to directly or indirectly drive the driving device. In this way, the information processing device 1 may appropriately change the observation direction of the sensor S.

[0126] In addition, the environment information 22 may include a designation list of objects to be observed. In step S101, the control unit 11 of the information processing device 1 may acquire the environment information 22 including the designation list. The attribute information 241 may also serve as the designation list of objects. In step S102, the control unit 11 may determine whether all objects included in the designation list have been detected from the observation data 20. In the example of FIG. 4, the control unit 11 may determine whether all objects included in the designation list have been detected from the observation data 20 in the calculation processing of the object estimator 31 or the initial state estimator 33. For example, when an object detector such as the trained model 311 of FIG. 5A is provided, the control unit 11 may determine whether all objects included in the designation list have been detected based on the object detection results of the detector for the observation data 20. In another example, the control unit 11 may determine whether all objects included in the designation list have been detected based on whether all objects included in the designation list are included in at least one of the object description 50 and the initial state description 51. If at least some of the objects included in the designation list are not detected from the observation data 20, the control unit 11 may drive the drive device of the sensor S and appropriately change the observation direction of the sensor S. The direction and amount of change may be determined by any method, for example, randomly, according to a predetermined rule, or the like. The control unit 11 may then reacquire the observation data 20 and perform the process of step S102 again. The control unit 11 may repeatedly perform these processes until all of the objects included in the designation list are detected. On the other hand, if all of the objects included in the designation list are detected from the observation data 20, the control unit 11 may proceed with the process of generating the problem description 5. According to this modification, the observation direction of the sensor S is corrected to an appropriate direction, and the accuracy of generating the problem description 5 can be expected to be improved.

[0127] The above-described process for checking detected objects may be modified as appropriate depending on the embodiment. For example, the designation list may further include objects that may or may not be observed (recommended objects). If only the recommended object is not detected, the control unit 11 may output an alert and then proceed with the process of generating the problem description 5. Alternatively, the control unit 11 may stop the process of generating the problem description 5 and ask the operator whether or not to continue the process of generating the problem description 5.

[0128] <4.2> The domain description 23 may include a portion dependent on the robot device R (e.g., a description defining the skills of the robot device R) and a portion related to the environment (e.g., a description defining the type of target object). Therefore, similar to the method for generating the problem description 5 in the above embodiment, the information processing device 1 may use a calculation module to generate at least a portion of the portion related to the environment of the domain description 23 from at least a portion of the observation data 20, the instruction information 21, and the environmental information 22. The calculation module may be configured by at least one of a trained model and a rule-based model. This modification is expected to reduce the effort required to prepare the domain description 23.

[0129] <4.3> In the example of Figure 4 above, the reasoning module 3 is configured to individually generate descriptions (50, 51, 52) of the objects, initial states, and goal states in the problem statement 5. However, the configuration of the reasoning module 3 is not limited to this example. In another example, the reasoning module 3 may be configured to integrally generate at least any combination of descriptions (50, 51, 52) of the objects, initial states, and goal states in the problem statement 5.

[0130] FIG. 9 shows a schematic diagram of an example of the configuration of an inference module 3A according to another embodiment. The inference module 3A includes an object detector 371, a caption model 373, and a large-scale language model 375. In the example of FIG. 9, the observation data 20 may be composed of image data, and the instruction information 21 may be composed of text data instructing the target in natural language. The object detector 371 uses the open vocabulary object model (OVO) proposed in the above-mentioned references 11 and 12. A detection model or an open vocabulary object segmentation model may be used. The caption model 373 may be a visual question answering model proposed in the above-mentioned Reference 13, etc. The large-scale language model 375 may be a visual question answering model proposed in the above-mentioned Reference 14, etc. The proposed model may be used.

[0131] First, the control unit 11 of the information processing device 1 may use the object detector 371 to detect objects from the observation data 20. The control unit 11 may extract partial data (partial image data) of each object from the observation data 20 according to the object detection results by the object detector 371, and provide the extracted partial data to the caption model 373. The control unit 11 may use the caption model 373 to generate a caption for each object from the extracted partial data of each object. Then, the control unit 11 may use the large-scale language model 375 to generate each description (50, 51, 52) of the problem description 5 from the text data of the object detection results by the object detector 371, the text data of the caption by the caption model 373, and the instruction information 21. This modification can simplify the calculation process.

[0132] <4.4> In the above embodiment, the re-prompting method in FIG. 3 may be appropriately improved depending on the embodiment. For example, the re-prompting method in FIG. 3 may be modified as described in Reference 15 (Jason Wei et al., "Chain-of-Thought Prompting Elicits Reasoning in Large Language Models", [online], [Retrieved October 24, 2023], Internet<URL:https: / / arxiv.org / pdf / 2201.11903.pdf> ), Reference 16 (Takeshi Kojima et al., “Large Language Models are Zero-Shot Reasoners”, [online], [Retrieved October 24, 2020], In Internet<URL:https: / / arxiv.org / abs / 2205.11916> ), Reference 17 (Denny Zh ou et al., “Least-to-Most Prompting Enables Complex Reasoning in Large Language Models”, [online], [Retrieved October 24, 2023], Internet<URL:https: / / arxiv.org / abs / 2205.10625> Chain-of-Thought prompts such as those proposed by The Chain-of-Thought prompt is a method of obtaining a final inference result (the result of generating the problem description 5 in this embodiment) by repeating stepwise inferences through intermediate inference steps. For example, the control unit 11 of the information processing device 1 may display a question such as "What part of the PDDL problem do you think is causing this error?" together with the problem description 5 and the error message 615. The control unit 11 may provide the inference module 3 with template data that queries the error, causing the inference module 3 to generate an explanation for the error. Thereafter, the control unit 11 may provide the problem description 5 and the error message 615 to the inference module 3, causing the inference module 3 to generate a new problem description 5.

[0133] §5 Experimental Examples The following experiments were carried out to verify the effectiveness of the above-described embodiment, however, the present invention is not limited to the following examples.

[0134] (Inference module) The inference module configuration shown in FIG. 4 was used for the first example. The object detector configuration was the configuration shown in FIG. 5A. The trained model for the object detector was the model proposed in Reference 11. The initial state estimator configuration was the configuration shown in FIG. 5B. The detector for the initial state estimator was the model proposed in Reference 13. The trained model for the initial state estimator was the model proposed in Reference 14. The target estimator configuration was the configuration shown in FIG. 5C. The trained model for the target estimator was the model proposed in Reference 14.

[0135] Image data was used as the observation data. Text data specifying the target in natural language was used as the instruction information. The trained model of the object estimator was given attribute information indicating the object's attributes in linguistic expressions along with the observation data (image data). The trained model of the initial state estimator was given text data indicating the object detection results, text data of captions by the detector, a description of the object generated by the object estimator, and input / output samples used for few-shot prompting. The trained model of the target estimator was given instruction information (text data), a description of the object generated by the object estimator, a description of the initial state generated by the initial state estimator, and input / output samples used for few-shot prompting. There were three input / output samples.

[0136] The inference module configuration shown in Fig. 9 was used for the second example. The object detector of the inference module was the model proposed in Reference 11. The caption model was the model proposed in Reference 13. The large-scale language model was the model proposed in Reference 14. The large-scale language model was provided with instruction information (text data), text data of the detection results by the object detector, text data of captions by the caption model, and input / output samples used for few-shot prompting.

[0137] PDDL was used as the problem description format. A symbolic planner was prepared to confirm whether an action plan could be generated using the generated problem description. The environment for running the symbolic planner was Fast Downward (Reference 1) and VAL (Reference 18: “KCL-Planning / VAL”, [online], [Retrieved October 24, 2023], Internet).<URL:https: / / github.com / KCL-Planning / VAL> ) was used. In the theory module, we prepared a correction mechanism by re-prompt as shown in Figure 3. The re-prompt consisted of the input data, the generated problem description, and the error message. The method used was the Chain-of-Thought prompt described above. The maximum number of times the problem statement could be revised by re-prompting was set to two.

[0138] (Dataset) Figure 10 shows the types (Object types), predicates, and actions of the domain description for each domain. As shown in Figure 10, the Cooking, Block World We prepared datasets for three domains: Blocksworld, Tower of Hanoi, and Tower of Hanoi. The cooking task was assumed to be slicing vegetables and putting them into a bowl. The robot device R was assumed to have two robot arms, one on the left and one on the right. The target state specified the state and location of the vegetables.

[0139] Blocks-World is based on the reference 19 (Naresh Gupta et al., “On the Complexity of Blocks-World Planning”, [online], [Retrieved October 24, 2023], Internet URL: https: / / www.semanticscholar.org / paper / On-the-Complexity-of-Blocks-World-Planning-Gupta-Nau / db01349fd0d29c9443e37b0b0aa4ddb948ace5ce>) This is a typical domain. Seven unique colored blocks were used for each problem. The robot arm was initially set up so that it did not always grasp something. The goal state specified the relationship between the blocks.

[0140] Hanoi is based on the reference 20 (Ron Alford et al., “Translating HTNs to PDDL: A Small “The Amount of Domain Knowledge Can Go a Long Way,” [online], [Retrieved October 24, 2023], Internet<URL:https: / / www.cs.umd.edu / ~nau / papers / alford2009translating.pdf> ) is a classic domain used in games such as the 3 pegs and 10 cards in 6 colors. The disks were set as follows. Disks of the same color were numbered in order of width. The three pegs were numbered from left to right. The initial state and goal state specified the positions of the disks.

[0141] Figures 11A, 11B, and 11C show examples of observation data and instructions given in the Cooking, Block World, and Hanoi domains. For each domain, one domain description was prepared. Sample problem descriptions were also prepared for each problem. For Hanoi, the same instructions were used for all problems.

[0142] (Evaluation indicators) To evaluate whether the problem description was properly generated, four indicators (R syntax , R plan , R part , R all ) was prepared. R syntaxis defined as the ratio of generated problem descriptions that conform to the PDDL syntax. If VAL returns no warnings or exit codes, the generated problem description is considered syntactically correct. plan is generated when generating an action plan from the generated problem description. We define the ratio of the number of times that the symbolic planner does not generate an error message. We try to generate an action plan using the symbolic planner, and VAL returns no error message. If the action plan was valid, it was considered to have been generated. part and R all was used to evaluate reproducibility. Therefore, we defined it as the ratio of generated problem descriptions that contain all descriptions in the correct sample. part in R computed the descriptions of the object, the initial state, and the goal state separately. all In this paper, we performed calculations on the entire generated problem description.

[0143] (Experimental example) In the first experimental example, problem descriptions were generated for each of the three domains using the inference module of the first embodiment. The problem description could be modified by re-prompting up to two times. 100 problems were prepared for each domain. Ten problem descriptions were generated for each problem by changing the combination of input and output samples. The generated problem descriptions were evaluated using the above four evaluation indices (R syntax , R plan , R part , R all ) was calculated.

[0144] In the second experimental example, problem descriptions were generated in each of the three domains using the inference module of the second embodiment. Three evaluation indices (R syntax , R plan , R all ) was calculated. The other conditions were set to be the same as in the first experimental example.

[0145] In the third experimental example, the inference module of the first embodiment was used, and (A) correction reprompt (CR) and Chain-of-Thought (CoT) prompts were used (up to two corrections were allowed), and (B) correction reprompt (CR) and Chain-of-Thought (CoT) prompts (only one correction allowed), (C) Correction reprompt (CR) and no Chain-of-Thought (CoT) prompt (correction is 1 time) (D) Corrective Reprompts (CR) and Chain-of-Thought (CoT) Prompts We generated problem descriptions in each of the three domains under four conditions: syntax , R plan , R all ) was calculated. The other conditions were set to be the same as in the first experimental example. The other conditions were set to be the same as in the first experimental example.

[0146] (Experimental results) 12, 13, and 14 show the calculation results of each evaluation index in Experimental Example 1, Experimental Example 2, and Experimental Example 3. The values ​​in parentheses in Fig. 13 indicate the difference from the result of Experimental Example 1.

[0147] As shown in Figure 12, in the Hanoi domain of the first experimental example, R plan was slightly low However, in each domain, R syntax and R plan The scores of R were high, i.e., it was able to generate syntactically correct problem descriptions. all Although the score was low, in the cooking domain, R part and R all The scores for both Therefore, it was found that high reproducibility could be achieved in generating problem descriptions. These results demonstrate that effective problem descriptions can be generated according to this embodiment.

[0148] As shown in Figure 13, the inference module of the second embodiment was able to generate syntactically correct problem descriptions, equivalent to the inference module of the first embodiment. all Although the score of R dropped, in the Hanoi domain, all These results show that it is possible to generate effective problem descriptions even when the integrated structure shown in Figure 9 is adopted as the inference module structure.

[0149] In addition, as shown in Figure 14, by overlapping the correction re-prompt (CR) and Chain-of-Thought (CoT) prompts, R syntax , R plan and R all All scores increased. From the results, correction reprompts (CR) and Chain-of-Thought (CoT) prompts are used to answer the appropriate questions. It was found to be effective in obtaining title descriptions.

[0150] This specification includes the following disclosure. [Appendix 1] Obtaining observation data of an environment in which the robotic device operates and instruction information regarding a goal of a task to be given to the robotic device; generating a problem description of the task from the acquired observation data and the instruction information using a reasoning module, the problem description including descriptions of initial and goal states of objects in the environment; and outputting the generated problem description; a control unit configured to perform Information processing device. [Appendix 2] the problem description generated follows a predetermined format; 2. The information processing device according to claim 1. [Appendix 3] the generated problem description further includes a description of the objects present in the environment; 3. The information processing device according to claim 1 or 2. [Appendix 4] the observation data is composed of sensing data from a sensor; the instruction information is composed of linguistic information that indicates the target in natural language; 4. An information processing device according to any one of claims 1 to 3. [Appendix 5] the inference module is configured to include a trained model for in-context learning; The control unit provides the generated problem description to a planner, and when the planner outputs an error message during a process of generating a behavior plan for the robot device, acquiring the output error message; generating a new problem description from the problem description and the error message using the reasoning module; and further configured to perform 5. An information processing device according to any one of claims 1 to 4. [Appendix 6] In the acquiring step, the control unit is configured to further acquire environmental information related to the environment; generating the problem description from the acquired observation data and the instruction information comprises generating the problem description from the acquired observation data, the instruction information, and the environment information; 6. An information processing device according to any one of Supplementary Note 1 to Supplementary Note 5. [Appendix 7] the generated problem description further comprises a description of the objects present in the environment; the inference module includes an object estimator; generating the problem statement using the reasoning module includes generating descriptions of the objects present in the environment from the acquired observation data using the object estimator; 7. An information processing device according to any one of Supplementary Note 1 to Supplementary Note 6. [Appendix 8] the object estimator comprises a trained model for in-context learning; 8. The information processing device according to claim 7. [Appendix 9] In the acquiring step, the control unit is configured to further acquire attribute information of the object present in the environment; generating a description of the object from the acquired observation data comprises generating a description of the object from the acquired observation data and the attribute information; 9. The information processing device according to claim 7 or 8. [Appendix 10] the inference module includes an initial state estimator; generating the problem statement using the reasoning module includes generating a description of the initial states of the objects in the environment using the initial state estimator. 10. An information processing device according to any one of Supplementary Note 1 to Supplementary Note 9. [Appendix 11] the initial state estimator comprises a trained model for in-context learning; 11. The information processing device according to claim 10. [Appendix 12] the inference module includes a target estimator; generating the problem statement using the reasoning module includes generating a description of the goal state of the objects in the environment using the goal estimator. 12. An information processing device according to any one of claims 1 to 11. [Appendix 13] the goal estimator comprises a trained model for in-context learning; 13. The information processing device according to claim 12. [Appendix 14] The computer Obtaining observation data of an environment in which the robotic device operates and instruction information regarding a goal of a task to be given to the robotic device; generating a problem description of the task from the acquired observation data and the instruction information using a reasoning module, the problem description including descriptions of initial and goal states of objects in the environment; and outputting the generated problem description; To execute Information processing methods. [Appendix 15] On the computer, Obtaining observation data of an environment in which the robotic device operates and instruction information regarding a goal of a task to be given to the robotic device; generating a problem description of the task from the acquired observation data and the instruction information using a reasoning module, the problem description including descriptions of initial and goal states of objects in the environment; and outputting the generated problem description; In order to execute program. [Explanation of symbols]

[0151] 1...information processing device, 11...control unit, 12...storage unit, 13...external interface, 14...input device, 15...output device, 16...drive, 81...program, 91...storage medium, 111...acquisition section, 112...generation section, 113...output section, 3...inference module, 300...module data, 31...object estimator, 33...initial state estimator, 35...target estimator, 20...observation data, 21...instruction information, 22...Environmental information, 23...Domain description, 24...Domain information, 241...Attribute information, 5...Problem statement, 50...Description of the object, 51...Description of the initial state, 52...Description of the target state

Claims

1. Obtaining observation data of an environment in which the robotic device operates and instruction information regarding a goal of a task to be given to the robotic device; generating a problem description of the task from the acquired observation data and the instruction information using a reasoning module, the problem description including descriptions of initial and goal states of objects in the environment; and outputting the generated problem description; a control unit configured to perform Information processing device.

2. the problem description generated follows a predetermined format; The information processing device according to claim 1 .

3. the generated problem description further includes a description of the objects present in the environment; The information processing device according to claim 1 .

4. the observation data is composed of sensing data from a sensor; the instruction information is composed of linguistic information that indicates the target in natural language; The information processing device according to claim 1 .

5. the inference module is configured to include a trained model for in-context learning; The control unit provides the generated problem description to a planner, and when the planner outputs an error message during a process of generating a behavior plan for the robot device, acquiring the output error message; generating a new problem description from the problem description and the error message using the reasoning module; and further configured to perform The information processing device according to claim 1 .

6. In the acquiring step, the control unit is configured to further acquire environmental information related to the environment; generating the problem description from the acquired observation data and the instruction information comprises generating the problem description from the acquired observation data, the instruction information, and the environment information; The information processing device according to claim 1 .

7. the generated problem description further comprises a description of the objects present in the environment; the inference module includes an object estimator; generating the problem statement using the reasoning module includes generating descriptions of the objects present in the environment from the acquired observation data using the object estimator; The information processing device according to claim 1 .

8. the object estimator comprises a trained model for in-context learning; The information processing device according to claim 7 .

9. In the acquiring step, the control unit is configured to further acquire attribute information of the object present in the environment; Generating a description of the object from the acquired observation data includes: generating a description of the object from the data and the attribute information; 9. The information processing device according to claim 7 or 8.

10. the inference module includes an initial state estimator; generating the problem statement using the reasoning module includes generating a description of the initial states of the objects in the environment using the initial state estimator. The information processing device according to claim 1 .

11. the initial state estimator comprises a trained model for in-context learning; The information processing device according to claim 10.

12. the inference module includes a target estimator; generating the problem statement using the reasoning module includes generating a description of the goal state of the objects in the environment using the goal estimator. The information processing device according to claim 1 .

13. the goal estimator comprises a trained model for in-context learning; The information processing device according to claim 12.

14. The computer Obtaining observation data of an environment in which the robotic device operates and instruction information regarding a goal of a task to be given to the robotic device; generating a problem description of the task from the acquired observation data and the instruction information using a reasoning module, the problem description including descriptions of initial and goal states of objects in the environment; and outputting the generated problem description; To execute Information processing methods.

15. On the computer, Obtaining observation data of an environment in which the robotic device operates and instruction information regarding a goal of a task to be given to the robotic device; generating a problem description of the task from the acquired observation data and the instruction information using a reasoning module, the problem description including descriptions of initial and goal states of objects in the environment; and outputting the generated problem description; In order to execute program.

Images