Information processing device, and information processing method

Abstract

[PROBLEM] To describe robot operations by combining granular units having higher versatility. [SOLUTION] This information processing device comprises: an instruction acquisition unit for acquiring an operation instruction configured by combining a control primitive for controlling a function unit of a robot, a sub-goal input to the control primitive, and an output primitive for outputting the sub-goal; and an output generation unit for, on the basis of the operation instruction, generating output for controlling the function unit of the robot.

Description

Information processing device and information processing method

[0001] The present disclosure relates to an information processing device and an information processing method.

[0002] In recent years, pet-type robots that mimic quadrupedal animals or humanoid robots that walk upright on two legs have been put to practical use in the service field.

[0003] Such service robots are required to perform a variety of actions in unknown environments. Therefore, it is desirable to describe a variety of actions by combining general-purpose elements, rather than preparing dedicated action patterns for each environment.

[0004] For example, the authoring tool disclosed in the following Patent Document 1 allows a user to create a robot's motion pattern by connecting boxes that define the robot's motion states. The processing operations in each box are written in, for example, a programming language designed for robot control.

[0005] Japanese Patent Application Laid-Open No. 2001-353678

[0006] However, in the authoring tool disclosed in Patent Document 1, the processing operations in each box are described in units of actions such as "taking a certain posture," "executing a certain motion," or "stopping an action currently being executed." Therefore, for robots that can execute different actions, different programming languages ​​corresponding to the robots are required.

[0007] Therefore, the present disclosure proposes a new and improved information processing device and information processing method that are capable of describing the behavior of a robot by combining units of higher granularity with higher versatility.

[0008] According to the present disclosure, an information processing device is provided, comprising: a control primitive that controls a functional unit possessed by a robot; an instruction acquisition unit that acquires an operation instruction constituted by combining a subgoal input to the control primitive and an output primitive that outputs the subgoal; and an output generation unit that generates an output for controlling the functional unit of the robot based on the operation instruction.

[0009] Furthermore, according to the present disclosure, there is provided an information processing method by a computer, which includes obtaining an operation instruction constituted by combining a control primitive that controls a functional unit possessed by a robot, a subgoal that is input to the control primitive, and an output primitive that outputs the subgoal, and generating an output for controlling the functional unit of the robot based on the operation instruction.

[0010] 1 is a schematic diagram showing an example of a robot whose behavior is described in an information processing device according to an embodiment of the present disclosure. FIG. 1 is a schematic diagram showing another example of a robot whose behavior is described in an information processing device according to the embodiment. FIG. 2 is a schematic diagram showing another example of a robot whose behavior is described in an information processing device according to the embodiment. FIG. 3 is a block diagram showing a functional configuration of an information processing device according to the embodiment. FIG. 4 is an explanatory diagram explaining a relationship between functional units of a robot and control primitives. FIG. 5 is an explanatory diagram showing examples of control primitives, subgoals, and output primitives. FIG. 6 is an explanatory diagram showing an example of describing a "Pick" behavior of grasping and lifting an object by hierarchically combining subgoals, primitives, and actions. FIG. 7 is an explanatory diagram showing an example of describing a "Place" behavior of placing a grasped object on a floor surface or the like by hierarchically combining subgoals, primitives, and actions. FIG. 8 is an explanatory diagram showing an image in which a behavior instruction to a robot is generated by combining primitives and subgoals. FIG. 9 is an explanatory diagram showing an image in which a behavior instruction to a robot is generated by combining generated actions. FIG. 10 is an explanatory diagram showing a correspondence relationship between execution of primitives and an execution log generated by a log generation unit. 14A is an explanatory diagram showing the correspondence between the execution of a primitive that acquires information about the environment and the environmental log generated by the log generation unit. FIG. 14B is an explanatory diagram conceptually showing a directed graph generated based on the input / output relationship of the primitive. FIG. 14C is a perspective view schematically showing a state in which a robot executes an action of placing an object on a floor surface. FIG. 14B is a top view looking down from above the state shown in FIG. 14A. FIG. 14C is a perspective view schematically showing a state in which a robot executes an action of lifting an object from the floor surface. FIG. 14D is a block diagram showing an example of the hardware configuration of an information processing device according to the embodiment.

[0011] Preferred embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings. In this specification and drawings, components having substantially the same functional configurations are designated by the same reference numerals, and redundant description will be omitted.

[0012] The description will be given in the following order: 1. Overview of the information processing device 2. Configuration of the information processing device 3. Modifications 3.1. First modification 3.2. Second modification 3.3. Third modification 4. Example of hardware configuration

[0013] 1. Overview of Information Processing Device First, an overview of an information processing device according to an embodiment of the present disclosure will be described with reference to Figures 1 to 4. Figure 1 is a schematic diagram showing an example of a robot whose behavior is described by the information processing device according to this embodiment. Figures 2 to 4 are schematic diagrams showing other examples of robots whose behavior is described by the information processing device according to this embodiment.

[0014] 1, the information processing device according to this embodiment is used to describe the operation of, for example, a robot 1 having multiple functional units. For example, the robot 1 is a mobile manipulator robot including a main body 10 and multiple functional units such as a carriage 11A, a head 11B, a manipulator 11C, and a hand 11D.

[0015] The main body 10 is a main part of the robot 1 to which the carriage 11A, the head 11B, and the manipulator 11C are attached. The main body 10 is further equipped with a control device, a power supply device, a communication device, and the like.

[0016] The carriage 11A is a support for the main body 10 and includes a movement mechanism capable of moving the robot 1 to any position. The carriage 11A may include, for example, a wheeled movement mechanism. The carriage 11A may also include a legged movement mechanism, a leg-wheeled movement mechanism, or a crawler movement mechanism.

[0017] The head 11B is provided on the upper part of the main body 10 and includes, for example, an imaging device that captures an image within the field of view of the robot 1. The head 11B may further include various sensors such as LiDAR (Light Detection And Ranging) or Radar (Radio Detection And Ranging), and may further include a display unit or an audio output unit for communicating with the user.

[0018] The manipulator 11C is an arm having a structure in which a plurality of links are connected by at least one joint. A hand 11D capable of grasping an object is provided at the tip of the manipulator 11C. Only one manipulator 11C may be provided, or a pair of manipulators 11C may be provided on the left and right.

[0019] The hand 11D is an end effector provided at the tip of the manipulator 11C and includes various sensors for smoothly gripping an object. The hand 11D may be, for example, a two- or three-fingered gripper, a four- or five-fingered hand, or a suction-type gripper. The hand 11D may include a proximity sensor for measuring the distance to the object, or a pressure sensor for detecting the force applied to the object.

[0020] In the robot 1, the operation of each of the functional units, namely the carriage 11A, the head 11B, the manipulator 11C, and the hand 11D, is performed to execute the operation of the entire robot 1. Therefore, the information processing device according to this embodiment can describe the operation of the robot 1 using, as basic units, control units (control primitives) that control these functional units, target values ​​(subgoals) that are input to the control units, and output units (output primitives) that output the target values. This makes it possible for the information processing device according to this embodiment to describe the operation of the entire robot 1 as a combination of the operations of the functional units. Accordingly, when describing the operation of another robot having a common functional unit, the information processing device according to this embodiment can reuse the control primitives, subgoals, and output primitives of the common functional units.

[0021] For example, the information processing device according to this embodiment can describe the behavior of various robots shown in Figures 2 to 4 by reusing the control primitives, subgoals, and output primitives used to describe the behavior of robot 1 shown in Figure 1.

[0022] 2 is a mobile manipulator robot that includes a main body 10, a carriage 11A, a head 11B, and a manipulator 11C, with the tip of the manipulator 11C being a fixed hook or hand without a sensor. The main body 10, carriage 11A, head 11B, and manipulator 11C of the robot 2 are essentially the same as those of the robot 1. Therefore, the information processing device according to this embodiment can describe the behavior of the robot 2 by reusing the control primitives, subgoals, and output primitives used to describe the behavior of the carriage 11A, head 11B, and manipulator 11C of the robot 1.

[0023] 3 is an interactive robot that includes a main body 10, a carriage 11A, and a head 11B and is capable of communicating with a user, such as by conversation. The main body 10, carriage 11A, and head 11B of the robot 3 are essentially the same as those of the robot 1. Therefore, the information processing device according to this embodiment can describe the behavior of the robot 3 by reusing the control primitives, subgoals, and output primitives used to describe the behavior of the carriage 11A and head 11B of the robot 1.

[0024] 4 is a transport robot equipped with a carriage 11A that transports an object placed on it. The carriage 11A of the robot 4 is essentially the same as that of the robot 1. Therefore, the information processing device according to this embodiment can describe the behavior of the robot 4 by reusing the control primitives, subgoals, and output primitives used to describe the behavior of the carriage 11A of the robot 1.

[0025] In other words, the information processing device according to this embodiment can uniquely determine the basic unit for describing the motion based on the functional units of the robot, and therefore the information processing device according to this embodiment can systematically describe the motion of a robot having various functional units for each functional unit.

[0026] 2. Configuration of Information Processing Apparatus> Next, the configuration of the information processing apparatus 100 according to this embodiment will be described with reference to FIGS. 5 to 12. FIG.

[0027] 5 is a block diagram showing the functional configuration of the information processing device 100 according to this embodiment. As shown in FIG. 5, the information processing device 100 includes an input / output unit 110, an image generation unit 120, an instruction acquisition unit 130, an output generation unit 140, a DB storage unit 150, a log generation unit 160, and a log storage unit 170.

[0028] The input / output unit 110 is an input interface or an output interface for exchanging information with a user or developer.

[0029] Examples of input interfaces included in the input / output unit 110 include a mouse, a keyboard, a touch panel, a button, a switch, a lever, etc. The input / output unit 110 can output to the instruction acquisition unit 130, an operation instruction that instructs the behavior or operation of the robot 1, input by a user or a developer via these input interfaces.

[0030] Examples of output interfaces included in the input / output unit 110 include display devices such as a liquid crystal display (LCD), a plasma display panel (PDP), an organic light emitting diode (OLED) display, a hologram, or a projector. The input / output unit 110 can present the image generated by the image generation unit 120 to a user or a developer via these output interfaces.

[0031] The input / output unit 110 may be a connection interface for connecting to the above-mentioned external input interface or output interface.

[0032] The instruction acquisition unit 130 acquires operation instructions that instruct the behavior or operation of the robot 1, which are input from a user or developer via the input / output unit 110. The operation instructions acquired by the instruction acquisition unit 130 are configured by combining at least control primitives that control functional units possessed by the robot 1, subgoals that are input to the control primitives, and output primitives that output the subgoals.

[0033] The control primitives, subgoals, and output primitives will be described with reference to Figures 6 and 7. Figure 6 is an explanatory diagram illustrating the relationship between the functional unit 11 of the robot 1 and the control primitive PR1. Figure 7 is an explanatory diagram showing an example of the control primitive PR1, subgoal SG, and output primitive PR2.

[0034] As shown in FIG. 6, it is assumed that the robot 1 has functional units 11 including a carriage 11A, a head 11B, a manipulator 11C, and a hand 11D.

[0035] A control primitive PR1 that controls the behavior of the functional unit 11 is set for each functional unit 11. The control primitive PR1 is a basic unit of a module that controls the behavior of the functional unit 11, and controls the functional unit 11 so that it reaches a subgoal SG, which is an input target value. The control primitive PR1 is created, for example, by a developer through coding using a programming language.

[0036] For example, a "cart movement" control primitive is set as the control primitive PR1 that controls the movement of the cart 11A. Also, a "joint trajectory tracking" control primitive is set as the control primitive PR1 that controls the movement of the rear end side and head 11B of the manipulator 11C. Furthermore, a "hand trajectory tracking" control primitive is set as the control primitive PR1 that controls the movement of the tip end side of the manipulator 11C.

[0037] However, some control primitives PR1 may not have a subgoal SG, which is a target value, input. For example, the "grasp / release" control primitive that controls the opening and closing movement of the hand 11D is a binary control between a grasped state and an released state, so no subgoal SG, which is a target value, is input. Also, the "proximity servo" control primitive that controls the hand 11D's proximity to an object performs control based on feedback from a proximity sensor provided in the hand 11D, so no subgoal SG, which is a target value, is input.

[0038] As shown in Fig. 7, the subgoal SG is set as a target value to be input to the control primitive PR1. Specifically, the subgoal SG of "cart movement (target)" is input to the control primitive of "cart movement." The subgoal SG of "joint trajectory (target)" is input to the control primitive of "joint trajectory tracking." The subgoal SG of "hand trajectory (target)" is input to the control primitive of "hand trajectory tracking."

[0039] Furthermore, an output primitive PR2 that outputs the subgoal SG is set for each subgoal SG. The output primitive PR2 is the basic unit of the module that outputs the subgoal SG, and outputs the subgoal SG based on other input subgoals SG, for example.

[0040] 6 and 7 , the format of the subgoal SG input to the control primitive PR1 and the format of the subgoal SG output from the output primitive PR2 are represented by the concave-convex shapes provided on the control primitive PR1 and the output primitive PR2. That is, a subgoal SG having a concave-convex shape on its lower side that fits with the concave-convex shape on its upper side of the control primitive PR1 is input to the control primitive PR1. Furthermore, a subgoal SG having a concave-convex shape on its upper side that fits with the concave-convex shape on the lower side of the output primitive PR2 is output from the output primitive PR2. Therefore, the control primitive PR1 and the output primitive PR2 can be connected to each other via the subgoal SG with the convex-convex shape that fits.

[0041] An example of an output primitive PR2 that outputs a subgoal SG based on other input subgoals SG is a planning primitive that outputs a target value (subgoal SG) to be reached by each of the functional units 11 based on the input subgoal SG.

[0042] For example, the output primitive for "cart placement planning" outputs a subgoal of "cart position (target)" by planning the position of the cart 11A based on the input subgoal of "object name." The output primitive for "joint space planning" outputs a subgoal of "joint trajectory (target)" by planning the trajectory of the manipulator 11C based on the input subgoal of "joint angle (target)." The output primitive for "attention planning" outputs a subgoal of "joint trajectory (target)" by planning the trajectory of the head 11B based on the input subgoal of "attention target." The output primitive for "task space planning" outputs a subgoal of "joint trajectory (target)" by planning the trajectory of the hand 11D based on the input subgoal of "hand target."

[0043] Another example of an output primitive PR2 that outputs a subgoal SG based on another input subgoal SG is, for example, a goal setting primitive that sets a different target value (subgoal SG) based on the input subgoal SG.

[0044] For example, the output primitive for "gaze target setting" outputs the subgoal "gaze target" by setting the target to look at based on the input subgoal of "object name." The output primitive for "hand target setting" outputs the subgoal of "hand target" by setting the target to be grasped by the hand based on the input subgoal of "object name."

[0045] The output primitive PR2 can also output a subgoal SG without inputting other subgoals SG. An example of such an output primitive PR2 is a setting primitive that sets a movement target for the robot 1.

[0046] For example, the output primitive for "object setting" outputs the subgoal "object name." The output primitive for "movement target setting" outputs the subgoal "cart position (target)." The output primitive for "target setting in task space" outputs the subgoal "hand trajectory (target)." Furthermore, the output primitive for "posture reading" outputs the subgoal "joint angle (target)."

[0047] By combining the control primitives PR1, subgoals SG, and output primitives PR2 described above, the user can describe the actions to be instructed to the robot 1 by combining a certain number of subgoals and primitives. Furthermore, by further combining actions, the user can hierarchically describe behavior that includes multiple actions.

[0048] In this embodiment, first, a control primitive PR1 is set to control the movement of the functional unit 11 of the robot 1. Next, an output primitive PR2 is set as a setting primitive, a goal setting primitive, or a planning primitive to output a subgoal SG input to the control primitive PR1. Subsequently, an output primitive is set to output a subgoal SG input to these output primitives PR2. By repeating this primitive setting process until there is no more input or until the input is provided from outside, a primitive is set that is a basic unit that describes the movement of the robot 1 based on the subgoal SG. This allows the information processing device 100 to systematically set primitives that describe the movement of the robot 1.

[0049] 8A and 8B, a hierarchical description of actions to be instructed to the robot 1 will be described. Fig. 8A is an explanatory diagram showing an example of describing a "Pick" action of grasping and lifting an object by hierarchically combining subgoals, primitives, and actions. Fig. 8B is an explanatory diagram showing an example of describing a "Place" action of placing a grasped object on a floor surface or the like by hierarchically combining subgoals, primitives, and actions.

[0050] 8A and 8B, each of the movements of the robot 1 is described by sequentially linking an output primitive PR2 to the control primitive PR1 via a subgoal SG before the control primitive PR1. Furthermore, the movements described by linking the control primitive PR1, the subgoal SG, and the output primitive PR2 are combined with each other to describe the behavior to be instructed to the robot 1.

[0051] 8A , the action of “bringing a cart closer to an object” is described by a combination of a setting primitive of “object setting,” a subgoal of “object name,” a planning primitive of “cart placement plan,” a subgoal of “cart position (target),” and a control primitive of “cart movement.” The action of “looking at an object” is described by a combination of a setting primitive of “object setting,” a subgoal of “object name,” a goal setting primitive of “gaze target setting,” a subgoal of “gaze target (target),” a planning primitive of “gaze plan,” a subgoal of “joint trajectory (target),” and a control primitive of “joint trajectory tracking.” The action of “lifting a hand” is described by a combination of a goal setting primitive of “goal setting in task space,” a subgoal of “hand trajectory (target),” and a control primitive of “hand trajectory tracking.” The action of "bringing the hand closer to the grasping position" is described by a combination of the planning primitive of "grasp planning," the goal setting primitive of "fine-tuning the hand position," the subgoal of "hand trajectory (target)," the control primitive of "hand trajectory tracking," and the control primitive of "proximity servo." The action of "returning to the base posture" is described by a combination of the setting primitive of "load posture," the subgoal of "joint angle (target)," the planning primitive of "planning in joint space," the subgoal of "joint trajectory (target)," and the control primitive of "joint trajectory tracking."

[0052] The action of "bringing the hand closer to the object" is described by combining the actions of "bringing the cart closer to the object" and "looking at the object." The action of "grasping an object" is described by combining the "grasp / release" control primitive and the action of "lifting the hand."

[0053] Furthermore, the "Pick" action of grasping and lifting an object is described by combining the actions of "bringing the hand closer to the object," "bringing the hand closer to the grasping position," "grasping the object," and "returning to the basic position."

[0054] As shown in Figure 8B, the action of "bringing the cart closer to the object" is described by a combination of the setting primitive of "object setting," the subgoal of "object name," the planning primitive of "cart placement plan," the subgoal of "cart position (target)," and the control primitive of "cart movement," as in Figure 8A. The action of "looking at an object" is described by a combination of the setting primitive of "object setting," the subgoal of "object name," the goal setting primitive of "gaze target setting," the subgoal of "gaze target (target)," the planning primitive of "gaze plan," the subgoal of "joint trajectory (target)," and the control primitive of "joint trajectory tracking," as in Figure 8A. The action of "lowering the hand" is described by a combination of the goal setting primitive of "goal setting in task space," the subgoal of "hand trajectory (target)," and the control primitive of "hand trajectory tracking." The action of "returning to the basic posture" is described by combining the setting primitive of "load posture", the subgoal of "joint angle (target)", the planning primitive of "planning in joint space", the subgoal of "joint trajectory (target)", and the control primitive of "following the joint trajectory", as in FIG. 8A.

[0055] 8A, the action of "bringing the hand closer to the object" is described by combining the action of "bringing the cart closer to the object" and the action of "looking at the object." The action of "releasing the object" is described by combining the control primitive of "grasp / release" and the action of "lowering the hand."

[0056] Furthermore, the "Place" action of placing a grasped object is described by combining the actions of "bringing the hand closer to the object," "releasing the object," and "returning to the base position."

[0057] In this embodiment, a description using such control primitives PR1, subgoals SG, and output primitives PR2 is used, and common primitives, subgoals, and actions are reused to describe a variety of actions or behaviors.

[0058] In this embodiment, the motion or behavior of the robot 1 is described based on the subgoal SG that is input and output between the control primitive PR1 and the output primitive PR2. This clarifies the input and output of data between the control primitive PR1, which is the basic unit for describing the motion, and the output primitive PR2, preventing the formation of a dependency relationship between the primitives. Therefore, in this embodiment, it is possible to set the control primitive PR1 and the output primitive PR2 at a granularity that allows for greater versatility.

[0059] The image generation unit 120 generates an image for a user or developer to generate an operation instruction that instructs the behavior or operation of the robot 1. The image generated by the image generation unit 120 is presented to the user or developer via the input / output unit 110.

[0060] The images generated by the image generation unit 120 will be described with reference to Fig. 9 and Fig. 10. Fig. 9 is an explanatory diagram showing an image in which a motion instruction to the robot 1 is generated by combining primitives and subgoals. Fig. 10 is an explanatory diagram showing an image in which a behavior instruction to the robot 1 is generated by combining the generated motions.

[0061] As shown in FIG. 9, a user or developer can create an action to instruct the robot 1 to perform by referring to the list image M1 and the action creation image M2 generated by the image generation unit 120.

[0062] The list image M1 displays a list of various primitives and subgoals created by the developer, and various actions created by the user or developer. The various primitives displayed in the list image M1 are created by the developer through coding using a programming language, as shown in the speech bubble image P1, for example. The speech bubble image P1 may be hidden, or may be displayed when the user or developer selects a primitive. The user may also be able to select whether to display the speech bubble image P1. The various primitives may be created, for example, in a separate window not shown.

[0063] The action creation image M2 displays primitives and subgoals selected by the user or developer from the various primitives and various subgoals in the list image M1. The user or developer can intuitively create actions for the robot 1 by combining the primitives and subgoals selected from the list image M1 in the vertical direction on the action creation image M2. The created actions of the robot 1 are stored in the DB storage unit 150 (described later) in response to a save instruction from the user or developer, and then added to the list of various actions in the list image M1.

[0064] In the action creation image M2, various primitives and various subgoals are displayed as block-shaped images. The block-shaped images representing the primitives have convex and concave shapes on the top and bottom that correspond to the types of the input and output subgoals. Similarly, the block-shaped images representing the subgoals have convex and concave shapes on the top and bottom that fit with the convex and concave shapes of the input and output primitives. This allows a user or developer to intuitively match the input and output of primitives and subgoals by combining primitives and subgoals that have convex and concave shapes that fit with each other.

[0065] As shown in FIG. 10, a user or developer can create an action instruction to instruct the robot 1 to perform by referring to the list image M3 and the action creation image M4 generated by the image generation unit 120.

[0066] The list image M3 displays a list of various actions and behaviors created by a user or developer. The actions displayed in the list image M3 are actions created by the user or developer by combining primitives and subgoals in the action creation image M2, for example.

[0067] The behavior creation image M4 displays actions selected by the user or developer from the various actions in the list image M3. The user or developer can intuitively create behavior instructions for the robot 1, including multiple action instructions, by combining the actions selected from the list image M3 in the vertical direction on the behavior creation image M4. The created behavior of the robot 1 is stored in the DB storage unit 150, which will be described later, in response to a save instruction from the user or developer, and then added to the list of various actions in the list image M3.

[0068] The DB storage unit 150 stores the primitives and actions created by a user or developer in a database. Specifically, the DB storage unit 150 may store a database in which various primitives coded by a developer using a programming language are registered. The DB storage unit 150 may also store a database in which various actions created by a user or developer by referring to the action creation image M2 described above are registered. The DB storage unit 150 may be configured, for example, by a magnetic storage device such as a hard disk drive (HDD), a semiconductor storage device, an optical storage device, or a magneto-optical storage device.

[0069] The output generation unit 140 generates an output for controlling the robot 1 based on the operation instruction acquired by the instruction acquisition unit 130. Specifically, the output generation unit 140 may generate an output for controlling each of the functional units 11 of the robot 1 based on the primitives and operations included in the acquired operation instruction. For example, the output generation unit 140 may generate an output for controlling each of the functional units 11 by converting the contents of the primitives into a format understandable by each of the functional units 11 based on various databases stored in the DB storage unit 150.

[0070] The log generating unit 160 generates an execution log of various primitives included in the operation instruction. Specifically, the log generating unit 160 may generate an execution log that chronologically records the start and end times of each of the various primitives included in the operation instruction. The execution log generated by the log generating unit 160 is stored in the log storage unit 170, which will be described later.

[0071] 11, a specific example of an execution log generated by the log generating unit 160 will be described. FIG. 11 is an explanatory diagram showing the correspondence between the execution of a primitive and the execution log generated by the log generating unit 160.

[0072] 11, in a series of action instructions, four primitives, "SeEeTarget," "PlanEePose," "FollowJointTrajectory," and "SetEeTarget," are executed in sequence. Of these, "SeEeTarget" and "PlanEePose" are output primitives (setting primitive and planning primitive), and "FollowJointTrajectory" is a control primitive.

[0073] In such a case, the log generation unit 160 may generate an execution log including the name of the executed primitive, the start (TICK) time, the end (SUCCESS) time, and the event tag. Furthermore, for a control primitive among the output primitives, the log generation unit 160 may further store in the execution log the state of the robot 1 at each of the start time and end time. For example, in the execution log shown in FIG. 11 , the "joint state," "odometry," and "estimated self-position" at the start time and end time of the control primitive "FollowJointTrajectory" are further stored as the state of the robot 1.

[0074] In this embodiment, the timing at which the state of the robot 1 changes is clearly defined as the execution time of a control primitive. Therefore, the log generation unit 160 can limit the timing at which the state of the robot 1 is stored in the execution log to the start and end of a control primitive. Therefore, the log generation unit 160 can reduce the amount of data in the execution log that stores the time-series changes in the state of the robot 1.

[0075] Furthermore, the log generating unit 160 may generate an environmental log that stores the state of the environment in which the robot 1 exists. Specifically, the log generating unit 160 may generate an environmental log that stores environmental information acquired from sensing results detected by various sensors possessed by the robot 1. The environmental log generated by the log generating unit 160 is stored in the log storage unit 170, which will be described later.

[0076] A specific example of an environment log generated by the log generating unit 160 will be described with reference to Fig. 12. Fig. 12 is an explanatory diagram showing the correspondence between the execution of a primitive that acquires information about an environment and the environment log generated by the log generating unit 160.

[0077] 12 , the primitive "SetObject" acquires the position of an object present in the environment from the sensing results detected by a sensor possessed by the robot 1. The log generating unit 160 may store the position of the object present in the environment and the time when the object's position was acquired in the environment log each time the primitive "SetObject" is executed. This allows the information processing device 100 to easily reproduce past states of the environment in which the robot 1 exists by tracing back the environment log stored in the log storage unit 170 in chronological order.

[0078] The log storage unit 170 stores the execution log and the environment log generated by the log generation unit 160. The execution log and the environment log stored in the log storage unit 170 may be shared with, for example, other robots. The execution log and the environment log stored in the log storage unit 170 may be used to understand the transition over time of the state of the robot 1 and the environment in which the robot 1 exists. The log storage unit 170 may be configured by, for example, a magnetic storage device such as a hard disk drive (HDD), a semiconductor storage device, an optical storage device, or a magneto-optical storage device.

[0079] In the information processing device 100 according to this embodiment, the operation instructions for instructing the operation of the robot 1 are described based on subgoals input and output between primitives. This allows the information processing device 100 to set primitives and subgoals, which are the basic units for describing operation at a more versatile granularity by eliminating the dependency between primitives. Therefore, the information processing device 100 can describe the operation of the robot 1, which has a variety of functional units, in a more versatile format.

[0080] 3. Modifications (3.1. First Modification) Next, a first modification of this embodiment will be described with reference to Fig. 13. The first modification of this embodiment is a modification in which operation instructions to be instructed to the robot 1 are automatically or semi-automatically generated using a directed graph indicating the connection relationships between primitives and subgoals.

[0081] 13 is an explanatory diagram conceptually showing a directed graph 50 generated based on the input / output relationships of primitives. In the directed graph 50 shown in FIG. 13, primitives are represented by rectangles and subgoals are represented by octagons. Furthermore, the input / output relationships of subgoals between primitives are represented by arrows.

[0082] 13, directed graph 50 is formed by starting with control primitive PR1 and sequentially tracing subgoals SG input to control primitive PR1 and output primitives PR2 that input and output subgoals SG based on a reachability matrix. For example, directed graph 50 is formed by calculating primitives reachable from control primitive PR1 via subgoals SG based on the reachability matrix, and then calculating primitives further reachable from the calculated reachable primitives based on the reachability matrix, repeating this process until the end is reached.

[0083] When the directed graph 50 is formed, the information processing device 100 can automatically or semi-automatically generate an operation instruction to be instructed to the robot 1 by tracing the directed graph 50 backward from the desired control primitive PR1. Specifically, the information processing device 100 can automatically generate an operation instruction that combines the specified control primitive PR1, output primitive PR2, and subgoal SG by specifying the control primitive PR1 that outputs the desired control to the robot 1 and the output primitive PR2 or subgoal SG to be passed through.

[0084] According to the first modification, the information processing device 100 can assist a user or developer in creating operation instructions for the robot 1 .

[0085] 14A and 14B, a second modification of the present embodiment will be described. The second modification of the present embodiment is a modification in which, in a series of action instructions, the operation of the robot 1 is controlled based on future changes in the environment.

[0086] Fig. 14A is a perspective view schematically showing the robot 1 performing an action of placing an object 20 on a floor surface G. Fig. 14B is a top view looking down from above the situation shown in Fig. 14A. As shown in Figs. 14A and 14B, in the "Place" action in which the robot 1 places the object 20 it is holding on the floor surface G, the environment of the floor surface G changes as the object 20 is placed. Specifically, the "Place" action adds the object 20 to the floor surface G as a new obstacle.

[0087] On the other hand, in the operation of moving the cart included in the "Place" action, the object 20 is not present on the floor surface G when the movement position of the cart 11A is planned, and therefore it is difficult for the information processing device 100 to plan the movement position of the cart 11A taking the object 20 into consideration. In such a case, the planned movement position 11N of the cart 11A overlaps with the placement position NP of the object 20, and the cart 11A interferes when placing the object 20, which may result in failure to place the object 20.

[0088] In the second modified example, in the output primitive of the "cart dispatch plan" that plans the movement position of the cart 11A, the planned placement position NP of the object 20 is acquired, and the movement position of the cart 11A is planned on the assumption that the object 20 is present at the acquired placement position NP. According to this, when placing the object 20, the information processing device 100 can plan the movement position 11N of the cart 11A so that the movement position 11N of the cart 11A does not interfere with the placement position NP of the object 20. Note that the planned placement position NP of the object 20 can be acquired, for example, from a trajectory plan of the manipulator 11C that grasps the object 20.

[0089] According to the second modification, the information processing device 100 can acquire future environmental changes occurring within a series of behavioral instructions and perform an operation plan that takes the acquired environmental changes into consideration. For example, the information processing device 100 can plan a movement position 11N of the carriage 11A of the robot 1 so as to avoid the placement position NP of the target object 20 placed by the robot 1. Therefore, the information processing device 100 can more smoothly control the robot 1 based on the acquired behavioral instructions.

[0090] (3.3. Third Modification) A third modification of this embodiment will be described with reference to Fig. 15. The third modification of this embodiment is a modification in which the operation of the robot 1 is controlled based on an environmental change that occurs during a series of behavior instructions.

[0091] 15 is a perspective view that schematically shows how the robot 1 performs the action of lifting the object 20 from the floor surface G. As shown in Fig. 15, when the robot 1 performs the "Pick" action of lifting the object 20 from the floor surface G, the environment of the floor surface G changes due to the lifting of the object 20. Specifically, the object 20 is removed from the floor surface G by the "Pick" action.

[0092] On the other hand, in the "Place" action immediately after the "Pick" action, it is not recognized that the object 20 has been removed from the floor surface G. Therefore, it is difficult for the information processing device 100 to plan the trajectory of the manipulator 11C taking into consideration the removal of the object 20. In such a case, the trajectory of the manipulator 11C is planned so as to avoid the placement position NP of the non-existent object 20, which may result in the trajectory being extended or the trajectory planning failing.

[0093] In the third modified example, in the output primitive of "planning in joint space" for planning the trajectory of the manipulator 11C, the placement position NP of the grasped and lifted object 20 is acquired, and the object 20 at the placement position NP is removed to plan the trajectory of the manipulator 11C. This allows the information processing device 100 to prevent a redundant trajectory that avoids a non-existent object 20 from being planned when planning the trajectory of the manipulator 11C.

[0094] According to the third modification, the information processing device 100 can acquire environmental changes that occur during a series of behavioral instructions and perform an operation plan that takes into account the acquired environmental changes. For example, the information processing device 100 can plan the trajectory of the manipulator 11C of the robot 1, taking into account the object 20 that the robot 1 is lifting. Therefore, the information processing device 100 can more smoothly control the robot 1 based on the acquired behavioral instructions.

[0095] 4. Hardware Configuration Example The hardware configuration of the information processing device 100 according to this embodiment will be described further with reference to Fig. 16. Fig. 16 is a block diagram showing an example of the hardware configuration of the information processing device 100 according to this embodiment.

[0096] The functions of the information processing device 100 according to this embodiment are realized by cooperation between software and hardware described below. The functions of the image generation unit 120, the instruction acquisition unit 130, the output generation unit 140, and the log generation unit 160 may be realized, for example, by a CPU 901. The input-related functions of the input / output unit 110 may be realized, for example, by an input device 906, a connection port 910, or a communication device 911. The output-related functions of the input / output unit 110 may be realized, for example, by an output device 907, a connection port 910, or a communication device 911. The functions of the DB storage unit 150 and the log storage unit 170 may be realized, for example, by a storage device 908.

[0097] As shown in FIG. 16, the information processing device 100 includes a CPU (Central Processing Unit) 901 , a ROM (Read Only Memory) 902 , and a RAM (Random Access Memory) 903 .

[0098] The information processing device 100 may further include a host bus 904a, a bridge 904, an external bus 904b, an interface 905, an input device 906, an output device 907, a storage device 908, a drive 909, a connection port 910, or a communication device 911. The information processing device 100 may have a processing circuit such as a DSP (Digital Signal Processor) or an ASIC (Application Specific Integrated Circuit) instead of or together with the CPU 901.

[0099] The CPU 901 functions as an arithmetic processing unit or a control unit, and controls operations within the information processing device 100 in accordance with various programs recorded in a storage device such as the ROM 902, the RAM 903, or the storage device 908, or a removable recording medium attached to the drive 909. The ROM 902 stores programs used by the CPU 901, calculation parameters, etc. The RAM 903 temporarily stores programs used in the execution of the CPU 901, and parameters used during the execution of the programs.

[0100] The CPU 901, ROM 902, and RAM 903 are interconnected by a host bus 904a capable of high-speed data transmission. The host bus 904a is connected to an external bus 904b, such as a PCI (Peripheral Component Interconnect / Interface) bus, via a bridge 904. The external bus 904b is connected to various components via an interface 905.

[0101] The input device 906 is a device that accepts input from a user, such as a mouse, keyboard, touch panel, button, switch, or lever. The input device 906 may also be a microphone that detects the user's voice. The input device 906 may also be, for example, a remote control device that uses infrared rays or other radio waves, or may be an externally connected device that supports operation of the information processing device 100.

[0102] The input device 906 further includes an input control circuit that outputs an input signal generated based on information input by the user to the CPU 901. By operating the input device 906, the user can input various data to the information processing device 100 or instruct the information processing device 100 to perform processing operations.

[0103] The output device 907 is a device capable of visually or audibly presenting information acquired or generated by the information processing device 100 to a user. The output device 907 may be, for example, a display device such as an LCD (Liquid Crystal Display), a PDP (Plasma Display Panel), an OLED (Organic Light Emitting Diode) display, a hologram, or a projector, or may be a sound output device such as a speaker or headphones, or a printing device such as a printer. The output device 907 can output information acquired by processing of the information processing device 100 as video such as text or an image, or sound such as voice or audio.

[0104] The storage device 908 is a data storage device configured as an example of a storage unit of the information processing device 100. The storage device 908 may be configured, for example, by a magnetic storage device such as a hard disk drive (HDD), a semiconductor storage device, an optical storage device, or a magneto-optical storage device. The storage device 908 can store programs executed by the CPU 901, various data, various data acquired from the outside, and the like.

[0105] The drive 909 is a device for reading or writing data from or to a removable recording medium such as a magnetic disk, optical disk, magneto-optical disk, or semiconductor memory, and is built into or externally attached to the information processing device 100. For example, the drive 909 can read information recorded on an attached removable recording medium and output the information to the RAM 903. The drive 909 can also write data to an attached removable recording medium.

[0106] The connection port 910 is a port for directly connecting an external device to the information processing device 100. The connection port 910 may be, for example, a Universal Serial Bus (USB) port, an IEEE 1394 port, or a Small Computer System Interface (SCSI) port. The connection port 910 may also be an RS-232C port, an optical audio terminal, or a High-Definition Multimedia Interface (HDMI) (registered trademark) port. By connecting the connection port 910 to an external device, various types of data can be transmitted and received between the information processing device 100 and the external device.

[0107] The communication device 911 is, for example, a communication interface configured with a communication device for connecting to the communication network 920. The communication device 911 may be, for example, a communication card for a wired or wireless LAN (Local Area Network), Wi-Fi (registered trademark), Bluetooth (registered trademark), or WUSB (Wireless USB). The communication device 911 may also be a router for optical communication, a router for ADSL (Asymmetric Digital Subscriber Line), or a modem for various types of communication.

[0108] The communication device 911 can transmit and receive signals, for example, via the Internet or other communication devices using a predetermined protocol such as TCP / IP. The communication network 920 connected to the communication device 911 is a wired or wireless network, and may be, for example, an Internet communication network, a home LAN, an infrared communication network, a radio wave communication network, or a satellite communication network.

[0109] It is also possible to create a program that causes hardware such as the CPU 901, ROM 902, and RAM 903 built into a computer to perform functions equivalent to those of the information processing device 100. It is also possible to provide a computer-readable recording medium on which the program is recorded.

[0110] Although the preferred embodiments of the present disclosure have been described in detail above with reference to the accompanying drawings, the technical scope of the present disclosure is not limited to such examples. It is clear that a person skilled in the art of the present disclosure can conceive of various modified or altered examples within the scope of the technical idea described in the claims, and it is understood that these also naturally fall within the technical scope of the present disclosure.

[0111] Furthermore, the effects described herein are merely descriptive or exemplary and are not limiting. In other words, the technology according to the present disclosure may achieve other effects that will be apparent to those skilled in the art from the description of this specification, in addition to or in place of the above-described effects.

[0112] Note that the following configurations also fall within the technical scope of the present disclosure. (1) An information processing device comprising: a control primitive that controls a functional unit of a robot; an instruction acquisition unit that acquires an operation instruction configured by combining a subgoal input to the control primitive with an output primitive that outputs the subgoal; and an output generation unit that generates an output for controlling the functional unit of the robot based on the operation instruction. (2) The information processing device according to (1), wherein the operation instruction is configured by combining, before the control primitive, at least the subgoal input to the control primitive with the output primitive that outputs the subgoal. (3) The information processing device according to (2), wherein the output primitive is a planning primitive that plans a trajectory of the functional unit based on the input subgoal, and wherein the operation instruction is configured by further combining, before the planning primitive, the subgoal input to the planning primitive with a setting primitive that sets the subgoal. (4) The information processing device according to any one of (1) to (3), further comprising an image generation unit that generates block-shaped images indicating the control primitive, the subgoal, and the output primitive, wherein the image generation unit generates an image that distinguishes whether the control primitive, the subgoal, and the output primitive can be combined with each other by a fitting shape of a combining portion of the block shape. (5) The information processing device according to any one of (1) to (4), wherein the control primitive and the output primitive are coded in a programming language. (6) The information processing device according to any one of (1) to (5), wherein the operation instruction is created based on a directed graph that indicates a combining relationship between the control primitive, the subgoal, and the output primitive. (7) The information processing device according to any one of (1) to (6), further comprising a log generation unit that generates an execution log that records start times and end times of the control primitive and the output primitive.(8) The information processing device according to (7), wherein, when recording the start time and end time of the control primitive, the log generation unit further records in the execution log the state of the robot and the state of the environment in which the robot is present at the time of recording. (9) The information processing device according to any one of (1) to (8), wherein the instruction acquisition unit acquires an action instruction that describes an action to be performed by the robot by combining a plurality of the operation instructions. (10) The information processing device according to (9), wherein the output generation unit generates an output for controlling another of the functional units of the robot based further on an environmental change that will occur in the future due to the functional unit controlled by the control primitive within the series of the action instructions. (11) The information processing device according to (9), wherein the output generation unit generates an output for controlling another of the functional units of the robot based further on an environmental change that will occur in the future due to the operation of the functional unit controlled by the control primitive within the series of the action instructions. (12) The information processing device according to any one of (1) to (11), wherein the functional unit is a manipulator, a moving mechanism, or a head equipped with various sensors that are provided to the robot. (13) An information processing method by a computer, comprising: acquiring an operation instruction configured by combining a control primitive that controls a functional unit of a robot, a subgoal that is input to the control primitive, and an output primitive that outputs the subgoal; and generating an output for controlling the functional unit of the robot based on the operation instruction.

[0113] DESCRIPTION OF SYMBOLS 1 Robot 10 Main body 11 Functional unit 11A Cart 11B Head 11C Manipulator 11D Hand 100 Information processing device 110 Input / output unit 120 Image generation unit 130 Instruction acquisition unit 140 Output generation unit 150 DB storage unit 160 Log generation unit 170 Log storage unit PR1 Control primitive PR2 Output primitive SG Subgoal

Claims

1. An information processing device comprising: a control primitive that controls a functional unit possessed by a robot; an instruction acquisition unit that acquires an operation instruction constituted by combining a subgoal input to the control primitive and an output primitive that outputs the subgoal; and an output generation unit that generates an output for controlling the functional unit of the robot based on the operation instruction.

2. The information processing device according to claim 1, wherein the action instruction is configured by combining, before the control primitive, at least the subgoal input to the control primitive and the output primitive that outputs the subgoal.

3. The information processing device according to claim 2, wherein the output primitive is a planning primitive that plans the trajectory of the functional unit based on an input subgoal, and the operation instruction is configured by further combining, before the planning primitive, the subgoal input to the planning primitive with a setting primitive that sets the subgoal.

4. An information processing device as described in claim 1, further comprising an image generation unit that generates block-shaped images indicating the control primitives, the subgoals, and the output primitives, wherein the image generation unit generates images that distinguish whether the control primitives, the subgoals, and the output primitives can be combined with each other by the fitting shape of the combining parts of the block shapes.

5. The information processing device according to claim 1, wherein the control primitives and the output primitives are coded in a programming language.

6. The information processing device according to claim 1, wherein the action instruction is created based on a directed graph that indicates the connection relationships between the control primitives, the subgoals, and the output primitives.

7. The information processing device according to claim 1, further comprising a log generating unit that generates an execution log in which start times and end times of said control primitives and said output primitives are recorded.

8. An information processing device as described in claim 7, wherein when the log generation unit records the start time and end time of the control primitive, it further records in the execution log the state of the robot at the time of recording and the state of the environment in which the robot exists.

9. The information processing device according to claim 1, wherein the instruction acquisition unit acquires an action instruction that describes an action to be performed by the robot by combining a plurality of the operation instructions.

10. An information processing device as described in claim 9, wherein the output generation unit generates an output for controlling other functional units of the robot based further on future environmental changes that will occur due to the functional units controlled by the control primitives within the series of behavioral instructions.

11. An information processing device as described in claim 9, wherein the output generation unit generates an output for controlling other functional units of the robot based further on environmental changes caused by the operation of the functional unit controlled by the control primitive within the series of behavioral instructions.

12. The information processing device according to claim 1, wherein the functional unit is a manipulator, a movement mechanism, or a head equipped with various sensors that the robot has.

13. An information processing method by a computer, comprising: obtaining an operation instruction constituted by combining a control primitive that controls a functional unit possessed by a robot, a subgoal that is input to the control primitive, and an output primitive that outputs the subgoal; and generating an output for controlling the functional unit of the robot based on the operation instruction.

Images