Information processing device and information processing method

Abstract

[Problem] Describing robot operations by combining granular units with higher versatility. [Solution] The information processing apparatus includes: an instruction acquisition unit for acquiring motion instructions composed of a control primitive for controlling a functional unit of the robot, sub-targets input to the control primitives, and output primitives for outputting the sub-targets; and an output generation unit for generating outputs for controlling the functional unit of the robot based on the motion instructions.

Description

Technical Field

[0001] This disclosure relates to information processing apparatus and information processing methods. Background Technology

[0002] In recent years, pet-type robots that mimic quadrupedal animals or humanoid robots that can walk upright on two legs have been increasingly put into practical application in the service sector.

[0003] Such service robots need to perform various movements in unknown environments. Therefore, it is expected that service robots are not prepared with dedicated movement patterns tailored to each environment, but rather that they describe various movements by combining general-purpose components.

[0004] For example, the creation tool disclosed in Patent Document 1 below enables users to create robot motion patterns by connecting boxes that define the robot's motion states to each other. The processing and motion performed in each box are described, for example, using a programming language specifically developed for robot control.

[0005] Reference List

[0006] Patent documents

[0007] Patent Document 1: Japanese Patent Application Publication No. 2001-353678 Summary of the Invention

[0008] [The problem this invention aims to solve]

[0009] However, in the creation tool disclosed in Patent Document 1, the processing and motion performed in each frame are described based on each action, such as "assuming a certain pose," "performing a certain movement," or "stopping an ongoing movement." Therefore, robots with different executable actions require their own specific programming languages.

[0010] Therefore, this disclosure proposes a novel and improved information processing apparatus and method that enables the description of robot motion by combining more general, high-granularity units.

[0011] [Solution to the problem]

[0012] According to this disclosure, an information processing apparatus is provided, comprising: an instruction acquisition unit configured to acquire motion instructions, the motion instructions being configured by coupling the following: a control primitive for controlling a functional unit provided in a robot, a sub-target input to the control primitive, and an output primitive for outputting the sub-target; and an output generation unit configured to generate an output for controlling the functional unit of the robot based on the motion instructions.

[0013] Furthermore, according to this disclosure, a computer-implemented information processing method is provided, the method comprising: acquiring motion instructions, the motion instructions being configured by coupling the following: a control primitive controlling a functional unit set in a robot, a sub-target input to the control primitive, and an output primitive outputting the sub-target; and generating an output for controlling the functional unit of the robot based on the motion instructions. Attached Figure Description

[0014] [ Figure 1 [This is a schematic diagram illustrating an example of a robot whose motion is described by an information processing apparatus according to an embodiment of the present disclosure.]

[0015] [ Figure 2 [This is a schematic diagram illustrating another example of a robot whose motion is described by an information processing device according to an embodiment.]

[0016] [ Figure 3 [This is a schematic diagram illustrating another example of a robot whose motion is described by an information processing device according to an embodiment.]

[0017] [ Figure 4 [This is a schematic diagram illustrating another example of a robot whose motion is described by an information processing device according to an embodiment.]

[0018] [ Figure 5 [Illustration 1] is a block diagram illustrating the functional configuration of an information processing apparatus according to an embodiment.

[0019] [ Figure 6 [Illustration] is an explanatory diagram used to describe the relationship between the functional units of a robot and the control primitives.

[0020] [ Figure 7 [] is an explanatory diagram showing examples of control primitives, sub-goals, and output primitives.

[0021] [ Figure 8A [This is an illustrative diagram showing an example of describing the "pick-up" action of grasping and lifting an object by combining sub-objects, primitives, and motions in a hierarchical manner.]

[0022] [ Figure 8B [This is an illustration diagram showing an example of a "placement" action that describes placing a grasped object on a floor or similar surface by combining sub-objectives, primitives, and actions in a hierarchical manner.]

[0023] [ Figure 9 [Illustration] is an illustration showing an image used to generate motion commands for a robot by combining primitives and sub-objectives.

[0024] [ Figure 10[Illustration] is an illustration showing an image used to generate motion commands for a robot by combining the generated motions.

[0025] [ Figure 11 [] is an explanatory diagram showing the correspondence between the execution of primitives and the execution logs generated by the log generation unit.

[0026] [ Figure 12 [ ] is an illustrative diagram showing the correspondence between the execution of primitives used to obtain information about the environment and the environment logs generated by the log generation unit.

[0027] [ Figure 13 [Illustration] is a conceptual illustration of a directed graph generated based on the input and output relationships between primitives.

[0028] [ Figure 14A [This is a 3D view schematically illustrating a scene where a robot performs the action of placing a target object on the floor.]

[0029] [ Figure 14B ]yes Figure 14A A top-down view of the scene shown.

[0030] [ Figure 15 [This is a stereoscopic view schematically illustrating a scene where a robot performs the action of lifting a target object from the floor.]

[0031] [ Figure 16 [ ] is a block diagram illustrating an example of the hardware configuration of an information processing apparatus according to an embodiment. Detailed Implementation

[0032] In the following, preferred embodiments of the present disclosure will be described in detail with reference to the accompanying drawings. Note that components having substantially the same functional configuration are indicated by the same reference numerals in this document and in the drawings, and therefore redundant descriptions are omitted.

[0033] Note that the descriptions will be given in the following order.

[0034] 1. Overview of Information Processing Devices

[0035] 2. Configuration of information processing devices

[0036] 3. Modification

[0037] 3.1. First Revision

[0038] 3.2. Second Revision

[0039] 3.3. Third Revision

[0040] 4. Hardware Configuration Example

[0041] <1. Overview of Information Processing Devices>

[0042] First, refer to Figures 1 to 4 This document provides an overview of an information processing apparatus according to embodiments of the present disclosure. Figure 1 This is a schematic diagram illustrating an example of a robot whose motion is described by an information processing apparatus according to this embodiment. Figures 2 to 4 This is a schematic diagram illustrating another example of a robot whose motion is described by an information processing apparatus according to this embodiment.

[0043] like Figure 1 As shown, the information processing apparatus according to this embodiment is used, for example, to describe the motion of a robot 1 having multiple functional units. For example, robot 1 is a mobile robotic arm robot including a main body 10 and multiple functional units such as a mobile platform 11A, a head 11B, a robotic arm 11C, and a hand 11D.

[0044] The main body 10 is the main part of the robot 1, and the mobile platform 11A, head 11B, and robotic arm 11C are attached to the main body 10. The main body 10 is also equipped with control devices, power supply devices, communication devices, etc.

[0045] The mobile platform 11A is a support portion of the main body 10 and includes a movement mechanism capable of moving the robot 1 to a desired position. The mobile platform 11A may include, for example, a wheel-based movement mechanism. Alternatively, the mobile platform 11A may include a leg-based movement mechanism, a leg-wheel hybrid movement mechanism, or a track-based movement mechanism.

[0046] The head 11B is disposed on the upper part of the body 10 and includes, for example, an imaging device for capturing images corresponding to the field of view of the robot 1. The head 11B may also include various sensors, including light detection and ranging (LiDAR) sensors or radio detection and ranging (Radar) sensors, and may also include a display unit or audio output unit for communicating with a user.

[0047] The robotic arm 11C is an arm with a structure that connects multiple links through at least one joint. A hand 11D capable of grasping a target object is provided at the distal end of the robotic arm 11C. The robotic arm 11C can be a single robotic arm, or it can be a pair of robotic arms on the left and right sides.

[0048] The hand 11D is an end effector located at the distal end of the robotic arm 11C and includes various sensors for easy and smooth grasping of a target object. The hand 11D may be, for example, a two- or three-finger gripper, a four- or five-finger hand, or a suction gripper. The hand 11D may include a proximity sensor for measuring the distance to the target object, and may also include a pressure sensor for detecting the force applied to the target object.

[0049] The corresponding movements of the functional units of robot 1—namely, the mobile platform 11A, head 11B, robotic arm 11C, and hand 11D—enable the overall movement of robot 1. Therefore, according to this embodiment, the information processing apparatus can describe the movement of robot 1 as a combination of the movements of functional units by using control units (control primitives) that control these functional units, target values ​​(sub-targets) input to the control units, and output units (output primitives) that output target values ​​as basic units. This configuration allows the information processing apparatus according to this embodiment to reuse the control primitives, sub-targets, and output primitives of common functional units when describing the movement of other robots having common functional units.

[0050] For example, the information processing apparatus according to this embodiment can be reused for describing Figure 1 The motion of robot 1 is described by the control primitives, sub-goals, and output primitives shown. Figures 2 to 4 The movements of various robots are shown.

[0051] Figure 2 The robot 2 shown is a mobile robotic arm robot including a main body 10, a mobile platform 11A, a head 11B, and a robotic arm 11C. The distal end of the robotic arm 11C is configured as a sensorless fixed hook or hand. The main body 10, mobile platform 11A, head 11B, and robotic arm 11C of robot 2 are substantially the same as those of robot 1. Therefore, the information processing apparatus according to this embodiment can describe the motion of robot 2 by repeatedly using control primitives, sub-objectives, and output primitives used to describe the motion of the mobile platform 11A, head 11B, and robotic arm 11C of robot 1.

[0052] Figure 3 The robot 3 shown is an interactive robot including a body 10, a mobile platform 11A, and a head 11B, and is capable of communicating with a user, such as through dialogue. The body 10, mobile platform 11A, and head 11B of robot 3 are substantially the same as those of robot 1. Therefore, the information processing apparatus according to this embodiment can describe the motion of robot 3 by repeatedly using control primitives, sub-targets, and output primitives used to describe the motion of the mobile platform 11A and head 11B of robot 1.

[0053] Figure 4The robot 4 shown is a transport robot that includes a mobile platform 11A and transports target objects loaded on it. The mobile platform 11A of robot 4 is substantially the same as that of robot 1. Therefore, the information processing apparatus according to this embodiment can describe the motion of robot 4 by repeatedly using control primitives, sub-targets, and output primitives used to describe the motion of the mobile platform 11A of robot 1.

[0054] In other words, the information processing apparatus according to this embodiment can uniquely determine the basic unit for describing motion based on the functional units provided in the robot. Therefore, the information processing apparatus according to this embodiment can systematically describe the motion of a robot having various functional units based on each functional unit.

[0055] <2. Configuration of Information Processing Device>

[0056] Next, we will refer to Figures 5 to 12 The configuration of the information processing apparatus 100 according to this embodiment is described.

[0057] Figure 5 This is a block diagram illustrating the functional configuration of the information processing apparatus 100 according to this embodiment. For example... Figure 5 As shown, the information processing device 100 includes an input / output unit 110, an image generation unit 120, an instruction acquisition unit 130, an output generation unit 140, a database storage unit 150, a log generation unit 160, and a log storage unit 170.

[0058] The input / output unit 110 is an input interface or output interface used to exchange information with users or developers.

[0059] Examples of input interfaces included in the input / output unit 110 include a mouse, keyboard, touch panel, button, switch, joystick, etc. The input / output unit 110 can output motion commands, input by a user or developer via these input interfaces, to the instruction acquisition unit 130 to instruct the robot 1 to perform actions or movements.

[0060] Examples of output interfaces included in the input / output unit 110 include display devices such as liquid crystal displays (LCDs), plasma display panels (PDPs), organic light-emitting diode (OLED) displays, holograms, and projectors. The input / output unit 110 can present images generated by the image generation unit 120 to users or developers via these output interfaces.

[0061] Note that the input / output unit 110 may be a connection interface configured to connect to the aforementioned external input interface or output interface.

[0062] The instruction acquisition unit 130 acquires motion instructions that are input by a user or developer via the input / output unit 110, instructing the robot 1 to perform actions or movements. The motion instructions acquired by the instruction acquisition unit 130 are configured by at least coupling control primitives that control functional units set in the robot 1, sub-objectives input to the control primitives, and output primitives that output the sub-objectives.

[0063] Reference Figure 6 and Figure 7 Describe the control primitives, sub-goals, and output primitives. Figure 6 This is an explanatory diagram used to describe the relationship between the functional unit 11 of robot 1 and the control primitive PR1. Figure 7 This is an illustrative diagram used to describe examples of the control primitive PR1, sub-goal SG, and output primitive PR2.

[0064] like Figure 6 As shown, assume that robot 1 has a functional unit 11 including a mobile platform 11A, a head 11B, a robotic arm 11C, and a hand 11D.

[0065] For each functional unit 11, a control primitive PR1 is set to control the motion of the functional unit 11. The control primitive PR1 is the basic unit of the module that controls the motion of the functional unit 11, and controls the functional unit 11 to achieve the sub-objective SG, which is the input target value. For example, the control primitive PR1 is created by the developer by coding using a programming language.

[0066] For example, as the control primitive PR1 for controlling the movement of the mobile platform 11A, a control primitive for "mobile platform movement" is set. Furthermore, as the control primitive PR1 for controlling the movement of the proximal end of the robotic arm 11C and the head 11B, a control primitive for "joint trajectory tracking" is set. Additionally, as the control primitive PR1 for controlling the movement of the distal end of the robotic arm 11C, a control primitive for "end-effector trajectory tracking" is set.

[0067] However, some control primitives PR1 may not input sub-target SG as the target value. For example, the control primitive for "grasping / releasing" that controls the opening and closing movement of hand 11D is a binary control between the grasping state and the release state; therefore, no sub-target SG is input as the target value. Furthermore, the control primitive for "proximity-based servo" that controls the approach of hand 11D towards a target object performs control based on feedback from a proximity sensor installed in hand 11D; therefore, no sub-target SG is input as the target value.

[0068] like Figure 7As shown, the sub-target SG is set to the target value input to the control primitive PR1. Specifically, the sub-target SG of "Mobile Platform Movement (Target)" is input to the control primitive of "Mobile Platform Movement". Specifically, the sub-target SG of "Joint Trajectory (Target)" is input to the control primitive of "Joint Trajectory Tracking". Specifically, the sub-target SG of "End effector trajectory (Target)" is input to the control primitive of "End effector trajectory tracking".

[0069] Furthermore, for each corresponding subtarget SG, an output primitive PR2 is set to output the subtarget SG. The output primitive PR2 is the basic unit of the module that outputs the subtarget SG, and outputs the subtarget SG based, for example, on other input subtarget SGs.

[0070] exist Figure 6 and Figure 7 In this diagram, the form of the subtarget SG input to the control primitive PR1 and the form of the subtarget SG output from the output primitive PR2 are represented by recesses and protrusions provided on the control primitive PR1 and the output primitive PR2, respectively. That is, the subtarget SG input to the control primitive PR1 has recesses and protrusions on its lower side that engage with the recesses and protrusions provided on the upper side of the control primitive PR1. Similarly, the subtarget SG output from the output primitive PR2 has recesses and protrusions on its upper side that engage with the recesses and protrusions provided on the lower side of the output primitive PR2. Therefore, the control primitive PR1 and the output primitive PR2 can be connected via their subtarget SGs that engage with each other through their recesses and protrusions.

[0071] As an example of the output primitive PR2 of the sub-target SG based on another input, a planning primitive is shown for outputting the target value (sub-target SG) to be achieved by each functional unit 11 based on the input of the sub-target SG.

[0072] For example, the output primitive of "Mobile Platform Positioning Planning" plans the position of mobile platform 11A based on the sub-target of the input "Object Name" to output the sub-target of "Mobile Platform Position (Target)". The output primitive of "Joint Space Planning" plans the trajectory of robotic arm 11C based on the input "Joint Angle (Target)" to output the sub-target of "Joint Trajectory (Target)". The output primitive of "Gaze Planning" plans the trajectory of head 11B based on the input sub-target of "Gaze Target" to output the sub-target of "Joint Trajectory (Target)". The output primitive of "Task Space Planning" plans the trajectory of hand 11D based on the input sub-target of "Hand Target" to output the sub-target of "Joint Trajectory (Target)".

[0073] As another example of the output primitive PR2 of the subtarget SG based on another input, a target setting primitive is shown for setting different target values ​​(subtarget SG) based on the input of the subtarget SG.

[0074] For example, the output primitive "Gaze Target Setting" sets the target the gaze is pointing at based on the sub-target of the input "Object Name" to output the sub-target of "Gaze Target". The output primitive "Hand Target Setting" sets the target the hand will grasp based on the sub-target of the input "Object Name" to output the sub-target of "Hand Target".

[0075] Furthermore, the output primitive PR2 can also output a sub-target SG without inputting another sub-target SG. For example, a setting primitive for setting the motion target of robot 1 is shown as an example of such an output primitive PR2.

[0076] For example, the output primitive for "Object Settings" outputs a sub-target for "Object Name". The output primitive for "Movement Target Settings" outputs a sub-target for "Movement Platform Position (Target)". The output primitive for "Task Space Target Settings" outputs a sub-target for "End effector trajectory (Target)". Furthermore, the output primitive for "Pose Reading" outputs a sub-target for "Joint Angle (Target)".

[0077] By combining the control primitive PR1, sub-objective SG, and output primitive PR2 described above, the user can use a certain number of combinations of sub-objectives and primitives to describe the motion instructions of robot 1. Furthermore, the user can further combine motions to describe actions comprising multiple movements in a hierarchical manner.

[0078] In this embodiment, firstly, a control primitive PR1 is set to control the motion of each functional unit 11 installed in the robot 1. Next, an output primitive PR2, whose output is to be input to the control primitive PR1, is set as a setting primitive, a target setting primitive, or a planning primitive. Subsequently, output primitives for the sub-target SGs whose output is to be input to these output primitives PR2 are set. By repeatedly performing such primitive settings until there is no more input or until an input is provided from an external source, the basic units—primitives—used to describe the motion of the robot 1 based on the sub-target SGs are set. This configuration enables the information processing device 100 to systematically set primitives for describing the motion of the robot 1.

[0079] Reference Figure 8A and Figure 8B To describe the hierarchical description of the motion instructions for robot 1. Figure 8A This is an illustrative diagram showing an example of how the "pick-up" action of grasping and lifting an object is described by combining sub-objects, primitives, and motions in a hierarchical manner. Figure 8B This is an illustrative diagram showing an example of a "placement" action that describes placing a grasped object on a floor or similar surface by combining sub-objectives, primitives, and motions in a hierarchical manner.

[0080] like Figure 8A and Figure 8B As shown, each motion of robot 1 is described by using control primitive PR1 as a basis, and then sequentially coupling output primitive PR2 via sub-target SG before control primitive PR1. Furthermore, the motion commands of robot 1 are described by combining the motions described by coupling control primitive PR1, sub-target SG, and output primitive PR2.

[0081] For example, such as Figure 8A As shown, the motion of "moving the mobile platform closer to the object" is described by coupling the setting primitive of "object settings", the sub-target of "object name", the planning primitive of "mobile platform positioning planning", the sub-target of "mobile platform position (target)", and the control primitive of "mobile platform movement". The motion of "looking at the object" is described by coupling the setting primitive of "object settings", the sub-target of "object name", the target setting primitive of "view target settings", the sub-target of "gaze target (target)", the planning primitive of "gaze planning", the sub-target of "joint trajectory (target)", and the control primitive of "joint trajectory tracking". The motion of "raising a hand" is described by coupling the target setting primitive of "task space target settings", the sub-target of "end effector trajectory (target)", and the control primitive of "end effector trajectory tracking". The motion of "bringing the hand closer to the gripping position" is described by coupling the planning primitives for "grip planning," the target setting primitives for "hand position fine-tuning," the sub-targets for "end-effector trajectory (target)," the control primitives for "end-effector trajectory tracking," and the control primitives for "proximity-based servoing." The motion of "returning to the default posture" is described by coupling the setting primitives for "pose reading," the sub-targets for "joint angle (target)," the planning primitives for "joint space planning," the sub-targets for "joint trajectory (target)," and the control primitives for "joint trajectory tracking."

[0082] Furthermore, the motion of "bringing the hand closer to the object" is described by coupling the motion of "bringing the mobile platform closer to the object" with the motion of "looking at the object." The motion of "grasping the object" is described by coupling the control primitive of "grasping / releasing" with the motion of "raising the hand."

[0083] Furthermore, the "pickup" action of grasping and lifting an object is described by coupling the motion of "bringing the hand closer to the object", the motion of "bringing the hand closer to the gripping position", the motion of "grasping the object", and the motion of "returning to the default posture".

[0084] like Figure 8B As shown, by using a method similar to Figure 8A The motion of "bringing the mobile platform closer to the object" is described by coupling the setting primitive of "object settings", the sub-target of "object name", the planning primitive of "mobile platform positioning planning", the sub-target of "mobile platform location (target)", and the control primitive of "mobile platform movement" in a similar manner. Figure 8A The motion of "looking at an object" is described by coupling the setting primitives of "object settings," the sub-goal of "object name," the target setting primitive of "gaze target settings," the sub-goal of "gaze target (goal)," the planning primitive of "gaze planning," the sub-goal of "joint trajectory (goal)," and the control primitive of "joint trajectory tracking." The motion of "lowering the hand" is described by coupling the target setting primitives of "task space target settings," the sub-goal of "end-effector trajectory (goal)," and the control primitive of "end-effector trajectory tracking." This is done in a manner similar to... Figure 8A The motion of "returning to the default posture" is described by coupling the setting primitive of "pose reading", the sub-objective of "joint angle (target)", the planning primitive of "joint space planning", the sub-objective of "joint trajectory (target)" and the control primitive of "joint trajectory tracking".

[0085] Furthermore, by using a method similar to Figure 8A The motion of "bringing the hand closer to the object" is described by coupling the motion of "bringing the mobile platform closer to the object" with the motion of "looking at the object." The motion of "releasing the object" is described by coupling the control primitive of "grasping / releasing" with the motion of "putting down the hand."

[0086] Furthermore, the "placement" action of placing a grasping object is described by coupling the motion of "bringing the hand closer to the object", the motion of "releasing the object", and the motion of "returning to the default posture".

[0087] In this embodiment, the use of a description based on the control primitive PR1, the sub-objective SG, and the output primitive PR2 enables the description of various motions or actions by reusing common primitives, sub-objectives, and motions.

[0088] In this embodiment, the motion or action of robot 1 is described based on the sub-target SG input and output between control primitive PR1 and output primitive PR2. This configuration allows the data input and output between control primitive PR1 and output primitive PR2, which serve as the basic units for describing motion, to become clear, thereby preventing the formation of dependencies between primitives. Therefore, in this embodiment, control primitive PR1 and output primitive PR2 can be set at a more general granular level.

[0089] The image generation unit 120 generates images for users or developers to generate motion commands instructing the robot 1 to perform actions or movements. The images generated by the image generation unit 120 are presented to the user or developer via the input / output unit 110.

[0090] Reference Figure 9 and Figure 10 Describes the image generated by the image generation unit 120. Figure 9This is an explanatory diagram showing an image used to generate motion commands for robot 1 by combining primitives and sub-objectives. Figure 10 This is an explanatory diagram showing an image used to generate motion commands for robot 1 by combining generated motions.

[0091] like Figure 9 As shown, users or developers can create motions to be performed by robot 1 by referring to list image M1 and motion creation image M2 generated by image generation unit 120.

[0092] In list image M1, various primitives and sub-goals created by the developer, as well as various movements created by the user or developer, are displayed in list form. The primitives displayed in list image M1 are created by the developer using a programming language, for example, as shown in balloon image P1. Balloon image P1 can be hidden and can be configured to be displayed when the user or developer selects a primitive. Furthermore, the user can toggle the visibility of balloon image P1. For example, various primitives can be created in a separate window (not shown).

[0093] In motion creation image M2, primitives and sub-objects selected by the user or developer from various primitives and sub-objects in list image M1 are displayed. The user or developer can visually create the motion of robot 1 by vertically coupling the primitives and sub-objects selected from list image M1 onto motion creation image M2. The created motion of robot 1 is stored in DB storage unit 150, described later, according to save instructions issued by the user or developer, and then added to the list of various motions in list image M1.

[0094] In the motion creation image M2, various primitives and sub-targets are displayed as block images. The block image representing each primitive has recesses and protrusions at the top and bottom, which conform to the form of a sub-target input to or output from the primitive. Similarly, the block image representing each sub-target has recesses and protrusions at the top and bottom, which are configured to interlock with the recesses and protrusions of the primitive from which the sub-target is input or output. This configuration allows users or developers to visually align the input and output between primitives and sub-targets by coupling primitives and sub-targets with interlocking recesses and protrusions.

[0095] like Figure 10 As shown, users or developers can create motion instructions that indicate actions to be performed by robot 1 by referring to the list image M3 and motion creation image M4 generated by image generation unit 120.

[0096] In list image M3, various motions and actions created by the user or developer are displayed in list form. The various motions displayed in list image M3 are, for example, motions created by the user or developer by coupling primitives and sub-objects in motion creation image M2.

[0097] In the motion creation image M4, the motion selected by the user or developer from various motions in the list image M3 is displayed. The user or developer can visually create motion instructions for robot 1 by vertically coupling the motion selected from list image M3 onto motion creation image M4. These motion instructions include multiple motion commands. The created motions of robot 1 are stored in the DB storage unit 150, described later, according to a save command issued by the user or developer, and then added to the list of various motions in list image M3.

[0098] DB storage unit 150 stores primitives and motions created by users or developers in a separate database. Specifically, DB storage unit 150 can store a database that registers various primitives created by developers using a programming language. Furthermore, DB storage unit 150 can store a database that registers various motions created by users or developers by referencing the aforementioned motion to create image M2. DB storage unit 150 may include, for example, magnetic storage devices such as hard disk drives (HDDs), semiconductor storage devices, optical storage devices, magneto-optical storage devices, etc.

[0099] The output generation unit 140 generates outputs for controlling the robot 1 based on motion commands acquired by the instruction acquisition unit 130. Specifically, the output generation unit 140 can generate outputs for controlling each functional unit 11 in the robot 1 based on primitives and motions included in the acquired motion commands. For example, the output generation unit 140 can generate outputs for controlling each functional unit 11 by converting the content of each primitive into a format that each functional unit 11 can understand, based on various databases stored in the DB storage unit 150.

[0100] The log generation unit 160 generates execution logs for various primitives included in the motion instructions. Specifically, the log generation unit 160 can generate execution logs that record the start and end times of each primitive included in the motion instructions in chronological order. The execution logs generated by the log generation unit 160 are stored in the log storage unit 170, which will be described later.

[0101] Reference Figure 11 A specific example of the execution log generated by the log generation unit 160 is described. Figure 11This is an explanatory diagram showing the correspondence between the execution of primitives and the execution logs generated by the log generation unit 160.

[0102] like Figure 11 As shown, assume that the four primitives "SeEeTarget", "PlanEePose", "FollowJointTrajectory", and "SetEeTarget" are executed sequentially in a series of action instructions. Among them, "SeEeTarget" and "PlanEePose" are output primitives (setting primitives and planning primitives), and "FollowJointTrajectory" is a control primitive.

[0103] In this case, the log generation unit 160 can generate an execution log that includes the name of the executed primitive, the start (tick) time, the end (success) time, and the event label. Furthermore, for control primitives but not for output primitives, the log generation unit 160 can also store the state of robot 1 at each start and end time in the execution log. For example, in Figure 11 In the execution log shown, the start and end times of the "joint state", "odometer" and "estimated position" of the control primitive "FollowJointTrajectory" are also stored as the state of robot 1.

[0104] In this embodiment, the timing of the state change of robot 1 is explicitly defined as the execution time of each control primitive. This allows the log generation unit 160 to store the state of robot 1 in the execution log at the beginning and end of each control primitive. Therefore, the log generation unit 160 can reduce the amount of data in the execution log storing the state changes of robot 1 over time.

[0105] Furthermore, the log generation unit 160 can generate an environmental log that stores the state of the environment in which the robot 1 is located. Specifically, the log generation unit 160 can generate an environmental log that stores environmental information obtained from sensing results detected by various sensors installed in the robot 1. The environmental log generated by the log generation unit 160 is stored in the log storage unit 170, which will be described later.

[0106] Reference Figure 12 A specific example of the environment logs generated by the log generation unit 160 is described. Figure 12 This is an explanatory diagram showing the correspondence between the execution of primitives that obtain information about the environment and the environment logs generated by the log generation unit 160.

[0107] like Figure 12As shown, in the "SetObject" primitive, the position of an object in the environment is obtained from the sensing results detected by sensors installed in robot 1. The log generation unit 160 can store the position of the object in the environment and the time of obtaining the object's position in the environment log each time the "SetObject" primitive is executed. This configuration allows the information processing device 100 to easily reproduce the past state of the environment in which robot 1 is located by tracing the environment log stored in the log storage unit 170 in chronological order.

[0108] Log storage unit 170 stores the execution logs and environment logs generated by log generation unit 160. For example, the execution logs and environment logs stored in log storage unit 170 can be shared with other robots. The execution logs and environment logs stored in log storage unit 170 can be used to monitor the state of robot 1 and changes in the environment in which robot 1 is located over time. Log storage unit 170 may include, for example, magnetic storage devices such as hard disk drives (HDDs), semiconductor storage devices, optical storage devices, magneto-optical storage devices, etc.

[0109] In the information processing apparatus 100 according to this embodiment, motion instructions instructing the movement of the robot 1 are described based on sub-targets input and output between primitives. This configuration allows the information processing apparatus 100 to set primitives and sub-targets at a more general granular level by eliminating dependencies between primitives, which serve as basic units for describing motion. Therefore, the information processing apparatus 100 can describe the motion of the robot 1, which has various functional units, in a more general format.

[0110] <3. Revision>

[0111] (3.1. First revision)

[0112] Next, refer to Figure 13 The first modification of this embodiment will be described below. The first modification of this embodiment is to automatically or semi-automatically generate motion commands for robot 1 using a directed graph that represents the coupling relationship between primitives and sub-objectives.

[0113] Figure 13 This is a conceptual illustration of a directed graph 50 generated based on the input and output relationships between primitives. Figure 13 In the directed graph 50 shown, primitives are represented as rectangles, and sub-goals are represented as octagons. Furthermore, the input and output relationships between primitives and their sub-goals are represented as arrows.

[0114] like Figure 13As shown, a directed graph 50 is formed by starting with the control primitive PR1 and sequentially traversing the sub-goals SG that are input to the control primitive PR1, as well as the output primitive PR2 that is input to and outputs the sub-goals SG, based on the reachability matrix. For example, the directed graph 50 is formed by calculating the primitives reachable from the control primitive PR1 via the sub-goals SG based on the reachability matrix, and then sequentially calculating other reachable primitives from the calculated reachable primitives based on the reachability matrix until the end is reached.

[0115] With the directed graph 50 formed, the information processing device 100 can automatically or semi-automatically generate motion commands for the robot 1 by traversing the directed graph 50 backward from the desired control primitive PR1. Specifically, the information processing device 100 can automatically generate motion commands by specifying the control primitive PR1 that outputs the desired control to the robot 1, as well as the intermediate output primitive PR2 or the sub-target SG, and coupling the specified control primitive PR1, output primitive PR2, and sub-target SG.

[0116] According to the first modification, the information processing device 100 can assist users or developers in creating motion instructions for the robot 1.

[0117] (3.2. Second revision)

[0118] Reference Figure 14A and Figure 14B The second modification of this embodiment will now be described. The second modification of this embodiment is as follows: The movement of robot 1 is controlled based on future environmental changes within a series of action commands.

[0119] Figure 14A It is a perspective view schematically showing a scene in which robot 1 performs the action of placing target object 20 on floor G. Figure 14B yes Figure 14A A top-down view of the scene shown. (As shown) Figure 14A and Figure 14B As shown, during the "placement" action where the target object 20, grasped by robot 1, is placed onto floor G, the environment of floor G changes due to the placement of the target object 20. Specifically, the "placement" action adds the target object 20 as a new obstacle to floor G.

[0120] On the other hand, during the movement of the mobile platform included in the "placement" action, the target object 20 is not present on the floor G when the movement position of the mobile platform 11A is planned; therefore, the information processing device 100 has difficulty planning the movement position of the mobile platform 11A while considering the target object 20. In such a case, the planned movement position 11N of the mobile platform 11A may overlap with the placement position NP of the target object 20, causing interference to the mobile platform 11A during the placement of the target object 20 and potentially leading to the failure of the placement of the target object 20.

[0121] In the second modification, the planned placement position NP of the target object 20 is obtained in the output primitive of the "mobile platform scheduling plan" that plans the movement position of the mobile platform 11A, and the movement position of the mobile platform 11A is planned by assuming that the target object 20 exists at the obtained placement position NP. This configuration enables the information processing device 100 to plan the movement position 11N of the mobile platform 11A during the placement of the target object 20, thereby avoiding interference between the movement position 11N of the mobile platform 11A and the placement position NP of the target object 20. Note that the planned placement position NP of the target object 20 can be obtained, for example, from the trajectory planning of the robotic arm 11C used to grasp the target object 20, etc.

[0122] According to the second modification, the information processing device 100 can acquire future environmental changes within a series of action commands and perform motion planning while considering the acquired environmental changes. For example, the information processing device 100 can plan the movement position 11N of the robot 1's mobile platform 11A to avoid the placement position NP of the target object 20 placed by the robot 1. Therefore, the information processing device 100 can control the robot 1 more smoothly based on the acquired action commands.

[0123] (3.3. Third revision)

[0124] Reference Figure 15 The third modification of this embodiment will now be described. The third modification of this embodiment is as follows: the movement of robot 1 is controlled based on observed environmental changes within a series of action commands.

[0125] Figure 15 This is a perspective view schematically illustrating a scene where robot 1 performs the action of lifting target object 20 from floor G. (Example) Figure 15 As shown, in the "pick-up" action performed by robot 1 to lift target object 20 from floor G, the environment of floor G changes due to the lifting of target object 20. Specifically, as a result of the "pick-up" action, target object 20 is removed from floor G.

[0126] On the other hand, in the "placement" action immediately following the "pickup" action, it is not recognized that the target object 20 has been removed from the floor G. Therefore, the information processing device 100 has difficulty planning the trajectory of the robotic arm 11C while considering that the target object 20 has been removed. In such a case, the trajectory of the robotic arm 11C is planned to avoid the placement position NP of the non-existent target object 20, which may lead to an excessively long trajectory or trajectory planning failure.

[0127] In the third modification, the placement position NP of the grasped and lifted target object 20 is obtained in the output primitive of "joint space planning" for planning the trajectory of the robotic arm 11C, and the trajectory of the robotic arm 11C is planned when the target object 20 is removed from the placement position NP. This configuration enables the information processing device 100 to prevent planning redundant trajectories that avoid the non-existent target object 20 while planning the trajectory of the robotic arm 11C.

[0128] According to the third modification, the information processing device 100 can acquire observed environmental changes within a series of motion commands and perform motion planning while considering the acquired environmental changes. For example, the information processing device 100 can plan the trajectory of the robotic arm 11C of robot 1 while considering that the target object 20 is lifted by robot 1. Therefore, the information processing device 100 can control robot 1 more smoothly based on the acquired motion commands.

[0129] <4. Hardware Configuration Example>

[0130] In addition, it will refer to Figure 16 The hardware configuration of the information processing apparatus 100 according to this embodiment is described. Figure 16 This is a block diagram illustrating an example of the hardware configuration of the information processing apparatus 100 according to this embodiment.

[0131] The functions of the information processing apparatus 100 according to this embodiment can be achieved through the cooperation between software and hardware as described below. The functions of the image generation unit 120, instruction acquisition unit 130, output generation unit 140, and log generation unit 160 can be implemented, for example, by a central processing unit (CPU) 901. The input function of the input / output unit 110 can be implemented, for example, by an input device 906, a connection port 910, or a communication device 911. The output function of the input / output unit 110 can be implemented, for example, by an output device 907, a connection port 910, or a communication device 911. The functions of the DB storage unit 150 and the log storage unit 170 can be implemented, for example, by a storage device 908.

[0132] like Figure 16As shown, the information processing device 100 includes a CPU 901, a read-only memory (ROM) 902, and a random access memory (RAM) 903.

[0133] In addition, the information processing device 100 may also include a host bus 904a, a bridge 904, an external bus 904b, an interface 905, an input device 906, an output device 907, a storage device 908, a driver 909, a connection port 910, or a communication device 911. Instead of CPU 901 or in addition to CPU 901, the information processing device 100 may include processing circuitry such as a digital signal processor (DSP) or an application-specific integrated circuit (ASIC).

[0134] The CPU 901 functions as an arithmetic processing device or a control device, and controls the operation of the information processing device 100 according to various programs recorded in a memory device such as ROM 902, RAM 903, or storage device 908, or on a removable recording medium attached to the driver 909. ROM 902 stores programs used by the CPU 901, calculation parameters, etc. RAM 903 temporarily stores programs used by the CPU 901 during execution, parameters used during execution, etc.

[0135] CPU 901, ROM 902, and RAM 903 are interconnected via a host bus 904a capable of high-speed data transfer. The host bus 904a is connected via a bridge 904 to an external bus 904b, such as a Peripheral Component Interconnect / Interface (PCI) bus, and the external bus 904b is connected to various components via an interface 905.

[0136] Input device 906 may be a device for receiving input from a user, such as a mouse, keyboard, touch panel, button, switch, or joystick. Note that input device 906 may be a microphone for detecting the user's voice. Input device 906 may be a remote control device using infrared or other radio waves, or it may be an external connection device adapted to the operation of information processing device 100.

[0137] The input device 906 also includes an input control circuit that outputs an input signal generated based on user input to the CPU 901. The user can input various types of data or issue instructions for processing operations to the information processing device 100 by operating the input device 906.

[0138] Output device 907 is a device capable of visually or audibly presenting information acquired or generated by information processing device 100 to a user. Output device 907 may be a display device such as a liquid crystal display (LCD), plasma display panel (PDP), organic light-emitting diode (OLED) display, hologram, or projector; an audio output device such as a speaker or headphones; or a printing device such as a printer. Output device 907 can output information obtained as a processing result of information processing device 100 as visual content such as text or images, or audio content such as speech or sound.

[0139] Storage device 908 is an example data storage device configured as a storage unit of information processing device 100. Storage device 908 may include, for example, magnetic storage devices such as hard disk drives (HDDs), semiconductor storage devices, optical storage devices, magneto-optical storage devices, etc. Storage device 908 can store programs executed by CPU 901, various types of data, various types of data acquired from external sources, etc.

[0140] The drive 909 is a reading or writing device for removable recording media such as magnetic disks, optical disks, magneto-optical disks, or semiconductor memories, and is either built into the information processing device 100 or externally attached to the information processing device 100. For example, the drive 909 can read information recorded on the attached removable recording medium and output the information to RAM 903. Furthermore, the drive 909 can write records to the attached removable recording medium.

[0141] Connection port 910 is a port used to directly connect external devices to information processing device 100. Connection port 910 can be, for example, a Universal Serial Bus (USB) port, an IEEE 1394 port, a Small Computer System Interface (SCSI) port, etc. Furthermore, connection port 910 can be an RS-232C port, an optical audio terminal, a High Definition Multimedia Interface (HDMI) port, etc. When connected to an external device, connection port 910 enables the transmission and reception of various types of data between information processing device 100 and the external device.

[0142] The communication device 911 includes, for example, a communication interface for connecting to a communication device 920. The communication device 911 can be, for example, a communication card for wired or wireless local area networks (LANs), Wi-Fi (registered trademark), Bluetooth (registered trademark), or wireless USB (WUSB). Furthermore, the communication device 911 can be a router for optical communication, a router for asymmetric digital subscriber line (ADSL), a modem for various types of communication, etc.

[0143] For example, communication device 911 can send signals to or receive signals from the Internet or another communication device using a predetermined protocol such as TCP / IP. Furthermore, the communication network 920 connected to communication device 911 is a network connected via wired or wireless communication, and can be, for example, an Internet communication network, a home LAN, an infrared communication network, a radio wave communication network, a satellite communication network, etc.

[0144] Note that a program can also be created that enables hardware built into the computer, such as CPU 901, ROM 902, or RAM 903, to perform the same functions as the information processing device 100 described above. Furthermore, a computer-readable recording medium storing the program can also be provided.

[0145] Although preferred embodiments of the present disclosure have been described in detail above with reference to the accompanying drawings, the technical scope of the present disclosure is not limited to such examples. It will be apparent to those skilled in the art that various changes or modifications can be conceived within the scope of the technical concept set forth in the claims, and it is naturally understood that such changes or modifications also fall within the technical scope of the present disclosure.

[0146] Furthermore, the effects described herein are merely exemplary or illustrative and are not intended to be limiting. In other words, in addition to or instead of the effects described herein, the technology according to this disclosure can achieve other effects that will be apparent to those skilled in the art.

[0147] Note that the following configurations also fall within the technical scope of this disclosure.

[0148] (1) An information processing device, comprising:

[0149] An instruction acquisition unit is configured to acquire motion instructions, the motion instructions being constructed by coupling the following: a control primitive that controls a functional unit disposed in the robot, a sub-target input to the control primitive, and an output primitive that outputs the sub-target; and

[0150] An output generation unit is configured to generate outputs for controlling the robot's functional units based on the motion commands.

[0151] (2) The information processing apparatus according to (1) above, wherein the motion command is constituted by at least coupling a sub-target input to the control primitive and an output primitive that outputs the sub-target to the control primitive in front of the control primitive.

[0152] (3) The information processing apparatus according to (2) above, wherein the output primitive corresponds to a planning primitive for planning the trajectory of the functional unit based on the input sub-target, and

[0153] The motion command is constructed by further coupling the input to the sub-objective of the planning primitive and the setting primitive that sets the sub-objective to the preceding level of the planning primitive.

[0154] (4) The information processing apparatus according to any one of (1) to (3) above further includes: an image generation unit configured to generate an image representing a block of the control primitive, the subtarget, and the output primitive, wherein,

[0155] The image generation unit generates an image that indicates whether the control primitive, the subtarget, and the output primitive can be coupled based on the interlocking shape of the coupled portions of the block.

[0156] (5) The information processing apparatus according to any one of (1) to (4) above, wherein the control primitive and the output primitive are encoded in a programming language.

[0157] (6) The information processing apparatus according to any one of (1) to (5) above, wherein the motion command is created based on a directed graph representing the coupling relationship between the control primitive, the subtarget and the output primitive.

[0158] (7) The information processing apparatus according to any one of (1) to (6) above further includes a log generation unit, the log generation unit being configured to generate an execution log, wherein the execution log records the start time and end time of the control primitive and the output primitive, respectively.

[0159] (8) According to the information processing device described in (7) above, in the case of recording the start time and end time of the control primitive, the log generation unit further records the state of the robot and the state of the environment in which the robot exists at the recording time in the execution log.

[0160] (9) The information processing apparatus according to any one of (1) to (8) above, wherein the instruction acquisition unit acquires an action instruction, the action instruction describing an action to be performed by the robot by combining multiple motion instructions.

[0161] (10) The information processing apparatus according to (9) above, wherein the output generation unit further generates the output of another functional unit for controlling the robot based on future environmental changes caused by the functional unit controlled by the control primitive within a series of action instructions.

[0162] (11) According to the information processing device described in (9) above, wherein the output generation unit further generates the output of another functional unit for controlling the robot based on the environmental changes caused by the movement of the functional unit controlled by the control primitive within a series of action instructions.

[0163] (12) The information processing apparatus according to any one of (1) to (11) above, wherein the functional unit corresponds to a robotic arm, a moving mechanism, or a head equipped with various sensors disposed in the robot.

[0164] (13) An information processing method implemented by a computer, the method comprising:

[0165] Acquire motion commands, which are constructed by coupling the following: a control primitive that controls functional units set in the robot, a sub-objective input to the control primitive, and an output primitive that outputs the sub-objective; and

[0166] The motion commands are used to generate outputs for the functional units that control the robot.

[0167] List of reference numerals

[0168] 1 robot

[0169] 10 main bodies

[0170] 11 functional units

[0171] 11A Mobile Platform

[0172] 11B head

[0173] 11C robotic arm

[0174] 11D hand

[0175] 100 Information Processing Device

[0176] 110 Input / Output Units

[0177] 120 image generation units

[0178] 130 Instruction Fetch Unit

[0179] 140 output generation unit

[0180] 150dB storage unit

[0181] 160 Log Generation Unit

[0182] 170 log storage units

[0183] PR1 control primitives

[0184] PR2 Output Primitives

[0185] SG sub-target

Claims

1. An information processing apparatus, comprising: An instruction acquisition unit is configured to acquire motion instructions, the motion instructions being constructed by coupling the following: a control primitive that controls a functional unit disposed in the robot, a sub-target input to the control primitive, and an output primitive that outputs the sub-target; and An output generation unit is configured to generate outputs for controlling the robot's functional units based on the motion commands.

2. The information processing apparatus according to claim 1, wherein, The motion command is constructed by coupling at least a sub-target input to the control primitive and an output primitive that outputs the sub-target to the control primitive at the preceding stage of the control primitive.

3. The information processing apparatus according to claim 2, wherein, The output primitive corresponds to the planning primitive that plans the trajectory of the functional unit based on the input sub-objective, and The motion command is constructed by further coupling the input to the sub-objective of the planning primitive and the setting primitive that sets the sub-objective to the preceding level of the planning primitive.

4. The information processing apparatus according to claim 1, further comprising: An image generation unit is configured to generate an image representing the block shape of the control primitive, the sub-target, and the output primitive, wherein... The image generation unit generates an image that indicates whether the control primitive, the subtarget, and the output primitive can be coupled based on the interlocking shape of the coupled portion of the block shape.

5. The information processing apparatus according to claim 1, wherein, The control primitives and the output primitives are encoded in a programming language.

6. The information processing apparatus according to claim 1, wherein, The motion command is created based on a directed graph representing the coupling relationship between the control primitive, the sub-target, and the output primitive.

7. The information processing apparatus according to claim 1 further includes a log generation unit, the log generation unit being configured to generate an execution log, wherein the execution log records the start time and end time of the control primitive and the output primitive, respectively.

8. The information processing apparatus according to claim 7, wherein, In recording the start and end times of the control primitives, the log generation unit further records the state of the robot and the state of the environment in which the robot exists at the recorded times in the execution log.

9. The information processing apparatus according to claim 1, wherein, The instruction acquisition unit acquires action instructions, which describe the actions to be performed by the robot by combining multiple motion instructions.

10. The information processing apparatus according to claim 9, wherein, The output generation unit further generates outputs for controlling additional functional units of the robot based on future environmental changes caused by the functional units controlled by the control primitives, within a series of action commands.

11. The information processing apparatus according to claim 9, wherein, The output generation unit further generates outputs for controlling other functional units of the robot based on environmental changes caused by the movement of the functional units controlled by the control primitives, within a series of action commands.

12. The information processing apparatus according to claim 1, wherein, The functional units correspond to the robotic arms, moving mechanisms, or heads equipped with various sensors that are installed in the robot.

13. An information processing method implemented by a computer, the method comprising: Acquire motion commands, which are constructed by coupling the following: a control primitive that controls functional units set in the robot, a sub-objective input to the control primitive, and an output primitive that outputs the sub-objective; and The motion commands are used to generate outputs for the functional units that control the robot.

Images