Information processing apparatus, information processing method, and non-transitory storage medium

Abstract

The present application provides an information processing device, an information processing method, and a non-transitory storage medium. The information processing device includes an acquisition unit, an identification unit, and a determination unit. The acquisition unit is configured to acquire information indicating an action trajectory of an expert for a specific task. The identification unit is configured to identify each action for an object based on the action trajectory and in chronological order. The determination unit determines each action for the robot arm to perform the task based on each action identified by the identification unit and the structure of the robot arm.

Description

Technical Field

[0001] This disclosure relates to an information processing apparatus, an information processing method, and a non-transitory storage medium. Background Technology

[0002] A technique has been known for which a robotic arm performs a predetermined task by executing actions based on state transitions (state machines) designed and installed by a human developer (e.g., see Japanese Patent Application Laid-Open No. 2019-200792). Summary of the Invention

[0003] However, in existing technologies, there are situations where it is difficult for robotic arms to perform new tasks (tasks that were not supported when the robotic arm was developed).

[0004] The purpose of this disclosure is to provide an information processing device, an information processing method, and a non-transitory storage medium that enable a robotic arm to properly perform new tasks.

[0005] The information processing apparatus according to the first aspect of the present invention includes: an acquisition unit configured to acquire information representing the action trajectory of an expert for a specific task; an identification unit configured to identify each action for an object based on the action trajectory and in a time sequence; and a determination unit configured to determine each action for the robotic arm to perform the task based on each action identified by the identification unit and the structure of the robotic arm.

[0006] In the information processing device involved in the first method described above, it can also be configured such that the movement trajectory includes an action of applying force to other parts of the object while using one arm to fix one part of the object. The determining unit is configured to determine the action of applying force to the other parts after the object has been fixed using a specific tool.

[0007] In the information processing device involved in the first method described above, the object may also be configured such that the object includes multiple objects, and the movement trajectory includes the action of using one arm to fix one object while using another arm to remove another object. The determining unit is configured to determine the action of causing one arm of the robotic arm to perform the action of removing another object after fixing one object with a specific tool.

[0008] In the information processing device involved in the first method described above, the object may also be configured such that the object includes multiple objects, and the movement trajectory includes the action of using one arm to hold one object while using another arm to apply force to another object. The determining unit is configured to determine the action of causing one arm of the robotic arm to perform the action of applying force to the other object after the object is placed.

[0009] In the information processing device involved in the first method described above, the object may also be configured such that the object includes multiple objects, and the movement trajectory includes the action of using one arm to hold one object while using another arm to apply force to another object. The determining unit is configured to determine the action of causing one arm of the robotic arm to perform the action of holding one object while applying force to the other object.

[0010] The information processing method involved in the second aspect of the present invention includes the following processing: obtaining information representing the action trajectory of an expert for a specific task; identifying each action for an object based on the action trajectory and in accordance with a time sequence; and determining each action that causes the robotic arm to perform the task based on the identified actions and the structure of the robotic arm.

[0011] The third aspect of the present invention relates to a non-transitory storage medium that stores instructions executable by one or more processors and causing the one or more processors to perform the following functions.

[0012] The functions include: obtaining information representing the action trajectory of an expert for a specific task; identifying each action for an object based on the action trajectory and in a time sequence; and determining each action that enables the robotic arm to perform the task based on the identified actions and the structure of the robotic arm.

[0013] This structure enables the robotic arm to perform new tasks appropriately. Attached Figure Description

[0014] The features, advantages, technical and industrial value of representative embodiments of the present invention will be depicted in the following drawings for reference, wherein the same symbols in the drawings denote the same elements, wherein:

[0015] Figure 1 This is a diagram illustrating an example of the structure of an information processing system according to an implementation method.

[0016] Figure 2 This diagram illustrates an example of the hardware structure of an information processing apparatus according to an embodiment.

[0017] Figure 3 This diagram illustrates an example of the structure of an information processing apparatus according to an embodiment.

[0018] Figure 4 This is a flowchart illustrating an example of the processing of an information processing apparatus according to an embodiment.

[0019] Figure 5 This diagram illustrates an example of the movement trajectory of the expert involved in the implementation and the actions in the robotic arm. Detailed Implementation

[0020] The principles of this disclosure are described with reference to some exemplary embodiments. These embodiments are described for illustrative purposes only and are intended to assist those skilled in the art in understanding and implementing the embodiments of this disclosure, and do not imply any limitation relating to the scope of this disclosure. The disclosures described herein are implemented by various methods other than those described below.

[0021] In the following description and technical solutions, unless otherwise defined, all technical and scientific terms used in this specification shall have the same meaning as commonly understood by those skilled in the art to which this disclosure pertains.

[0022] Hereinafter, embodiments of the present disclosure will be described with reference to the accompanying drawings.

[0023] <System Architecture>

[0024] Reference Figure 1 The structure of the information processing system 1 involved in the implementation method will be described. Figure 1 This diagram illustrates an example of the structure of the information processing system 1 according to an embodiment. Figure 1 In the example, the information processing system 1 has an information processing device 10, a robotic arm (an example of an “external device”) 20, and a sensor 30.

[0025] The information processing device 10 is a device that uses Artificial Intelligence (AI) to control the robotic arm 20. For example, the information processing device 10 acquires the movements of humans or others performing tasks (operations) via sensors 30, using this information as representation of the expert's movement trajectory. Furthermore, based on the acquired information, the information processing device 10 determines various movements corresponding to the physical characteristics of the robotic arm 20. Then, the information processing device 10 causes the robotic arm 20 to perform the determined movements, thereby enabling the robotic arm 20 to execute the task.

[0026] The robotic arm 20 is a robotic arm that performs tasks via an arm or similar device. The robotic arm 20 is not limited in shape, as long as it is a device capable of performing various tasks. The robotic arm 20 can be used for various purposes, such as home use, search and rescue, and factory use. The sensor 30 is a sensor that measures the area around the robotic arm 20. The sensor 30 can be, for example, a camera or a LiDAR (Laser Radar). Furthermore, the number of the information processing device 10, the robotic arm 20, and the sensor 30 is not limited. Figure 1 For example. Additionally, the information processing device 10 and the sensor 30 can also be housed inside the frame of the robotic arm 20.

[0027] <Hardware Structure>

[0028] Figure 2 This diagram illustrates an example of the hardware structure of the information processing apparatus 10 according to the embodiment. Figure 2 In the example, the information processing device 10 (computer 100) includes a processor 101, a memory 102, and a communication interface 103. These components may also be connected together via a bus or the like. The memory 102 stores at least a portion of the program 104. The communication interface 103 includes the interface required for communication with other network elements.

[0029] When program 104 is executed through the cooperative operation of processor 101 and memory 102, at least a portion of the processing of embodiments of the present disclosure is implemented using computer 100. Memory 102 can be any type of memory adapted to a local technology network. As a non-limiting example, memory 102 can also be a non-transitory computer-readable storage medium. Furthermore, memory 102 can be installed using any suitable data storage technology such as semiconductor-based memory devices, magnetic memory devices and systems, optical memory devices and systems, fixed memory, and removable memory. Although only one memory 102 is shown in computer 100, several physically different memory modules may exist in computer 100. Processor 101 can be any type of processor. Processor 101 can also include general-purpose computers, special-purpose computers, microprocessors, Digital Signal Processors (DSPs), and more than one processor based on a multi-core processor architecture as non-limiting examples. Computer 100 may also have multiple processors, such as application-specific integrated circuit chips that are time-dependent on a clock that synchronizes the main processor.

[0030] Embodiments of this disclosure can be installed via hardware or dedicated circuitry, software, logic, or any combination thereof. Some methods may utilize hardware for installation, while others may utilize a controller, microprocessor, or firmware or software executable by other computing devices.

[0031] This disclosure also provides at least one computer program product tangibly stored in a non-transitory computer-readable storage medium. The computer program product includes computer-executable instructions such as instructions contained in program modules, and is executed using a device on a physical or virtual processor of an object to perform the steps or methods of this disclosure. The program modules include routines, programs, libraries, objects, classes, components, data structures, etc., that perform specific tasks or install specific abstract data types. The functionality of the program modules can also be coupled or separated between program modules as desired by various implementation methods. The machine-executable instructions of the program modules can be executed in a local or distributed device. In a distributed device, the program modules can be configured in both local and remote storage media.

[0032] The program code used to perform the methods of this disclosure may also be written in any combination of more than one programming language. This program code is provided to the processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing device. When the program code is executed by the processor or controller, the functions / actions within the flowchart and / or the installed block diagram are performed. The program code may be executed entirely on the machine, or partially on the machine, or partially on the machine as a standalone software package, or partially on a remote machine, or entirely on a remote machine or server.

[0033] Programs can be stored and provided to a computer using various types of non-transitory computer-readable media. Non-transitory computer-readable media include various types of physical recording media. Examples of non-transitory computer-readable media include magnetic recording media, magneto-optical recording media, optical disc media, and semiconductor memory. Examples of magnetic recording media include floppy disks, magnetic tapes, and hard disk drives. Examples of magneto-optical recording media include magneto-optical disks. Examples of optical disc media include Blu-ray discs, Compact Disc Read Only Memory (CD-ROM), Compact Disc Recordable (CD-R), and Compact Disc ReWritable (CD-RW). Examples of semiconductor memory include solid-state drives, mask ROMs, Programmable Read Only Memory (PROM), Erasable Programmable Read Only Memory (EPROM), flash ROM, and random access memory (RAM). Furthermore, programs can also be provided to a computer using various types of temporary computer-readable media. Examples of temporary computer-readable media include electrical signals, optical signals, and electromagnetic waves. Temporary computer-readable media can provide programs to a computer via wired communication lines such as electrical wires and optical fibers, or via wireless communication lines.

[0034] <Structure>

[0035] Next, refer to Figure 3 The structure of the information processing device 10 involved in the implementation method will be described. Figure 3 This diagram illustrates an example of the structure of the information processing apparatus 10 according to an embodiment. Figure 3 In the example, the information processing device 10 has an acquisition unit 11, an identification unit 12, a determination unit 13, and a control unit 14. These units can also be implemented by one or more programs loaded into the information processing device 10 and working in conjunction with hardware such as the processor 101 and memory 102 of the information processing device 10.

[0036] The acquisition unit 11 acquires information representing the action trajectory of an expert for a specific task. The recognition unit 12 recognizes each action of the target object based on the action trajectory acquired by the acquisition unit 11 and in a time sequence.

[0037] The determining unit 13 determines (decides) each action that causes the robotic arm 20 to perform the specific task based on each action identified by the recognizing unit 12 and the structure of the robotic arm 20. The control unit 14 controls the robotic arm 20 based on information indicating each action determined by the determining unit 13.

[0038] <Processing>

[0039] Next, refer to Figure 4 and Figure 5 An example of the processing of the information processing device 10 involved in the implementation will be described. Figure 4 This is a flowchart illustrating an example of the processing of the information processing apparatus 10 according to an embodiment. Figure 5 This diagram illustrates an example of the movement trajectory of the expert involved in the implementation and the actions in the robotic arm 20.

[0040] In step S101, the acquisition unit 11 of the information processing device 10 acquires information representing the expert's movement trajectory for a specific task. The specific task could be, for example, retrieving an item from a bottle or from a refrigerator. Here, the information representing the expert's movement trajectory may also include, for example, information representing the position, speed, and shape changes of various parts of the object being manipulated according to the specific task. Furthermore, the information representing the expert's movement trajectory may also include, for example, information representing the position and posture of a human's arm at various points in time when the specific task is performed by a human.

[0041] Information representing the expert's movement trajectory can be generated, for example, by analyzing images captured by sensor 30, which acts as a camera, using a Convolutional Neural Network (CNN). This information may also include, for example, information about the position, speed, and shape changes of various parts of an object (the target object) being manipulated according to a specific task.

[0042] Next, the recognition unit 12 of the information processing device 10 recognizes the actions directed at the object based on the information representing the expert's action trajectory and according to the time sequence (step S102). Here, the recognition unit 12 may, for example, extract (decompose the task) the actions (processing, operation) directed at the object in a specific task from the expert's action trajectory according to the time sequence.

[0043] Next, the determination unit 13 of the information processing device 10 determines the actions (mismatched actions) that do not match the structure (body structure) of the robotic arm 20 among the actions identified by the recognition unit 12 in the time sequence (step S103). Here, the determination unit 13 may also determine multiple actions (methods) that can be performed by the robotic arm 20 to achieve the function of each action in the time sequence identified by the recognition unit 12. Then, the selection unit 13 may also search the multiple determined actions in the time sequence for each action and extract actions that do not match the structure of the robotic arm 20.

[0044] For example, if the robotic arm 20 has a structure with only one arm, the determination unit 13 can also use AI to determine the actions in the time sequence identified by the recognition unit 12 in which a human uses both arms to simultaneously apply force to an object. This allows for the determination of actions that cannot be matched with the structure of the robotic arm 20.

[0045] Next, the determination unit 13 of the information processing device 10 determines the action that can be performed by the robotic arm 20 corresponding to the determined mismatch action (step S104). As a result, it is possible to enable the robotic arm 20 to perform the new task, for example, even if a series of actions in a new task for which a human has demonstrated a template includes actions that do not match the structure of the robotic arm 20.

[0046] Here, the determination unit 13 can also determine multiple actions (methods) that can be performed by the robotic arm 20 and achieve the effect of the mismatched actions. Then, the determination unit 13 can also search the multiple actions determined for the mismatched actions in a time sequence, and determine the arrangement of actions with shorter time and lower difficulty from the arrangement of actions that cannot be matched with the structure of the robotic arm 20.

[0047] Furthermore, the actions that can be performed by the robotic arm 20 may include actions that can be performed using only the arm of the robotic arm 20, as well as actions that require both the arm of the robotic arm 20 and a usable tool (e.g., a tool capable of fixing an object). Additionally, the actions that can be performed by the robotic arm 20 may include, for example, actions in which a human is used as a tool based on a voice message output from the speaker of the robotic arm 20 (e.g., a message requesting the fixing of an object).

[0048] Next, the control unit 14 of the information processing device 10 causes the robotic arm 20 to perform various actions, including the actions determined by the determination unit 13 (step S105). Here, as Figure 5As shown, the control unit 14 can also use imitation learning or the like to make the robotic arm 20 execute actions other than mismatched actions among the actions identified by the recognition unit 12 in the time sequence. In addition, the control unit 14 can also make the robotic arm 20 execute actions determined by the determination unit 13 for mismatched actions among the actions identified by the recognition unit 12 in the time sequence.

[0049] exist Figure 5 In the example, in a certain task, within a series of actions 510 along the expert's action trajectory, actions 511, 512, and 513 are executed in chronological order. Furthermore, action 512, which is a mismatched action, is converted by the determining unit 13 into actions 521 and 522, which can be executed by the robotic arm 20. Therefore, as a series of actions 520 that can be executed by the robotic arm 20, actions 511, 521, 522, and 513 are executed by the robotic arm 20 in chronological order.

[0050] The following section illustrates an example of converting a mismatched motion into a motion that can be performed by the robotic arm 20 when the robotic arm 20 has only one arm. Additionally, the following examples can be combined appropriately.

[0051] (An example of applying force after the object has been secured)

[0052] If the expert's action trajectory includes an action of using one arm (hand) to fix one part of the object while using the other arm to apply force to other parts of the object, the determination unit 13 may also determine this action as a mismatch action in the processing of step S103. Then, in the processing of step S104, the determination unit 13 may also determine the action of applying force to the other parts after fixing the object with a specific tool as an action that can be performed by the robotic arm 20 corresponding to the mismatch action.

[0053] In this case, for example, in a task of retrieving an item from a bottle, if the expert's action trajectory includes the action of using one arm to hold the bottle while using the other arm to open the bottle cap, this action can also be identified as a mismatch action. Moreover, as a corresponding action that can be performed by the robotic arm 20, the action of opening the bottle cap after using a specific tool to fix the bottle can also be identified.

[0054] (An example of removing an item after securing a door or similar object)

[0055] If the expert's action trajectory includes an action of using one arm to fix one object while using the other arm to remove another object, the determination unit 13 can also determine this action as a mismatch action in the processing of step S103. Moreover, the determination unit 13 can also determine, in the processing of step S104, an action that corresponds to the mismatch action and can be performed by the robotic arm 20, the action of removing the other object after fixing one object with a specific tool.

[0056] In this case, for example, if the expert's action trajectory in the task of retrieving items from the refrigerator includes an action of using one arm to keep the refrigerator door open while using the other arm to retrieve the items from the refrigerator, this action can also be identified as a mismatch action. Moreover, as an action that can be performed by the robotic arm 20 corresponding to this mismatch action, the action of retrieving the items from the refrigerator after using a specific tool to secure the refrigerator door can also be identified.

[0057] (An example of closing the door or other doors after placing items down at one point)

[0058] If the expert's action trajectory includes an action of using one arm to hold an object while simultaneously applying force to another object using the other arm, the determination unit 13 can also determine this action as a mismatch action in the processing of step S103. Furthermore, in the processing of step S104, the determination unit 13 can also determine the action of applying force to the other object after placing one object on the floor or the like as an action that can be performed by the robotic arm 20, corresponding to this mismatch action.

[0059] In this case, for example, if the expert's movement trajectory in a task of retrieving an item from a refrigerator includes the action of closing the refrigerator door while holding the item with one arm, this action can be identified as a mismatched action. Moreover, as an action that can be performed by the robotic arm 20 corresponding to this mismatched action, the action of closing the refrigerator door after placing the item retrieved from the refrigerator on the floor or the like can also be identified.

[0060] (Examples of using an object to push a door or similar device to close it)

[0061] If the expert's action trajectory includes an action of using one arm to hold an object while simultaneously applying force to another object using the other arm, the determination unit 13 can also determine this action as a mismatch action in the processing of step S103. Furthermore, in the processing of step S104, the determination unit 13 can also determine, as an action that can be performed by the robotic arm 20 while holding one object, an action that corresponds to this mismatch action.

[0062] In this case, for example, if the expert's action trajectory in a task of retrieving an item from a refrigerator includes the action of closing the refrigerator door while holding the item with one arm, this action can also be identified as a mismatch action. Moreover, as an action that can be performed by the robotic arm 20 corresponding to this mismatch action, it can also be identified as the action of pushing the refrigerator door to close it while holding the item taken from the refrigerator.

[0063] <Change Example>

[0064] While the information processing apparatus 10 may also be a device contained within a housing, the information processing apparatus 10 of this disclosure is not limited thereto. Various parts of the information processing apparatus 10 may also be implemented, for example, through cloud computing comprised of one or more computers. Such information processing apparatuses are also included in one example of the "information processing apparatus" of this disclosure.

[0065] Furthermore, the present invention is not limited to the above-described embodiments, and can be appropriately modified without departing from the spirit of the invention.

Claims

1. An information processing device, comprising: The acquisition department is configured to acquire information representing the action trajectory of experts for a specific task; The recognition unit is configured to recognize each action on the object based on the action trajectory and in accordance with the time sequence. The determining unit determines each action that causes the robotic arm to perform the task, based on the actions identified by the recognition unit and the structure of the robotic arm. The robotic arm has a structure with only one arm. The trajectory includes a first action of using one arm to stabilize one part of the object while using the other arm to apply force to other parts of the object; a second action of using one arm to stabilize one object while using the other arm to remove another object; and a third action of using one arm to hold one object while using the other arm to apply force to another object. The determining unit identifies the first action, the second action, and the third action as mismatched actions, and converts the mismatched actions into actions that can be performed by the robotic arm. The determining unit is configured to determine an action such that, in response to a transition corresponding to the first action, the arm of the robotic arm performs an action of applying force to other parts after the object has been fixed using a specific tool. The object refers to multiple objects. The determining unit is configured to determine an action such that, in response to a transition corresponding to the second action, the arm of the robotic arm performs an action of removing another object after securing one object using a specific tool. The determining unit is configured to determine an action such that, in response to a transition corresponding to the third action, the arm of the robotic arm performs an action of applying force to another object after loading one object. The determining unit is configured to determine an action such that, in response to a transition corresponding to the third action, the arm of the robotic arm performs an action of applying force to another object while holding one object.

Images