Robot motion generation assistance device, robot motion generation assistance system, and robot motion generation assistance program

Abstract

The present invention provides a robot motion generation assistance device for a robot that performs motions with respect to an object, said device comprising: a storage means that stores API documentation for controlling the robot and information in an object database; a text data acquisition means that acquires text data in which a user describes, in natural language, details of an outcome obtained by the motion performed by the robot with respect to the object; a synthetic text data generation means that generates synthetic text data by adding the API documentation for controlling the robot and the information in the object database to the text data acquired by the text data acquisition means; a source code acquisition means that inputs the synthetic text data generated by the synthetic text data means into a natural language processing unit to obtain output from the natural language processing unit in the form of source code for motion of the robot for the purpose of achieving the outcome; a simulation means that uses the source code acquired by the source code acquisition means to run a simulation causing the robot to perform motions in a virtual environment; a transmission means that transmits the source code to the robot if the outcome is achieved in the simulation; and a source code revision means that, if the outcome is not achieved in the simulation, revises the source code by re-inputting information necessary for revision of the source code into the natural language processing unit on the basis of abnormality information that includes an error message detected during the simulation.

Description

Robot motion generation support device, robot motion generation support system, and robot motion generation support program

[0001] The present disclosure relates to a robot motion generation support device, a robot motion generation support system, and a robot motion generation support program.

[0002] It is well known that industrial robots are used for various manufacturing, assembly, and other purposes. Generally, robots pick, move, and place objects of various shapes. Conventionally, a user (operator) uses a teaching pendant to teach a robot how to pick, move, and place an object. Teaching a robot using such a teaching pendant is time-consuming and requires specialized programming skills from the user.

[0003] Therefore, Non-Patent Document 1 discloses a technology for generating trajectory data for a robot by natural language processing without using a teach pendant. The natural language processing disclosed in Non-Patent Document 1 can generate a robot operation program for achieving simple action sequences such as moving an object or opening and closing a door.

[0004] Naoki Wake, Atsushi Kanehira, Kazuhiro Sasabuchi, Jun Takamatsu, and Katsushi Ikeuchi, “ChatGPT Empowered Long-Step Robot Control in Various Environments: A Case Application”, 2023, https: / / www.microsoft.com / en-us / research / uploads / prod / 2023 / 04 / chatgpt_robot_manipulation_prompts.pdf

[0005] However, in conventional technologies, generating robot movements that achieve complex action sequences requires a user to have advanced specialized knowledge. Furthermore, the natural language processing disclosed in Non-Patent Document 1 has difficulty accurately generating complex movement programs that include conditional branching, repetitive processing, etc.

[0006] Therefore, the present disclosure aims to provide a robot motion generation support device, a robot motion generation support system, and a robot motion generation support program that enable the generation of robot motions that perform complex tasks without requiring specialized knowledge.

[0007] a source code acquisition means for acquiring information necessary for correcting the source code to the natural language processing unit based on abnormality information including an error message detected during the simulation; a source code correction means for correcting the source code by re-inputting ... storage means for storing an API document for controlling the robot and information of an object database; a text data acquisition means for acquiring text data described in a natural language by a user that describes in a natural language the content of a resultant object obtained by the robot's action on the object; a synthetic text data generation means for generating synthetic text data by adding the API document for controlling the robot and information of the object database to the text data acquired by the text data acquisition means; a source code acquisition means for inputting the synthetic text data generated by the synthetic text data means to a natural language processing unit and obtaining, from the natural language processing unit, output of source code for the robot's action to obtain the resultant object; a simulation means for performing a simulation of the robot's action in a virtual environment using the source code acquired by the source code acquisition means; a transmission means for transmitting the source code to the robot if the resultant object is obtained in the simulation; and a source code correction means for correcting the source code by re-inputting information necessary for correcting the source code to the natural language processing unit based on abnormality information including an error message detected during the simulation if the resultant object is not obtained in the simulation.

[0008] According to the present disclosure, it is possible to provide a robot motion generation support device, a robot motion generation support system, and a robot motion generation support program that enable the generation of robot motions to perform complex tasks without requiring specialized knowledge.

[0009] Fig. 1 is a diagram showing an example of the overall configuration of a robot motion generation support system according to an embodiment. Fig. 2 is a diagram showing an example of the configuration of a robot motion generation support server according to an embodiment. Fig. 3 is a diagram showing an example of a display on an input screen of a terminal device according to an embodiment. Fig. 4 is a diagram showing an example of synthesized text data according to an embodiment. Fig. 5 is a diagram showing an example of a motion sequence of a robot motion generation support system according to an embodiment.

[0010] Hereinafter, embodiments of the present disclosure will be described in detail. However, the present disclosure is not limited to these examples, but is defined by the claims, and is intended to include all modifications within the meaning and scope equivalent to the claims.

[0011] In the description of the specification and drawings of each embodiment, components having substantially the same or corresponding functional configurations may be designated by the same reference numerals to avoid redundant explanation. In addition, the scale of each part in the drawings may differ from the actual scale to facilitate understanding.

[0012] <<Embodiment>> <Overall Configuration of Robot Motion Generation Support System 1> First, a description will be given of the overall configuration of the robot motion generation support system 1. Fig. 1 is a diagram showing an example of the overall configuration of the robot motion generation support system 1 according to this embodiment. As shown in Fig. 1, the robot motion generation support system 1 includes a robot motion generation support server 100, a terminal device 200, and a robot 300.

[0013] The robot motion generation assistance server 100, the terminal device 200, and the robot 300 are connected to a network 10 and can communicate with each other via the network 10. The network 10 includes the Internet. The network 10 may include a local area network (LAN) and / or a wide area network (WAN). The robot motion generation assistance server 100, the terminal device 200, and the robot 300 may be directly connected via wired or wireless communication. The robot motion generation assistance server 100 may be a general-purpose computer such as a workstation or PC, or may be logically realized by cloud computing (distributed computing). In other words, it may be configured with one computer or multiple computers.

[0014] The robot motion generation assistance server 100 is an example of a robot motion generation assistance device that assists a user in generating a motion program (source code) for a robot 300 that performs complex tasks. The source code is a string of characters written using a programming language such as Python. In this embodiment, the source code is in Python format, but is not limited to this format and may be written in any programming language, such as C, C++, C#, Java, Visual Basic, Perl, or JavaScript.

[0015] Here, an example of a situation to which the present disclosure is applicable will be described. One example of a situation to which the present disclosure is applicable is a situation in which a user of a robot 300 installed at a manufacturing site for food such as hospital meals and boxed lunches generates an operation program (source code) for the robot 300. The robot 300 installed at a manufacturing site for food such as hospital meals and boxed lunches repeatedly identifies (selects) ingredients from a plurality of prepared foods, transports the identified ingredients to a predetermined location in a container, and releases them. The work of such a robot 300 involves different operations and repetitive operations depending on conditions, and therefore requires a complex operation program (source code). Note that the situation to which the present disclosure is applicable is not limited to manufacturing sites for food such as hospital meals and boxed lunches, and may also be applied to, for example, supporting the generation of an operation program (source code) for a robot 300 that performs complex tasks such as manufacturing and assembling parts and components.

[0016] In this embodiment, the robot motion generation assistance server 100 assists a user who generates source code for the robot 300 that performs an action on an object in generating the motion of the robot 300. For example, the robot motion generation assistance server 100 displays, on the user's terminal device 200, an input screen 131a that prompts the user to input the details of a resultant object obtained by the action of the robot 300 on an object, generates source code for the motion of the robot 300 according to the input details of the resultant object, and transmits the generated source code to the robot 300.

[0017] The robot motion generation support server 100 also stores information on an API document 122 for controlling the robot 300 and an object database 121, acquires text data written in natural language by a user about the content of a resultant object obtained by the robot 300's motion on an object, adds the information on the API document 122 for controlling the robot 300 and the object database 121 to the acquired text data to generate synthetic text data, inputs the generated synthetic text data to the natural language processing unit 135, obtains output of source code for the motion of the robot 300 to obtain the resultant object from the natural language processing unit 135, operates the robot 300 in a virtual environment using the acquired source code, and if the resultant object is obtained in the simulation, transmits the source code to the robot 300, and if the resultant object is not obtained in the simulation, re-inputs information necessary for correcting the source code to the natural language processing unit 135 based on abnormality information including an error message detected during the simulation, and corrects the source code.

[0018] The robot 300 has a robot hand 301. The target object is, for example, an ingredient of a prepared dish. The API document 122 includes, for example, information regarding the grasping of the robot hand 301, the movement of the robot hand 301, the release of the robot hand 301, and the layout information of the destination location. The result is, for example, a fried chicken lunch box, the contents of which are six pieces of fried chicken, white rice, and potato salad.

[0019] In this way, if the user describes in natural language the content of the result obtained by the action of the robot 300 on an object, source code for the action of the robot 300 corresponding to the content of the described result is automatically generated, and the source code is transmitted to the robot 300. Therefore, even a user with little programming skill can generate source code for the robot 300 that performs complex tasks including different actions depending on conditions, repeated actions, etc.

[0020] Furthermore, in manufacturing sites for hospital meals, boxed lunches, and other foods, the contents of boxed lunches and other foods (such as the type and quantity of ingredients) vary widely depending on the patient's health condition and customer orders, so it was previously necessary to generate source code for the operation of multiple robots 300. However, according to this embodiment, source code for the operation of robot 300 can be automatically generated, making it easy to generate multiple source codes.

[0021] Note that a natural language refers to a language that has arisen naturally in the history of mankind, such as Japanese, English, etc. In this embodiment, the natural language is Japanese, but it is not limited to Japanese and may be another type of natural language (for example, English).

[0022] The robot motion generation assistance server 100 is a server that executes natural language processing tasks using a large language model (LLM) constructed using machine learning techniques including artificial intelligence (AI). The LLM is a language model that learns a large amount of text data in advance and can execute various language processing tasks by simply providing a few example tasks. Examples of such an LLM include GPT-3 and GPT-4 developed by OpenAI, Inc.

[0023] In this embodiment, the robot motion generation assistance server 100 automatically generates source code from natural language using an LLM that has previously studied a large amount of source code. For example, the LLM has studied a large amount of source code publicly available on the Internet. For example, since source code generally includes natural language sentences such as comments, the LLM has already learned the correspondence between natural language sentences and programs. Therefore, when the robot motion generation assistance server 100 acquires natural language sentences (text data) written by a user, it can automatically generate source code and transmit the source code to the robot 300.

[0024] Furthermore, the robot motion generation support server 100 can generate source code that combines the grasping, transporting, and releasing of an object by the robot hand 301 by inputting information from the control API document 122 of the robot 300 and the object database 121 into the LLM in addition to the contents of the result obtained by the robot 300's motion on an object obtained from the user.

[0025] The terminal device 200 is, for example, a personal computer (PC), a smartphone, a tablet terminal, or a wearable terminal. The terminal device 200 has an interface with a user and an interface with the network 10. The interface with the user includes, for example, at least one of a display, a keyboard, a mouse, a touchpad, a touch panel display, a microphone, and a speaker. The interface with the network 10 includes at least one of a wired communication interface and a wireless communication interface. The terminal device 200 accesses the robot motion generation assistance server 100 via the network 10 and exchanges information with the robot motion generation assistance server 100 to generate source code for the motion of the robot 300.

[0026] The robot 300 includes, for example, a robot hand 301 that grasps an object, an imaging unit 302 that captures an image of the object, and a control unit 303 that controls the robot hand 301 and the imaging unit 302. The robot 300 can produce a result (fried chicken bento) by repeatedly performing a series of actions, such as identifying (selecting) an object (ingredient) to be grasped by performing image recognition on image data captured by the imaging unit 302, grasping the identified object (ingredient), transferring the grasped object (ingredient) to a predetermined location (inside a container such as a dish or lunch box), and releasing the object (ingredient) at the predetermined location (inside the container).

[0027] Furthermore, the robot 300 stores a trained model that identifies the names of ingredients included in image data, and performs image recognition processing using the trained model. That is, the robot 300 inputs image data in which multiple ingredients are photographed into the trained model, thereby identifying the names of the ingredients included in the image data and obtaining the positions of the identified ingredients.

[0028] <Robot Motion Generation Support Server 100> Next, the configuration of the robot motion generation support server 100 will be described. Fig. 2 is a diagram showing an example of the configuration of the robot motion generation support server 100 according to this embodiment. As shown in Fig. 2, the robot motion generation support server 100 includes a communication unit 110, a storage unit 120, and a processing unit 130. The communication unit 110, the storage unit 120, and the processing unit 130 are connected to each other by a bus or the like (not shown).

[0029] The communication unit 110 communicates with other devices (e.g., the terminal device 200 and the robot 300) via the network 10 under the control of the processing unit 130. The communication by the communication unit 110 may be wired communication or may include wireless communication.

[0030] The storage unit 120 stores an API document 122 for controlling the robot 300 and information on an object database 121. The API document 122 is also an API specification and includes information for executing functions such as grasping, moving, and releasing by the robot hand 301. The source code generated by the robot motion generation assistance server 100 uses the API document 122, and therefore can execute functions such as grasping, moving, and releasing by the robot hand 301 via the API. The object (ingredient) database 121 records information on the names of learned objects (ingredients) of the learned model stored in the robot 300. The names of the objects (ingredients) recorded in the object (ingredient) database 121 are the same as the names of the learned ingredients stored in the robot 300.

[0031] The storage unit 120 includes various types of memory, such as a ROM (Read Only Memory), a RAM (Random Access Memory), and an auxiliary storage device. The program (robot motion generation support program) executed by the processing unit 130 is stored in the ROM and / or the auxiliary storage device of the storage unit 120. The processing unit 130 includes one or more processors.

[0032] By executing the robot movement generation support program, the processing unit 130 realizes the functions of an information presentation unit 131, a text data acquisition unit 132, a synthetic text data generation unit 133, a source code acquisition unit 134, a natural language processing unit 135, a simulation unit 136, and a transmission unit 137.

[0033] The information presenting unit 131 displays an input screen 131a on the terminal device 200, which prompts the user to input the details of the result (fried chicken bento) obtained by the action of the robot 300 on the target object (ingredient), and also presents information for supporting the generation of robot action to the user on the input screen 131a. Examples of the screen display on the terminal device 200 will be described later.

[0034] The text data acquisition unit 132 acquires text data in which the user has written the content of the result (fried chicken bento) in natural language. For example, the text data acquisition unit 132 acquires a character string (sentence) written in natural language by the user in a text input field on the input screen 131a as text data. In either case, the user can input text in a free format using natural language.

[0035] However, the text data entered by the user may not satisfy the conditions for outputting source code for the robot 300's operations. For example, there may be an insufficient description of the object (insufficient description of the names and quantities of ingredients), and the natural language processing unit 135 may not be able to output source code. In such a case, the information presenting unit 131 may present the user with information prompting them to input the contents of a new result that satisfies the conditions for source code output. This can ensure that the source code for the robot 300's operations is created more reliably.

[0036] The information presenting unit 131 and the text data acquiring unit 132 may accept step-by-step text input from the user in a question-and-answer format. For example, the information presenting unit 131 presents question information to the user, and the text data acquiring unit 132 accumulates sentences input as answers to the question information. By repeating this process, a sufficient amount of text data can be obtained. That is, the information presenting unit 131 may present question information to the user multiple times, the text data acquiring unit 132 may accumulate text data input as answers to the question information, and the natural language processing unit 135 may output source code from the accumulated text data. This can assist in creating high-quality source code for the operation of the robot 300.

[0037] The synthetic text data generation unit 133 generates synthetic text data by adding at least the API document 122 for controlling the robot 300 and information from the object (ingredient) database 121 to the text data acquired by the text data acquisition unit 132. An example of the content of the synthetic text data will be described later. The information from the API document 122 and the object (ingredient) database 121 is text data. The synthetic text data generation unit 133 may also transmit the generated synthetic text data to the information presentation unit 131 to display it on the user's terminal device 200. As described above, the names of the objects (ingredients) recorded in the object (ingredient) database 121 are similar to the names of the learned ingredients stored in the robot 300. Therefore, by adding the information from the API document 122 and the object (ingredient) database 121 to the text data, it is possible to teach the natural language processing unit 135 objects (ingredients) that can be image-recognized by the learned model stored in the robot 300. 4, the natural language processing unit 135 can be instructed on objects (ingredients) that can be searched for by the find function, which is the API document 122. In this way, of the results described in natural language, the natural language processing unit 135 can be instructed on objects (ingredients) that the robot 300 can image-recognize. If the results include objects (ingredients) that the robot 300 cannot image-recognize, the information presentation unit 131 may display, for example, a screen that lists the objects (ingredients) that can be image-recognized and then prompts the user to re-input the details of the results, or a screen that displays a message indicating that additional learning of the trained model stored in the robot 300 is required.

[0038] The source code acquisition unit 134 inputs the synthetic text data generated by the synthetic text data generation unit 133 to the natural language processing unit 135, and obtains an output of source code for the movements of the robot 300 from the natural language processing unit 135. In this embodiment, the source code acquisition unit 134 provides the synthetic text data generated by the synthetic text data generation unit 133 to the natural language processing unit 135 via an API, and acquires the source code for the movements of the robot 300 from the natural language processing unit 135 via the API, thereby acquiring the source code for the movements of the robot 300. Note that in this embodiment, the natural language processing unit 135 is incorporated into the processing unit 130, but if, for example, the natural language processing unit 135 and the robot movement generation support server 100 are separate servers, the synthetic text data generated by the synthetic text data generation unit 133 may be provided to the natural language processing server via an API, and the source code for the movements of the robot 300 may be acquired from the natural language processing server via the API.

[0039] The natural language processor 135 executes natural language processing tasks using the LLM. The LLM is a language model that learns a large amount of text data in advance and can execute various language processing tasks by simply providing a few example tasks. In this embodiment, the natural language processor 135 outputs source code for the robot 300's operation corresponding to the synthesized text data.

[0040] The simulation unit 136 uses the source code acquired by the source code acquisition unit 134 in a virtual environment to test whether the robot 300 operates based on the contents of a result described by the user (a fried chicken bento box consisting of six pieces of fried chicken, white rice, and potato salad). In this embodiment, the robot 300 operates in the virtual environment using the source code acquired by the source code acquisition unit 134 by using open source ROS (Robot Operating System) middleware for the robot 300 and open source Gazebo physics simulator. Furthermore, the simulation unit 136 determines that the simulation has ended normally if the result described by the user can be obtained in the simulation, and determines that the simulation has ended abnormally if the result described by the user cannot be obtained.

[0041] When the simulation has ended normally, the simulation unit 136 transmits the source code of the operation of the robot 300 to the robot 300 via the transmission unit 137. Furthermore, the simulation unit 136 may transmit the source code when the simulation has ended normally to the information presentation unit 131, and display it on the user's terminal device 200.

[0042] An abnormal termination of the simulation occurs, for example, when an error message or warning occurs during the simulation. The simulation unit 136 can monitor log information related to the state of the robot 300 during the simulation, the progress of tasks, etc., in real time via the ROS. The simulation unit 136 can also detect and acquire abnormality information, including error messages and warnings, from the log information. The robot motion generation assistance server 100 uses this abnormality information to correct the source code.

[0043] The simulation unit 136 generates feedback data for correcting the source code based on the abnormality information. Furthermore, the source code acquisition unit 134 re-inputs the feedback data generated by the simulation unit 136 into the natural language processing unit 135, and obtains corrected source code output from the natural language processing unit 135. In this way, the robot motion generation assistance server 100 corrects the source code.

[0044] The content of the feedback data may be, for example, "ERROR: 'Fried chicken' cannot be released. An obstacle exists at the target position." In the present embodiment, the simulation unit 136 acquires abnormality information including error messages and warnings from log information via the ROS, but is not limited to this. Various data such as the state of the robot 300 during the simulation and abnormality information may be acquired via the Gazebo API.

[0045] Furthermore, the simulation unit 136 may transmit abnormality information when the simulation terminates abnormally to the information presentation unit 131 and display it on the user's terminal device 200. The simulation unit 136 may also transmit the acquired abnormality information to the information presentation unit 131, display it on the terminal device 200, and accept text input from the user. For example, the simulation unit 136 may present the abnormality information to the user, accept text input from the user regarding the correction details of functions, code, etc. that need to be corrected, and generate feedback data based on the content accepted from the user. In this case, the user may include in the source code an exception handling that displays an error message in response to some event, such as the occurrence of an error (e.g., by describing code in which an event may occur in a Try block). This allows the natural language processing unit 135 to receive specific correction instructions, thereby reducing the time required to correct the source code.

[0046] The simulation unit 136 uses the corrected source code acquired by the source code acquisition unit 134 in a virtual environment to re-simulate the movement of the robot 300. If the re-simulation ends abnormally, the above-described source code correction means is performed. That is, the robot movement generation support server 100 repeatedly corrects the source code and performs simulation using the corrected source code until the simulation ends normally. In this way, even if there is a problem with the source code, the source code can be automatically corrected based on the abnormality information from the simulation. This ensures that accurate source code is created.

[0047] Before using the source code to execute a simulation in the simulation unit 136, the robot motion generation support server 100 may check whether the source code includes code related to access to the file system or the network 10. If the source code includes code related to access to the file system or the network 10, the robot motion generation support server 100 deletes the unnecessary code and prevents access to the file system or the network 10. In other words, the robot motion generation support server 100 may perform a security check before executing a simulation.

[0048] The transmitting unit 137 transmits the source code when the simulation ends normally to the robot 300. The robot 300 acquires the source code transmitted from the transmitting unit 137. Using the source code, the robot 300 performs the actions of grasping, transferring, and releasing the target object (ingredients) according to the contents of the result (a lunch box consisting of six pieces of fried chicken, white rice, and potato salad).

[0049] <Screen Display of Terminal Device 200> Next, an example of a screen display of the terminal device 200 will be described. Fig. 3 is a diagram showing an example of a display of the input screen 131a of the terminal device 200 according to this embodiment. As shown in Fig. 3, the information presenting unit 131 displays the input screen 131a on the terminal device 200. The information presenting unit 131 displays, for example, a text input field A for text input as the input screen 131a, and also displays information prompting the user to input text into the text input field A. The information presenting unit 131 may also display sample information showing an example of text input into the text input field A.

[0050] When the content of the result (fried chicken bento) written in natural language is input by the user into the text input field A, the information presenting unit 131 displays a button B for performing an operation to acquire source code. When button B is operated, the text data acquiring unit 132 acquires the text data. In the illustrated example, the content of the result is "fried chicken bento consisting of six pieces of fried chicken, white rice, and potato salad," and the user has input the content of the result in Japanese.

[0051] <Contents of Synthesized Text Data> Next, an example of the contents of the synthesized text data will be described. Fig. 4 is a diagram showing an example of synthesized text data according to this embodiment. The synthesized text data generation unit 133 generates synthesized text data as shown in Fig. 4 by adding information from the API document 122 for controlling the robot 300 and the target object (ingredient) database 121 to the contents of the resultant obtained by the text data acquisition unit 132.

[0052] <Example of Motion Sequence> Next, a description will be given of the flow of the motion of the robot motion generation support system 1. Fig. 5 is a diagram showing an example of the motion sequence of the robot motion generation support system 1 according to this embodiment.

[0053] In step S1 , the user accesses the robot motion generation support server 100 from the terminal device 200 .

[0054] In step S2, the robot motion generation support server 100 transmits the input screen 131a to the terminal device 200 to display it.

[0055] In step S3, the terminal device 200 accepts text input into the text input field on the input screen 131a. The user inputs "Fried chicken bento consisting of six pieces of fried chicken, white rice, and potato salad" into the text input field, as shown in Fig. 3. The terminal device 200 transmits the input text data to the robot motion generation assistance server 100.

[0056] In step S4, the robot motion generation support server 100 generates synthetic text data by adding the information of the API document 122 and the object (ingredient) database 121 to the text data received from the terminal device 200. The synthetic text data has the content as shown in FIG.

[0057] In step S5 , the robot motion generation support server 100 inputs the synthesized text data to the natural language processing unit 135 and acquires the source code output from the natural language processing unit 135 .

[0058] In step S6, the robot motion generation support server 100 uses the acquired source code in the simulation unit 136 to test the motion of the robot 300.

[0059] In step S7, the robot motion generation support server 100 proceeds to step S8 if the simulation has ended normally, and proceeds to step S10 if the simulation has ended abnormally.

[0060] In step S8, the robot motion generation support server 100 transmits the source code used in the simulation to the robot 300.

[0061] In step S9, the robot 300 acquires the source code transmitted from the robot motion generation support server 100, and performs the motions of grasping, transferring, and releasing the target object (ingredient) using the acquired source code.

[0062] In step S10, the robot motion generation support server 100 detects and acquires abnormality information from the simulation log information.

[0063] In step S11, the robot motion generation support server 100 generates feedback data based on the abnormality information. The content of the feedback data is, for example, "ERROR: 'Fried chicken' cannot be released. An obstacle exists at the target position." Next, the process proceeds to step S5, where the robot motion generation support server 100 inputs the generated feedback data to the natural language processing unit 135 and obtains an output of corrected source code from the natural language processing unit 135.

[0064] <Modification> In the above-described embodiment, the source code is corrected based on anomaly information, but in the modification, the source code is also corrected in accordance with the execution result in the interpreter unit 140. Note that the same components as in the above-described embodiment are denoted by the same reference numerals, and description thereof will be omitted.

[0065] In a modified example, the robot motion generation assistance server 100 further includes an interpreter unit 140 for executing source code. The interpreter unit 140 executes the source code using, for example, a Python interpreter. The interpreter unit 140 executes the source code acquired by the source code acquisition unit 134 before the simulation by the simulation unit 136. If the execution by the interpreter unit 140 ends normally, the simulation by the simulation unit 136 follows.

[0066] If execution by the interpreter unit 140 ends abnormally, feedback data is generated based on an error message obtained by the execution of the interpreter unit 140. The source code acquisition unit 134 re-inputs the feedback data generated by the interpreter unit 400 into the natural language processing unit 135, and obtains corrected source code output from the natural language processing unit 135. This is repeated until execution by the interpreter unit 140 ends normally. That is, in this modified example, in addition to correcting the source code based on the abnormality information, the source code is also corrected based on the error message from the interpreter unit 140.

[0067] This ensures that more accurate source code for the operation of the robot 300 is generated. The source code may be corrected based on an error message from the interpreter unit 140 either before or after the simulation. The transmitter 137 may transmit the source code to the robot 300 when the simulation by the simulation unit 136 has ended normally and the execution result of the interpreter unit 140 has ended normally.

[0068] In the above-described embodiment, part of the processing performed by the robot motion generation support server 100 may be changed to be performed by the terminal device 200 or another device. In this case, the robot motion generation support server 100 and the terminal device 200 may constitute a robot motion generation support device or a robot motion generation support system 1.

[0069] Furthermore, the operational flows and operational examples in the above-described embodiments do not necessarily have to be executed in chronological order according to the order depicted in the flow diagrams or sequence diagrams. For example, steps in the operations may be executed in an order different from that depicted in the flow diagrams or sequence diagrams, or may be executed in parallel. Some steps in the operations may be deleted, or additional steps may be added to the processing. Furthermore, the operational flows and operational examples in the above-described embodiments may be executed independently, or two or more operational flows and operational examples may be combined and executed. For example, some steps of one operational flow may be added to another operational flow, or some steps of one operational flow may be replaced with some steps of another operational flow.

[0070] Furthermore, a robot motion generation support program may be provided that causes a computer to execute the motions according to the present embodiment described above. The robot motion generation support program may be recorded on a computer-readable medium. Using the computer-readable medium, it is possible to install the robot motion generation support program on a computer. Here, the computer-readable medium on which the robot motion generation support program is recorded may be a non-transitory recording medium. The non-transitory recording medium is not particularly limited, and may be, for example, a recording medium such as a CD-ROM or a DVD-ROM.

[0071] 1 Robot Motion Generation Support System 10 Network 100 Robot Motion Generation Support Server 110 Communication Unit 120 Storage Unit 121 Object Database 122 API Document 130 Processing Unit 131 Information Presentation Unit 131a Input Screen 132 Text Data Acquisition Unit 133 Synthesized Text Data Generation Unit 134 Source Code Acquisition Unit 135 Natural Language Processing Unit 136 Simulation Unit 137 Transmission Unit 140 Interpreter Unit 200 Terminal Device 300 Robot 301 Robot Hand

Claims

1. A robot operation generation support device for performing operations on an object, comprising: a storage means for storing API documents for robot control and information in an object database; a text data acquisition means for acquiring text data in which a user describes in natural language the content of a resultant obtained by the operation of the robot on the object; a synthetic text data generation means for adding the API documents for robot control and the information in the object database to the text data acquired by the text data acquisition means to generate synthetic text data; a source code acquisition means for inputting the synthetic text data generated by the synthetic text data means into a natural language processing unit and obtaining an output of the source code of the operation of the robot for obtaining the resultant from the natural language processing unit; a simulation means for performing a simulation of operating the robot in a virtual environment using the source code acquired by the source code acquisition means; a transmission means for transmitting the source code to the robot when the resultant is obtained in the simulation; and a source code correction means for re-inputting information necessary for correcting the source code into the natural language processing unit based on abnormal information including an error message detected during the simulation when the resultant is not obtained in the simulation and correcting the source code. A robot operation generation support device.

2. The robot operation generation support device according to claim 1, wherein the source code correction means generates feedback data based on the abnormal information by the simulation means, inputs the feedback data into the natural language processing unit by the source code acquisition means, and obtains an output of the corrected source code from the natural language processing unit.

3. The robot operation generation support device according to claim 2, wherein the simulation means performs a new simulation using the source code obtained by the source code correction means.

4. The apparatus further comprises interpreter means for executing the source code acquired by the source code acquisition means, and the interpreter means executes the source code acquired by the source code acquisition means before the simulation by the simulation means, and when the execution is normally terminated, the simulation means performs a simulation using the executed source code. The robot operation generation support apparatus according to claim 1.

5. The robot has a robot hand, the object is an ingredient, the API document includes information regarding gripping of the robot hand, movement of the robot hand, release of the robot hand, and layout information of the location of the destination of movement, and the resultant product is a food comprising a plurality of types of the ingredients. The robot operation generation support apparatus according to any one of claims 1 to 4.

6. A robot operation generation support system for a robot that performs operations on an object, comprising: a storage means for storing information on an API document for robot control and an object database; a text data acquisition means for acquiring text data in which a user describes in natural language the content of a resultant obtained by the operation of the robot on the object; a synthetic text data generation means for generating synthetic text data by adding the API document for robot control and the information of the object database to the text data acquired by the text data acquisition means; a source code acquisition means for inputting the synthetic text data generated by the synthetic text data means into a natural language processing unit and obtaining an output of the source code of the operation of the robot for obtaining the resultant from the natural language processing unit; a simulation means for performing a simulation of operating the robot in a virtual environment using the source code acquired by the source code acquisition means; a transmission means for transmitting the source code to the robot when the resultant is obtained in the simulation; and a source code correction means for re-inputting information necessary for correcting the source code into the natural language processing unit based on abnormal information including an error message detected during the simulation when the resultant is not obtained in the simulation and correcting the source code. A robot operation generation support system.

7. A robot operation generation support program for a robot that performs operations on an object, comprising: A storage means for storing information of the API document for robot control and the object database; A text data acquisition means for acquiring text data described in natural language by a user regarding the content of the resultant obtained by the operation of the robot on the object; A synthetic text data generation means for generating synthetic text data by adding the information of the API document for robot control and the object database to the text data acquired by the text data acquisition means; A source code acquisition means for inputting the synthetic text data generated by the synthetic text data means into a natural language processing unit and obtaining an output of the source code of the operation of the robot for obtaining the resultant from the natural language processing unit; A simulation means for performing a simulation of operating the robot in a virtual environment using the source code acquired by the source code acquisition means; A transmission means for transmitting the source code to the robot when the resultant is obtained in the simulation; A source code correction means for re-inputting information necessary for correcting the source code into the natural language processing unit based on abnormal information including an error message detected during the simulation and correcting the source code when the resultant is not obtained in the simulation. A robot operation generation support program.

Images