Command conversion system, method, and program

Abstract

In this command conversion system, a temporal logic analysis means divides an outline command, which is a command represented by temporal logic and expressing an outline of states up to a target state in which a robot is operated, and generates a temporal logic sequence, which is a set of components of the temporal logic. On the basis of a predefined correspondence between the components of the temporal logic and generation character strings (e.g., text) which are character strings used to generate the arrangement of target objects, a character string conversion means converts each component of the temporal logic sequence into a generation character string according to said correspondence, and combines the generation character strings to generate a character string representing the content of the outline command.

Description

Command conversion system, method, and program

[0001] The present disclosure relates to a command conversion system, a command conversion method, and a command conversion program that convert commands given to a robot.

[0002] The introduction of robots is progressing in logistics sites and other areas. To automate various tasks, it is necessary to plan the robot's motion in accordance with work instructions. One motion planning technique is to plan the robot's motion based on commands in temporal logic. For example, Non-Patent Document 1 describes optimization-based task and motion planning (TAMP) using signal temporal logic (STL) specifications.

[0003] Non-Patent Document 2 describes the introduction of a web-scale diffusion model into robotics.

[0004] R. Takano, H. Oyama and M. Yamakita, "Continuous Optimization-Based Task and Motion Planning with Signal Temporal Logic Specifications for Sequential Manipulation," 2021 IEEE International Conference on Robotics and Automation (ICRA), Xi'an, China, 2021, pp. 8409-8415

[0005] Kapelyukh, Ivan and Vosylius, Vitalis and Johns, Edward. "DALL-E-Bot: Introducing Web-Scale Diffusion Models to Robotics", CoRL 2022 Workshop PRL Program Chairs, 17 Nov 2022, CoRL 2022 Workshop PRL Paper19 Decision

[0006] Commands written in temporal logic are difficult to understand, making it difficult to uniquely determine a goal state. For example, commands written in temporal logic, which is a formal language that only provides general work instructions without specifying the specific coordinates required for robot operation, make it difficult to uniquely determine a goal state.

[0007] Therefore, an object of the present disclosure is to provide a command conversion system, a command conversion method, and a command conversion program that can uniquely determine a target state from a command expressed in temporal logic.

[0008] The command conversion system according to the present disclosure is characterized by comprising: temporal logic analysis means for dividing a general command, which is a command that expresses an outline of a target state for operating a robot expressed in temporal logic, and generating a temporal logic sequence that is a collection of temporal logic components; and character string conversion means for converting each component of the temporal logic sequence into a generated character string that corresponds to the predefined correspondence between the temporal logic components and a generated character string that is a character string used to generate the placement of a target object, and combining the generated character strings to generate a character string that expresses the content of the general command.

[0009] The command conversion method according to the present disclosure is characterized in that it divides a general command, which is a command that expresses an outline of a target state for operating a robot expressed in temporal logic, to generate a temporal logic sequence that is a collection of temporal logic components, converts each component of the temporal logic sequence into a generated string that corresponds to the predefined correspondence between the temporal logic components and a generated string that is a string used to generate the placement of a target object, and combines the generated strings to generate a string that expresses the content of the general command.

[0010] The command conversion program according to the present disclosure is characterized in that it causes a computer to execute a temporal logic analysis process that divides a general command, which is a command that expresses an outline of a target state for operating a robot expressed in temporal logic, and generates a temporal logic sequence that is a collection of temporal logic components, and a string conversion process that converts each component of the temporal logic sequence into a generated string that corresponds to a predefined correspondence between the temporal logic components and a generated string that is a string used to generate the placement of a target object, and combines the generated strings to generate a string that expresses the content of the general command.

[0011] According to the present disclosure, commands expressed in temporal logic can be translated into a format that is understandable to a user.

[0012] Fig. 1 is a block diagram showing a configuration example of an embodiment of a command conversion system according to the present disclosure; Fig. 2 is a flowchart showing an operation example of the command conversion system; Fig. 3 is a block diagram showing a configuration example of another embodiment of the command conversion system according to the present disclosure; Fig. 4 is a flowchart showing another operation example of the command conversion system; Fig. 5 is a block diagram showing a configuration example of yet another embodiment of the command conversion system according to the present disclosure; Fig. 6 is a flowchart showing another operation example of the command conversion system; Fig. 7 is a block diagram showing an overview of the command conversion system; Fig. 8 is a schematic block diagram showing the configuration of a computer according to at least one embodiment.

[0013] First, an overview of the present disclosure will be described. As described above, commands are sometimes used that describe only general work instructions in a formal language without specifying specific coordinates required for the robot's operation. Hereinafter, such commands will be referred to as general commands. In other words, a general command is a command described in a formal language that does not describe a specific position at which the robot should be operated, but rather describes an outline of the operation of the robot to achieve a target state. However, a general command does not uniquely determine a target state.

[0014] One method for solving this problem is to determine the goal state in advance. For example, Non-Patent Document 2 describes a method for generating a goal state as an image. However, in order to generate a goal state that can be used when planning an action from the generated image, trial and error is required to generate the correct input text.

[0015] Therefore, in this disclosure, a method for uniquely determining a goal state from a general command in temporal logic, which is one of formal language systems, will be described. Hereinafter, an embodiment of the present disclosure will be described with reference to the drawings.

[0016] [First embodiment] Fig. 1 is a block diagram showing a configuration example of a first embodiment of a command conversion system according to the present disclosure. The command conversion system 1 shown in Fig. 1 includes a storage unit 10, an input unit 20, a temporal logic analysis unit 30, a character string conversion unit 40, and an output unit 21.

[0017] The storage unit 10 stores information used by the command conversion system 1 of this embodiment when performing various processes. The storage unit 10 of this embodiment also stores a correspondence between the components of the temporal logic described above and a character string (hereinafter referred to as a generated character string) used to generate the arrangement of the target object. The correspondence is determined in advance by a user or the like so that the target state can be uniquely identified from the components of the temporal logic. The generated character string and the content of the correspondence will be described later. The storage unit 10 is realized, for example, by a magnetic disk or the like. The storage unit 10 may also store a general command, which will be described later.

[0018] The input unit 20 receives an input of a general command expressed in temporal logic. The input unit 20 may also receive an input of a general command stored in the storage unit 10.

[0019] The temporal logic analysis unit 30 divides the general command expressed in temporal logic and outputs a temporal logic sequence, which is a set of temporal logic components. Since temporal logic is a logic that mathematically describes properties related to trajectories using general logical operators (logical AND, logical OR, implication, negation, etc.) and temporal operators, the components of temporal logic include combinations of logical operators and temporal operators.

[0020] The temporal logic analysis unit 30 outputs a temporal logic sequence by dividing the global command based on the priority of operators and parentheses. In doing so, the temporal logic analysis unit 30 divides the global command according to the components of the temporal logic defined in the correspondence relationships described above. For example, the temporal logic analysis unit 30 may divide the global command so that it matches the components of the temporal logic defined in the correspondence relationships.

[0021] The character string conversion unit 40 converts the divided temporal logic sequence into generated character strings corresponding to the components. As described above, the correspondence between the components of temporal logic and the generated character strings is predefined. Therefore, the character string conversion unit 40 identifies the generated character strings corresponding to the predefined correspondence from each component of the temporal logic sequence, and combines the generated character strings to generate a character string indicating the content of the general command.

[0022] The generated character string is defined in a manner necessary for the character string conversion unit 40 to generate a character string indicating the content of the general command. For example, when the character string conversion unit 40 generates a character string using a large language model (LLM), the generated character string may be defined as text expressed in a natural language or program code expressed in a predetermined programming language. Furthermore, the character string generated in this manner can also be considered information indicating the specific command content (such as position and orientation) for operating the robot.

[0023] The output unit 21 outputs the converted character string.

[0024] The input unit 20, the temporal logic analysis unit 30, the character string conversion unit 40, and the output unit 21 are realized by a computer processor (e.g., a CPU (Central Processing Unit), a GPU (Graphics Processing Unit)) that operates according to a program (a command conversion program).

[0025] For example, the program may be stored in the storage unit 10 of the command conversion system 1, and the processor may read the program and operate in accordance with the program as the input unit 20, the temporal logic analysis unit 30, the character string conversion unit 40, and the output unit 21. Furthermore, the functions of the command conversion system 1 may be provided in the form of SaaS (Software as a Service).

[0026] Furthermore, the input unit 20, the temporal logic analysis unit 30, the character string conversion unit 40, and the output unit 21 may each be realized by dedicated hardware. Furthermore, some or all of the components of each device may be realized by general-purpose or dedicated circuits, processors, etc., or a combination of these. These may be configured by a single chip, or by multiple chips connected via a bus. Some or all of the components of each device may be realized by a combination of the above-mentioned circuits, etc., and a program.

[0027] Furthermore, when some or all of the components of the command conversion system 1 are realized by a plurality of information processing devices, circuits, etc., the plurality of information processing devices, circuits, etc. may be centrally or decentralized. For example, the information processing devices, circuits, etc. may be realized as a client-server system, a cloud computing system, or the like, in a form in which each is connected via a communication network.

[0028] Next, the operation of the command conversion system 1 of this embodiment will be described. Fig. 2 is a flowchart showing an example of the operation of the command conversion system 1 of this embodiment. The temporal logic analysis unit 30 divides the general command to generate a temporal logic sequence (step A1). The character string conversion unit 40 converts each component of the temporal logic sequence into generated character strings based on predefined correspondence relationships (step A2). The character string conversion unit 40 combines these generated character strings to generate a character string representing the content of the general command (step A3).

[0029] As described above, in this embodiment, the temporal logic analysis unit 30 divides the general command to generate a temporal logic sequence, and the character string conversion unit 40 converts each component of the temporal logic sequence into a generated character string that corresponds to the correspondence based on a predefined correspondence, and combines the generated character strings to generate a character string that represents the content of the general command. Therefore, a goal state can be uniquely determined from a command expressed in temporal logic.

[0030] [Second Embodiment] Next, a second embodiment of the command conversion system according to the present disclosure will be described. Fig. 3 is a block diagram showing an example configuration of the second embodiment of the command conversion system according to the present disclosure. The command conversion system 2 shown in Fig. 3 includes a storage unit 10, an input unit 20, a temporal logic analysis unit 30, a character string conversion unit 41, an image acquisition unit 50, an image processing unit 60, a layout generation unit 70, and an output unit 22.

[0031] That is, the command conversion system 2 of this embodiment differs from the command conversion system 1 of the first embodiment in that it further includes an image acquisition unit 50, an image processing unit 60, and a layout generation unit 70, and includes an output unit 22 and a character string conversion unit 41 instead of the output unit 21 and the character string conversion unit 40. The rest of the configuration is the same as that of the first embodiment.

[0032] The general command in this embodiment also includes a symbol that corresponds to a target object detected from an image by an image processing unit 60, which will be described later.

[0033] The image acquisition unit 50 is a device that acquires an image including a target object to be moved by operating the robot. The image acquisition unit 50 may be realized by a device that acquires images, such as a camera. The image acquisition unit 50 may also be a device that outputs an image selected by a user or the like.

[0034] The image processing unit 60 detects a target object included in an image. Specifically, the image processing unit 60 estimates and outputs the type of target object included in the image acquired by the image acquisition unit 50. The image processing unit 60 may also estimate not only the type of target object but also the orientation of the target object. Note that methods for estimating the type and orientation of an object in an image are widely known, and detailed description thereof will be omitted here.

[0035] The character string converter 41 converts the divided temporal logic string into generated character strings corresponding to the components, similar to the character string converter 40 of the first embodiment. Furthermore, the character string converter 41 of this embodiment converts the detected target objects into character strings corresponding to symbols.

[0036] The placement generation unit 70 generates a placement of target objects representing the content of the general command from the character string generated by the character string conversion unit 40 and the target objects detected by the image processing unit 60. The content of the placement generated by the placement generation unit 70 is arbitrary and may be an image or text. For example, the placement generation unit 70 may generate an image resulting from the placement of the target objects based on the content of the general command.

[0037] The method by which the placement generation unit 70 generates the placement may also be arbitrary. For example, the placement generation unit 70 may generate the placement using a trained model. In this case, the placement generation unit 70 may generate an image using a generation technique such as a diffusion model for the model.

[0038] The input unit 20, the temporal logic analysis unit 30, the character string conversion unit 41, the image acquisition unit 50, the image processing unit 60, the layout generation unit 70, and the output unit 22 are realized by a computer processor that operates according to a program (command conversion program).

[0039] Next, an operation of the command conversion system 2 of this embodiment will be described. Fig. 4 is a flowchart showing an example of the operation of the command conversion system 2 of this embodiment. In this example of operation, it is assumed that the general command includes a symbol that corresponds to a target object detected from an image.

[0040] The image acquisition unit 50 acquires an image including a target object (step C1). The image processing unit 60 detects the target object included in the image (step C2). The temporal logic analysis unit 30 divides the general command to generate a temporal logic sequence (step C3). The character string conversion unit 41 converts each component of the temporal logic sequence into a generated character string that corresponds to a symbol for the target object detected from the image (step C4). The character string conversion unit 41 combines these generated character strings to generate a character string representing the content of the general command (step C5). The layout generation unit 70 then generates a layout of the target objects representing the content of the general command from the generated character string and the detected target object (step C6).

[0041] As described above, in this embodiment, the image acquisition unit 50 acquires an image including a target object, and the image processing unit 60 detects the target object included in the image. Then, the character string conversion unit 41 converts the target object detected from the image into a generated character string corresponding to a symbol, and the layout generation unit 70 generates a layout of the target object representing the content of the general command from the generated character string and the detected target object. Therefore, in addition to the effects of the first embodiment, a layout that satisfies the general command can be generated.

[0042] [Third Embodiment] Next, a third embodiment of the command conversion system according to the present disclosure will be described. Fig. 5 is a block diagram showing a configuration example of the third embodiment of the command conversion system according to the present disclosure. The command conversion system 3 shown in Fig. 5 includes a storage unit 10, an input unit 20, a temporal logic analysis unit 30, a character string conversion unit 41, an image acquisition unit 50, an image processing unit 60, a placement generation unit 70, an operation planning unit 80, and an output unit 23.

[0043] That is, the command conversion system 3 of this embodiment differs from the command conversion system 2 of the second embodiment in that it further includes an action planning unit 80 and an output unit 23 instead of the output unit 22. Other configurations are the same as those of the second embodiment.

[0044] The motion planning unit 80 generates a motion plan for operating the robot from the character string generated by the placement generation unit 70. The motion plan may be generated in a natural language that humans can recognize, or may be generated as a control code for directly controlling the robot. Note that methods for generating motion plans for various devices from character strings are widely known, and therefore detailed explanations will be omitted here. The motion planning unit 80 may also control the motion of the robot based on the generated motion plan.

[0045] The input unit 20, the temporal logic analysis unit 30, the character string conversion unit 41, the image acquisition unit 50, the image processing unit 60, the placement generation unit 70, the motion planning unit 80, and the output unit 23 are realized by a computer processor that operates according to a program (command conversion program).

[0046] Next, the operation of the command conversion system 3 of this embodiment will be described. Fig. 6 is a flowchart showing an example of the operation of the command conversion system 3 of this embodiment. The process from acquiring an image to generating a target object arrangement that satisfies the general command is the same as the process from step C1 to step C6 illustrated in Fig. 4. Thereafter, the action planning unit 80 generates an action plan for operating the robot from the generated character string (step D1).

[0047] As described above, in this embodiment, the motion planning unit 80 generates a motion plan for operating the robot from the generated character string. Therefore, in addition to the effect of the second embodiment, it is possible to convert a general command into a motion plan for operating the robot.

[0048] The present disclosure will be described below using specific examples, but the scope of the present disclosure is not limited to the contents described below.

[0049] The input unit 20 receives input of the following general commands, examples of which are expressed in temporal logic.

[0050]

[0051] The temporal logic analysis unit 30 divides the general command and outputs the temporal logic sequence shown below.

[0052]

[0053] For example, it is assumed that the correspondence between the components of temporal logic and the generated character strings is predefined as shown below.

[0054]

[0055] In this case, the character string conversion unit 40 converts the divided temporal logic string into a generated character string corresponding to the components based on the above correspondence relationship, thereby generating the following example character string (sentence) that represents the content of the general command: "Between time 1 and time T, place object x1 in area H, and also place object x2 in area H."

[0056] Next, an overview of the present disclosure will be described. Fig. 7 is a block diagram showing an overview of a command conversion system according to the present disclosure. A command conversion system 90 (e.g., command conversion systems 1, 2, and 3) according to the present disclosure includes: temporal logic analysis means 91 (e.g., the temporal logic analysis unit 30) that divides a general command, which is a command that expresses a general outline of a target state for operating a robot, expressed in temporal logic, and generates a temporal logic sequence that is a collection of temporal logic components; and character string conversion means 92 (e.g., the character string conversion unit 40) that converts each component of the temporal logic sequence into a generated character string that corresponds to a predefined correspondence between the temporal logic components and a generated character string (e.g., text) that is a character string used to generate the placement of a target object, and combines the generated character strings to generate a character string that represents the content of the general command.

[0057] With such a configuration, the goal state can be uniquely determined from commands expressed in temporal logic.

[0058] The command conversion system 90 may also include image acquisition means (e.g., image acquisition unit 50) for acquiring an image including a target object to be moved by operating the robot, image processing means (e.g., image processing unit 60) for detecting the target object included in the image, and arrangement generation means (e.g., arrangement generation unit 70) for generating an arrangement of the target objects representing the content of the general command from the generated character string and the detected target object. Here, the general command may include a symbol that corresponds to the target object detected from the image, and the character string conversion means 92 may convert the target object detected from the image into a character string that corresponds to the symbol.

[0059] Furthermore, the arrangement generating means may generate an image resulting from arranging the target objects based on the content of the general command.

[0060] The command conversion system 90 may also include an action planning means (for example, an action planning unit 80) that generates an action plan for operating the robot from the generated character string.

[0061] Furthermore, the motion planning means may control the motion of the robot based on the generated motion plan.

[0062] Additionally, the building blocks of temporal logic may include combinations of logical and temporal operators.

[0063] 8 is a schematic block diagram showing the configuration of a computer according to at least one embodiment. The computer 1000 includes a processor 1001, a main memory device 1002, an auxiliary memory device 1003, and an interface 1004. The computer 1000 may also be connected to a computer that executes a mathematical programming solver, an annealing machine, a simulator, or the like.

[0064] The above-described command conversion system 90 is implemented in a computer 1000. The operations of the above-described processing units are stored in the form of a program (command conversion program) in an auxiliary storage device 1003. The processor 1001 reads the program from the auxiliary storage device 1003, loads it into the main storage device 1002, and executes the above-described processing in accordance with the program.

[0065] In at least one embodiment, the auxiliary storage device 1003 is an example of a non-transitory tangible medium. Other examples of non-transitory tangible media include a magnetic disk, a magneto-optical disk, a CD-ROM (Compact Disc Read-only memory), a DVD-ROM (Read-only memory), and a semiconductor memory connected via the interface 1004. In addition, when this program is distributed to the computer 1000 via a communication line, the computer 1000 that receives the program may load the program into the main storage device 1002 and execute the above-described processing.

[0066] The program may be a program for realizing part of the above-described functions. Furthermore, the program may be a so-called differential file (differential program) that realizes the above-described functions in combination with another program already stored in the auxiliary storage device 1003.

[0067] Although the present disclosure has been described above with reference to the embodiments, the present disclosure is not limited to the above-described embodiments. Various modifications that can be understood by those skilled in the art can be made to the configuration and details of the present disclosure within the scope of the present disclosure. Furthermore, each embodiment can be combined with other embodiments as appropriate.

[0068] The present disclosure is preferably applied to a command conversion system that converts commands given to a robot. For example, the present disclosure can be applied to a robot that packs boxes into containers.

[0069] 1, 2, 3 Command conversion system 10 Memory unit 20 Input unit 21, 22, 23 Output unit 30 Temporal logic analysis unit 40, 41 Character string conversion unit 50 Image acquisition unit 60 Image processing unit 70 Placement generation unit 80 Motion planning unit

Claims

1. A command conversion system comprising: temporal logic analysis means for dividing a general command, which is a command that expresses an outline of a target state for operating a robot expressed in temporal logic, and generating a temporal logic sequence that is a collection of components of the temporal logic; and character string conversion means for converting each component of the temporal logic sequence into a generated character string that corresponds to a predefined correspondence between the components of the temporal logic and a generated character string that is a character string used to generate the placement of a target object, and combining the generated character strings to generate a character string that expresses the content of the general command.

2. A command conversion system as described in claim 1, comprising: an image acquisition means for acquiring an image including a target object to be moved by operating a robot; an image processing means for detecting the target object included in the image; and an arrangement generation means for generating an arrangement of the target object representing the content of a general command from a generated character string and the detected target object, wherein the general command includes a symbol that corresponds to the target object detected from the image, and a character string conversion means for converting the target object detected from the image into a character string that corresponds to the symbol.

3. The command conversion system according to claim 2, wherein the arrangement generating means generates an image resulting from arranging the target objects based on the contents of the general command.

4. A command conversion system according to any one of claims 1 to 3, comprising an operation planning means for generating an operation plan for operating the robot from the generated character string.

5. A command conversion system according to claim 4, wherein the motion planning means controls the motion of the robot based on the generated motion plan.

6. A command conversion system according to any one of claims 1 to 3, wherein the components of temporal logic include a combination of logical operators and temporal operators.

7. A command conversion method comprising: dividing a general command, which is a command that expresses an outline of a target state for operating a robot expressed in temporal logic, to generate a temporal logic sequence that is a collection of components of the temporal logic; converting each component of the temporal logic sequence into a generated string that corresponds to a predefined correspondence between the components of the temporal logic and a generated string that is a string used to generate the placement of a target object; and combining the generated strings to generate a string that expresses the content of the general command.

8. A command conversion method as claimed in claim 7, comprising the steps of: acquiring an image including a target object to be moved by operating a robot; detecting the target object included in the image; converting the target object detected from the image into a character string corresponding to the character string; and generating an arrangement of the target object representing the content of the command from the generated character string and the detected target object.

9. A command conversion program for causing a computer to execute a temporal logic analysis process that divides a general command, which is a command that represents an outline of a target state for operating a robot expressed in temporal logic, and generates a temporal logic sequence that is a collection of components of the temporal logic, and a string conversion process that converts each component of the temporal logic sequence into a generated string that corresponds to a predefined correspondence between the components of temporal logic and a generated string that is a string used to generate the placement of a target object, and combines the generated strings to generate a string that represents the content of the general command.

10. A command conversion program as claimed in claim 9, which causes a computer to execute an image acquisition process for acquiring an image including a target object to be moved by operating a robot, an image process for detecting the target object included in the image, and an arrangement process for generating an arrangement of the target object representing the content of a general command from a generated character string and the detected target object, wherein the general command includes a symbol that corresponds to the target object detected from the image, and a character string conversion process converts the target object detected from the image into a character string that corresponds to the symbol.

Images