Controller, robot system, system, control method, and control program

Abstract

To provide a controller that enables a robot to work flexibly and properly as intended in response to user instructions in natural language, a robot system, a system, a control method, and a control program.SOLUTION: In a system 100, a controller 11 controls a robot 15 based on interactions with a user, the controller including: a function list storage unit 112a which stores a function list; a prerequisite storage unit 112d which stores a prerequisite that defines an operational procedure; an input unit 114 which receives, from the user, an instruction for the robot; an output unit 116 which outputs, to the user, a reaction to the instruction; an input buffer 112b which saves data of the instruction; and a processing unit 111 which transmits, to a natural language processing system 50, a prompt including the prerequisite, the data read out from the input buffer, and the function list, receives, from the natural language processing system, a response identifying one of the function modules to be executed, and generates an action command for the robot based on the response.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present invention relates to a technology for controlling a robot based on interaction with a user.

Background Art

[0002] In recent years, robots that can operate flexibly based on interaction with a user via instructions from the user in natural language have attracted attention.

[0003] Patent Document 1 describes a robot control device that can efficiently cause a robot to perform a target operation by simply instructing the target operation in a form close to natural language.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] However, in the robot control device described in Patent Document 1, by organizing a program file for performing a predetermined operation for each component and assigning an ID describing the operation content in natural language to each type of component, the user can only instruct the operation content to the robot control device. That is, in the robot control device described in Patent Document 1, the operation content can be instructed in a form close to natural language, but the target operation corresponding to the instruction is a predetermined one. Therefore, with the robot control device described in Patent Document 1, a robot cannot be controlled by a user's instruction in natural language, and since it is designed to cause the robot to perform a predetermined operation, the robot cannot be made to perform a flexible operation.

[0006] In view of the above problems, an object of the present invention is to provide a control device or the like that enables a robot to appropriately perform a flexible operation while performing an intended operation according to an instruction from a user in natural language.

Means for Solving the Problems

[0007] The present invention employs the following technical means to solve the above problems. The claims and the reference numerals in parentheses described in this section are an example showing the correspondence relationship with the specific means described in the embodiments to be described later as one aspect, and do not limit the technical scope of the present invention.

[0008] A control device (11) according to an aspect of the present invention is a control device for controlling a robot (15) based on an interaction with a user, wherein each of a plurality of predetermined operations to be executed by the robot is defined as a function module, and a function list storage unit (112a) that stores a function list of the function module; a precondition storage unit (112d) that stores preconditions for dividing the operations to be performed by the robot into a plurality of stages and defining the operation policies at each stage; an input unit (114) that receives an instruction from the user for the robot; an output unit (116) that outputs a response to the instruction to the user; an input buffer (112b) that stores the instruction data input from the input unit; and a processing unit (111) that transmits a prompt including the preconditions, the data read from the input buffer, and the function list to a natural language processing system (50) using a large language model, receives an answer from the natural language processing system as to which function module of the function list should be executed, and generates an operation command for the robot based on the answer.

[0009] In this way, by sending the instructions from the user, the preconditions, and the function list to the natural language processing system as a prompt, the processing unit can receive from the natural language processing system an answer as to which function list among the function lists should be executed, and can generate an operation instruction for the robot based on the answer. Since the function module defines the predetermined operations to be executed by the robot, the robot can be made to perform the intended operations.

[0010] A robot system (10) according to an aspect of the present invention includes the above-described control device (11) and the above-described robot (15).

[0011] A system (100) according to an aspect of the present invention includes the above-described control device, the above-described robot (15), and the above-described natural language processing system (50).

[0012] A control method as an aspect of the present invention is a control method for controlling a robot based on interaction with a user, wherein each of a plurality of predetermined operations to be executed by the robot is defined as a function module, and a function list reading step of reading the function list from a function list storage unit that stores the function list of the function module, a precondition reading step of reading the preconditions that divide the operations to be performed by the robot into a plurality of stages and define the operation policies at each stage from a precondition storage unit that stores the preconditions, an input step of receiving an instruction from the user for the robot, an output step of outputting a response to the instruction to the user, a storage step of storing the data of the instruction input from the input step in an input buffer, and a processing step of sending a prompt including the preconditions, the data read from the input buffer, and the function list to a natural language processing system using a large language model, receiving from the natural language processing system an answer as to which function module among the function lists should be executed, and generating an operation instruction for the robot based on the answer.

[0013] A control program as an aspect of the present invention is a control program for controlling a robot based on interaction with a user, which causes a computer to read a function list from a function list storage unit that stores a function list in which each of a plurality of predetermined operations to be executed by the robot is defined as a function module, a precondition reading step of reading the preconditions stored in a precondition storage unit that stores the preconditions for dividing the operations to be performed by the robot into a plurality of stages and defining the operation policies at each stage, an input step of receiving an instruction from the user for the robot, an output step of outputting a response to the instruction to the user, a storage step of storing the data of the instruction input from the input step in an input buffer, transmitting a prompt including the preconditions, the data read from the input buffer, and the function list to a natural language processing system using a large language model, receiving an answer from the natural language processing system as to which function module in the function list should be executed, and generating an operation instruction for the robot based on the answer, and executing it.

Advantages of the Invention

[0014] According to the present invention, it is possible to provide a control device or the like that enables a robot to appropriately perform a flexible operation while performing an intended operation according to an instruction from a user in natural language.

Brief Description of the Drawings

[0015]

Figure 1

Figure 2

Figure 3

Figure 4A

Figure 4B

Figure 5

Figure 6

Embodiments for Carrying Out the Invention

[0016] Hereinafter, embodiments of the present invention will be described with reference to the drawings. Note that the embodiments described below show examples of implementing the present invention and do not limit the present invention to the specific configurations described below. In implementing the present invention, specific configurations according to the embodiments may be appropriately adopted.

[0017] FIG. 1 is a diagram showing an example of the system configuration according to the present embodiment. The system 100 shown in FIG. 1 includes a robot system 10, a natural language processing system 50, and a search server 60. The robot system 10 and the natural language processing system 50 are connected so as to be able to transmit and receive data via a network. The robot system 10 and the search server 60 are connected so as to be able to transmit and receive data via a network.

[0018] The robot system 10 includes a control device 11, a mobile robot 12, a microphone 13, and a speaker 14. In this specification, the mobile robot 12, the microphone 13, and the speaker 14 are collectively referred to simply as the robot 15.

[0019] In the present embodiment, robot control will be described by taking as an example a case where a user orders a drink from a certain table to the robot 15 and the robot 15 brings the ordered drink to the table. When the user places an order for a drink with the robot 15, the user may ask for a recommended drink according to the weather or the like.

[0020] The natural language processing system 50 is a natural language processing system that utilizes a large language model. A large language model is a deep learning model generated through pre-training based on a vast amount of data, which is called a language model that models human spoken language, which is natural language, according to its occurrence probability. As a natural language processing system that utilizes a large language model, for example, GPT-3, GPT-3.5, and GPT-4 can be used. When the natural language processing system 50 receives a request, it statistically estimates the generation probability of the next word from the text included in the received request using the large language model, and transmits the estimation result to the requester.

[0021] The search server 60 performs a search process based on a search request transmitted from the robot system 10, and provides information corresponding to the search request by transmitting a search result indicating the result of such a search process to the robot system 10.

[0022] In the system 100 of the present embodiment, the robot system 10 transmits a prompt to inquire about the next operation of the robot 50 to the natural language processing system 50, and operates the robot 50 based on the response from the natural language processing system 50.

[0023] The control device 11 is a control device for controlling the robot 15 based on interaction with the user, and includes a processing unit 111, a storage unit 112, a communication unit 113, a voice input unit 114, a voice recognition unit 115, a voice synthesis unit 116, and a voice output unit 117. These are communicably connected to each other via, for example, a bus. In the present embodiment, the processing unit 111, the voice recognition unit 115, and the voice synthesis unit 116 are each configured separately, but the voice recognition unit 115 and the voice synthesis unit 116 may be part of the functional configuration of the processing unit 111.

[0024] The processing unit 111 is composed of, for example, a CPU (Central Processing Unit) and a RAM (Random Access Memory) used as the main memory device when the CPU executes processing. The CPU realizes various functions corresponding to the program by, for example, reading the program stored in the storage unit 112 into the RAM and executing it.

[0025] The processing unit 111 transmits a prompt to the natural language processing system 50 using, for example, a large language model, receives an answer from the natural language processing system 50, and generates an operation command for the robot 15 based on the answer.

[0026] The storage unit 112 is a so-called auxiliary storage device equipped with a non-volatile storage circuit such as an HDD (hard disk drive) and an SSD (solid state drive) that store various information. Note that the storage unit 112 may be a drive device or the like that reads and writes various information to and from portable storage media such as CD-ROMs, DVDs, and flash memories.

[0027] The communication unit 113 is realized by, for example, a circuit connected to a network. The communication unit 113 communicates with the natural language processing system 50 via the network.

[0028] The voice input unit 114 is connected to the microphone 13 that collects voice and outputs a voice signal, either wired or wirelessly, and receives the voice signal from the microphone 13.

[0029] The voice recognition unit 115 performs voice recognition processing on the voice signal from the voice input unit 114 and outputs the character information represented by the voice to the processing unit 111. For example, the voice recognition unit 115 can use a voice recognition technology from a voice signal to character information that utilizes deep learning technology.

[0030] The voice synthesis unit 116 synthesizes voice from the operation command of the robot 15 generated by the processing unit 111 and outputs it to the voice output unit 117. The voice synthesis unit 116 can use general voice synthesis technology.

[0031] The voice output unit 117 is connected to a speaker 14 that converts a voice signal into voice and outputs it to the outside, either wired or wirelessly, and outputs the voice signal from the voice synthesis unit 116 to the speaker 14.

[0032] The mobile robot 12 includes a robot control unit 121, a right wheel motor 122, a left wheel motor 123, a camera 124, and a sensor 125.

[0033] The robot control unit 121 receives the operation command of the robot 15 generated by the processing unit 111, and controls each part of the mobile robot 12 according to the operation command. Specifically, this robot control device 121 is a computer including hardware such as an arithmetic device such as a CPU, a main storage device such as a semiconductor memory, an auxiliary storage device such as a hard disk, and a communication device.

[0034] The right wheel motor 122 and the left wheel motor 123 are electric actuators that respectively rotate and drive a right wheel and a left wheel (not shown) provided on the mobile robot. The rotation speeds of the right wheel motor 122 and the left wheel motor 123 are respectively changed by the amount of electric power supplied from a battery (not shown) via an inverter (not shown). Therefore, the rotation speeds of the right wheel and the left wheel can be made different, and the mobile robot 12 can be turned by the speed difference between the right wheel and the left wheel.

[0035] The camera 124 acquires an image around the mobile robot 12. The sensor 125 includes, for example, a sensor that detects the temperature and humidity around the mobile robot 12, an infrared sensor, and the like.

[0036] Hereinafter, the function list storage unit 112a, the input buffer 112b, the history log storage unit 112c, and the prerequisite storage unit 112d, which are storage areas provided in the storage unit 112, will be described.

[0037] (Function List) The storage unit 112 includes a function list storage unit 112a that stores a function list. Here, the function list is a list having a plurality of skill packages obtained by packaging general-purpose operations of the robot 15 in units of skills. That is, the function list is a list having a plurality of function modules that define each of a plurality of predetermined operations to be executed by the robot 15 as a function module.

[0038] Currently, when trying to generate a program (for example, low-code) for the robot 15 from scratch by sending a request in natural language to the natural language processing system 50, it is difficult to generate a correct program. For this reason, simply sending a request in natural language to the natural language processing system 50 cannot cause the robot 15 to perform the intended operation.

[0039] Therefore, in the present embodiment, in addition to an instruction in natural language from the user, the processing unit 111 sends the natural language processing system 50 including the function list stored in the function list storage unit 112a as a prompt.

[0040] Thereby, the natural language processing system 50 can determine (or specify) a function module to be executed by the robot 15 based on the instruction of the user from the function list. Since the function module defines a predetermined operation to be executed by the robot 15 as described above, the processing unit 111 can generate an operation command for the intended robot 15 based on the response from the natural language processing system 50 regarding which function module in the function list should be executed. Therefore, the robot 15 can appropriately perform the intended operation.

[0041] In addition, some functional modules at least include parameters related to coordinates necessary for the operation of the robot 15 or parameters related to the text to be output to the user. In such a case, the processing unit 111 transmits the parameters used in each functional module to the natural language processing system 50 to be included in the function list. Then, the processing unit 111 receives, from the natural language processing system 50, data of the parameters to be given to the functional module together with the data for identifying the functional module, and generates an operation command for the robot 15 based on the identified functional module and the data of the parameters to be given to the functional module. Even in this case, since the data for identifying the functional module and the data of the parameters to be given to the functional module are transmitted as a response from the natural language processing system 50, the operation command for the robot 15 is limited to the functional module, and the robot can be appropriately operated using the data of the parameters corresponding thereto.

[0042] FIG. 2 is a diagram showing an example of a function list. The functional module 2a is a functional module that converts the text designated for the speaker 14 into voice and outputs it. The functional module 2b is a functional module that moves the mobile robot 12 to the designated coordinates (target_location). The functional module 2c is a functional module that checks the ordered table. The functional module 2d is a functional module that performs a waiting operation. The functional module 2e is a functional module for performing an Internet search using the search engine Bing (registered trademark). The functional module 2f is a functional module for performing Azure (registered trademark) Cognitive Search. Thus, some of the functional modules require parameters to be specified.

[0043] (History log) The storage unit 112 further includes an input buffer 112b that stores the input data and a history log storage unit 112c that accumulates the history log.

[0044] First, data of an instruction from a user input via the voice input unit 114 is stored in the input buffer 112b. Next, every time there is a response from the robot 15 that has operated according to the operation instruction generated by the processing unit 111, the response is overwritten by the processing unit 111 in the input buffer 112b. The response from the robot 15 means the operation result (so-called return value) of the robot 15 having executed the operation instruction. The history log is log data composed of data read from the input buffer 112b and a response from the natural language processing system 50 corresponding to the read data. Thus, in the history log storage unit 112c, a set consisting of an input to the natural language processing system 50 and an output for the input is appended as a history log.

[0045] Currently, the natural language processing system 50 can return only one response to one query. Therefore, since the natural language processing system 50 can return only one response to the processing unit 111 when receiving one instruction from the user, the processing unit 111 can generate only one operation instruction for the robot 15 for one instruction from the user. For this reason, the robot 15 cannot be continuously operated with one instruction from the user.

[0046] Therefore, in the present embodiment, after transmitting an instruction in natural language from the user, the processing unit 111 transmits to the natural language processing system 50 including the history log accumulated in the history log storage unit 112c in addition to the data read from the input buffer 112b, in the prompt.

[0047] As a result, a response based on the data read from the input buffer 112b and the history log is sent from the natural language processing system 50, so the processing unit 111 can generate an operation instruction based on the history log. Consider, for example, the case where an instruction is first issued to the robot 15 to make it operate. In this case, the data initially stored in the input buffer 112b is an instruction from the user. At this stage, no data is accumulated in the history log. Therefore, the natural language processing system 50 outputs the first response to the processing unit 111 based on the instruction from the user read from the input buffer 112b. In the history log, the data read from the input buffer 112b and the response from the natural language processing system 50 corresponding to the read data are added as a set, so the instruction from the user and the first response corresponding to the instruction are added to the history log. The processing unit 111 generates an operation instruction for the robot 15 based on the first response, and the robot 15 executes an operation according to the operation instruction. Then, the operation result of the robot 15 is overwritten in the input buffer 112b as a response, and the processing unit 111 sends the data read again from the input buffer 112b and the history log to the natural language processing system 50 including them in the prompt. Then, the natural language processing system 50 outputs the next response to the processing unit 111, and the processing unit 111 generates the next operation instruction.

[0048] In this way, the processing unit 111 can use the natural language processing system 50 to generate an operation instruction for the robot 15 considering the past history. Once the user gives an instruction to the robot 15, the processing unit 111 continues to send the prompt to the natural language processing system 50 until the natural language processing system 50 determines that the operation of the robot 15 is completed, so that the robot 15 can be made to perform continuous operations.

[0049] FIG. 3 is a diagram showing an example of a history log. The first line is log data indicating that the user gave an instruction "Make tea" to the robot 15 (specifically, the microphone 13). The second line is log data indicating that the natural language processing system 50 determined the function module 2a from the function list and determined "Please wait a moment" as the data of the parameter to be given to the function module 2a in response to the instruction "Make tea". Thus, the log data of the first line and the second line are respectively the input and output to the natural language processing system 50, and are added as a set to the history log. The third line is log data indicating that the function module 2a was executed and the operation result (so-called return value) that the speaker 14 output the voice "Please wait a moment" was returned. The fourth line is log data indicating that there was a response from the natural language processing system 50 when the prompt including the operation result of the third line was sent to the natural language processing system 50. The (n - 1)-th line is log data indicating that the function module 2a was executed and the operation result (so-called return value) that the speaker 14 output the voice "I have brought you the tea. Please take your time" was returned. The last line, the n-th line, is log data indicating that there was a response "Completion" of the operation of the robot 15 from the natural language processing system 50 when the prompt including the operation result of the (n - 1)-th line was sent to the natural language processing system 50. Due to the response "Completion" from the natural language processing system 50, the operation of the robot 15 is completed. The log data of the third line and the fourth line and the (n - 1)-th line and the n-th line are respectively the input and output to the natural language processing system 50, and are added as a set to the history log. Specifically, the "Completion" of the operation means the "Completion" of the flow in the flowchart described in the area 4b of the precondition described below.

[0050] The history log accumulates the history from the first instruction to the robot 15 as a set of inputs and outputs to the natural language processing system 50, enabling the natural language processing system 50 to understand what has been executed and to what extent. Specifically, it can be determined up to which part of the flowchart described in the precondition area 4b, which will be explained next, has been executed. By transmitting this data to the natural language processing system 50, an appropriate response regarding the actions to be taken next can be obtained from the natural language processing system 50, taking into account the past history.

[0051] (Preconditions) The storage unit 112 further includes a precondition storage unit 112d that stores the preconditions for the operation of the robot 15. The preconditions include the work procedure of the robot 15. Specifically, the work procedure of the robot 15 divides the actions to be performed by the robot 15 into multiple stages and defines the action guidelines for each stage. These action guidelines are described in the Unified Modeling Language (UML).

[0052] When the work procedure of the robot 15 is given to the natural language processing system 50 as a precondition, if the precondition is described in source code, the natural language processing system 50 will follow the source code, and although an accurate response can be obtained, a flexible response cannot be obtained. On the other hand, if the precondition is described in text using Japanese, English, etc., there may be ambiguous points in the expression, so a response that specifies the operation of the robot may not be obtained, and appropriate operation instructions cannot be generated based on such a response.

[0053] Therefore, in this embodiment, in addition to the instruction in natural language from the user, the processing unit 111 includes the preconditions stored in the precondition storage unit 112d in the prompt and transmits it to the natural language processing system 50.

[0054] As a prerequisite, since the operations to be performed by the robot 15 are divided into multiple stages and the operation guidelines for each stage are defined, the natural language processing system 50 can provide appropriate responses with flexibility according to the operation guidelines. Therefore, based on these responses, the processing unit 111 can generate operation commands that enable appropriate operations according to the operation guidelines while ensuring the flexibility of the operations of the robot 111.

[0055] FIGS. 4A and 4B are diagrams showing an example of the prerequisite conditions. The prerequisite conditions described in area 4a define the purpose. For example, in area 4a, it is defined to complete the flowchart (the part of "complete the flowchart" in area 4a). In area 4b, the code data representing the flowchart to be completed is described using UML (as an example, Mermaid). This flowchart corresponds to the above operation guidelines. Since it is defined to complete the flowchart in area 4a, the natural language processing system 50 outputs a response to complete this flowchart in response to an instruction from the user. Therefore, it is possible to provide the processing unit 111 with appropriate responses with flexibility according to the operation guidelines.

[0056] Here, the advantages of UML will be described. For example, in area 4b, there is a description of "Speak out something like ‘Brew coffee please’ or ‘A lot of water please’ or ‘A little water please’". As indicated by "something like", the natural language processing system 50 does not necessarily have to determine any one of "Brew coffee please", "A lot of water please", and "A little water please" as the data of the parameter, but can appropriately consider and determine the data of the parameter by itself in accordance with the purpose described in area 4a. Thereby, the natural language processing system 50 can give flexible responses while following the operation guidelines described in UML.

[0057] In area 4c, Target Location Information is described. This is an example of information about the environment in which robot 15 operates, and corresponds to the data of parameters related to the coordinates given to functional module 2b. For example, if table 1 is given to functional module 2b, it becomes an operation instruction to move the robot to table 1.

[0058] In area 4d, Remarks are described. Various contents such as conditions and restrictions may be described in the Remarks. For example, the Remarks include information about the region to which the place where robot 15 is used belongs (the part "This is a relaxation area located within a factory of DENSOCorporation in Aichi Prefecture." in area 4d). By including the information about the region to which the place where robot 15 is used belongs as prerequisite information, natural language processing system 50 can obtain various information such as weather information and event information in that region through Internet search. Thereby, a more flexible answer can be provided. Also, the Remarks include the description "Stick to the flowchart." If there is such a description, natural language processing system 50 will surely follow the operation policy. Therefore, an appropriate answer can be output efficiently. Also, the Remarks include the description "Think step by step." If there is such a description, natural language processing system 50 can think step by step, and can return an answer that enables robot 15 to operate appropriately without returning an answer with a logical leap.

[0059] Next, with reference to FIG. 5, the specific operation of processing unit 111 will be described. FIG. 5 is a flowchart for explaining the operation flow of processing unit 111.

[0060] In step S501, the processing unit 111 reads the function list stored in the function list storage unit 112a and the preconditions stored in the precondition storage unit 112d.

[0061] In step S502, the processing unit 111 determines whether there is an input from the voice recognition unit 115, that is, an instruction from the user via the microphone 13. If there is an input from the voice recognition unit 115, the flow proceeds to step S503. If there is no input from the voice recognition unit 115, the flow returns to step S502.

[0062] In step S503, the processing unit 111 overwrites the input from the voice recognition unit 115 in the input buffer 112b.

[0063] In step S504, the processing unit 111 transmits to the natural language processing system 50, in addition to the function list and preconditions read in step S501, the data overwritten in the input buffer 112b and the history log stored in the history log storage unit 112c. Note that if the history log has not been stored yet, empty data will be transmitted.

[0064] In step S505, the processing unit 111 adds the data in the input buffer 112b and the response from the natural language processing system 50 to the history log. Note that the data added to the history log in this step S505 is data in which the input to the natural language processing system 50 and the output for the input are set.

[0065] In step S506, the processing unit 111 branches the process based on the answer from the natural language processing system 50. If the answer from the natural language processing system 50 causes the speaker 14 to output voice (here, speech), that is, if the function module 2a in FIG. 2 is executed, the flow proceeds to step S507. If the answer from the natural language processing system 50 causes the mobile robot 12 to perform an operation (here, move or stop), that is, if the function module 2b or 2d in FIG. 2 is executed, the flow proceeds to step S509. If the answer from the natural language processing system 50 causes the search server 60 to perform a search, that is, if the function module 2e or 2f in FIG. 2 is executed, the flow proceeds to step S511. The branching in step S506 is an example, and there may be other cases where the answer from the natural language processing system 50 causes the processing unit 111 to check an ordered table, that is, if the function module 2c in FIG. 2 is executed. If the answer from the natural language processing system 50 is "completed", the program is terminated.

[0066] In step S507, the processing unit 111 transmits an operation command (here, a speech command) generated based on the answer from the natural language processing system 50 to the speech synthesis unit 116.

[0067] In step S508, the processing unit 111 overwrites the input buffer 112b with the reply from the speech synthesis unit 116 (for example, the result of speaking). Then the flow returns to step S504, and the processing unit 111 transmits the function list, preconditions, data overwritten in the input buffer 112b, and the history log to the natural language processing system 50. As a result, steps S504 to S510 are looped until the natural language processing system 50 determines that the operation of the robot 15 is completed due to the completion of the flow, and it becomes possible to execute continuous operations on the robot 15.

[0068] In step S509, the processing unit 111 transmits the operation command generated based on the answer from the natural language processing system 50 to the robot control unit 121.

[0069] In step S510, the processing unit 111 overwrites the reply (i.e., the operation result) from the robot control unit 121 in the input buffer 112b. Then the flow returns to step S504, and the processing unit 111 transmits the function list, preconditions, the data overwritten in the input buffer 112b, and the history log to the natural language processing system 50. As a result, steps S504 to S510 are looped until the natural language processing system 50 determines that the operation of the robot 15 is completed due to the completion of the flow, making it possible to execute continuous operations on the robot 15.

[0070] In step S511, the processing unit 111 transmits the search command generated based on the answer from the natural language processing system 50 to the search server 60. The search server 60 performs an Internet search using the search engine Bing (registered trademark) and Azure (registered trademark) Cognitive Search based on the search command.

[0071] In step S512, the processing unit 111 overwrites the search result from the search server 60 in the input buffer 112b. Then the flow returns to step S504, and the processing unit 111 transmits the function list, preconditions, the data overwritten in the input buffer 112b, and the history log to the natural language processing system 50. As a result, steps S504 to S510 are looped until the natural language processing system 50 determines that the operation of the robot 15 is completed due to the completion of the flow, making it possible to execute continuous operations on the robot 15.

[0072] In FIG. 5, the processing unit 111 has been described as an example of transmitting data in the input buffer, the function list, the history log, and the preconditions to the natural language processing system 50 including them in the prompt. However, in addition to the data in the input buffer 112b, the processing unit 111 may transmit at least one of these to the natural language processing system 50 including them in the prompt.

[0073] For example, when all of the data in the input buffer, the function list, the history log, and the preconditions are included in the prompt, it is possible to make the robot 15 perform flexible and appropriate operations by using the preconditions with the operation policy defined, and it is possible to make the robot 15 perform continuous operations considering the past history by looping the process using the history log. Furthermore, it is possible to make the robot 15 perform the intended operation by using the function list. Also, by using the function list, an answer as to which function module should be executed will be returned from the natural language processing system 50. Since data specifying the predefined function module is obtained as an answer, it can be accumulated as a history log in a form that can be used later. Therefore, the history log can be included in the prompt, and an appropriate next answer can be obtained.

[0074] Next, with reference to FIG. 6, the specific operation of the robot control unit 121 will be described. FIG. 6 is a flowchart showing the processing of the robot control unit 121.

[0075] In step S601, the robot control unit 121 determines whether there is an input from the processing unit 111. If there is an input from the processing unit 111, that is, if there is an operation command for the mobile robot 12, the flow proceeds to step S602. If there is no input from the processing unit 111, the flow returns to step S601.

[0076] In step S602, the robot control unit 121 executes the operation instructed in the operation command.

[0077] In step S603, the robot control unit 121 transmits the operation result to the processing unit 111.

[0078] (Other embodiments) As described above, the embodiments of the present disclosure have been described. However, the present disclosure is not construed as being limited to the above embodiments, and can be applied to various embodiments and combinations without departing from the gist of the present disclosure.

[0079] Also, the processing flow described in the above embodiments is merely an example, and unnecessary steps may be deleted, new steps may be added, or the processing order may be changed without departing from the gist of the present invention.

[0080] In the above embodiment, instructions were given to the robot 15 via the microphone 13. However, not only instructions by voice via the microphone 13 but also instructions by text information via a keyboard or a touch panel may be given to the robot 15. Further, a display may be used instead of the speaker 14 to provide text information to the user.

[0081] The following aspects may also be provided. (Aspect 1) A control device (11) for controlling a robot (15) based on interaction with a user, a function list storage unit (112a) that stores a function list of function modules, each of a plurality of predetermined operations to be executed by the robot being defined as a function module; a precondition storage unit (112d) that stores preconditions defining the operation policy at each stage by dividing the operation to be performed by the robot into a plurality of stages; an input unit (114) that receives an instruction from the user for the robot; an output unit (116) that outputs a response to the instruction to the user; an input buffer (112b) that stores the instruction data input from the input unit; For a natural language processing system (50) using a large language model, a prompt including the preconditions, data read from the input buffer, and the function list is sent, an answer is received from the natural language processing system as to which function module among the function list should be executed, and a processing unit (111) that generates an operation instruction for the robot based on the answer.

[0082] In this way, by sending the instruction from the user, the preconditions, and the function list as a prompt to the natural language processing system, the processing unit can receive an answer from the natural language processing system as to which function list among the function lists should be executed, and generate an operation instruction for the robot based on the answer. Since the function module defines a predetermined operation to be executed by the robot, the robot can be made to perform the intended operation.

[0083] (Aspect 2) The control device according to Aspect 1, wherein the processing unit receives data for specifying a function module from the natural language processing system, together with data of parameters to be given to the function module, and generates an operation instruction for the robot based on the specified function module and the data of the parameters.

[0084] (Aspect 3) The control device according to Aspect 2, wherein the data of the parameters includes at least data regarding coordinates necessary for the operation of the robot or data regarding the text to be output to the user.

[0085] (Aspect 4) The control device according to any one of Aspects 1 to 3, wherein the preconditions include information on the environment in which the robot operates.

[0086] (Aspect 5) The control device according to any one of Aspects 1 to 4, wherein the operation policy is described in UML.

[0087] (Aspect 6) The control device (11) according to any one of aspects 1 to 5, and the robot (15), a robot system (10) comprising the same.

[0088] (Aspect 7) The control device (11) according to any one of aspects 1 to 5, the robot (15), and the natural language processing system (50), a system (100) comprising the same.

[0089] (Aspect 8) A control method for controlling a robot based on interaction with a user, wherein each of a plurality of predetermined operations to be executed by the robot is defined as a functional module, and a functional list reading step of reading the functional list from a functional list storage unit that stores a functional list of the functional module; a precondition reading step of reading the preconditions from a precondition storage unit that stores the preconditions defining the operation policy at each stage by dividing the operation to be performed by the robot into a plurality of stages; an input step of receiving an instruction from the user for the robot; an output step of outputting a response to the instruction to the user; a storage step of storing the instruction data input from the input step in an input buffer; a processing step of transmitting a prompt including the preconditions, the data read from the input buffer, and the functional list to a natural language processing system using a large language model, receiving an answer from the natural language processing system as to which functional module among the functional lists should be executed, and generating an operation command for the robot based on the answer.

[0090] (Aspect 9) A control program for controlling a robot based on interaction with a user, causing a computer to, A function list reading step of reading the function list from a function list storage unit that stores a function list in which each of a plurality of predetermined operations to be executed by the robot is defined as a function module; A precondition reading step of reading the preconditions from a precondition storage unit that stores the preconditions for dividing the operations to be performed by the robot into a plurality of stages and defining the operation policies at each stage; An input step of receiving an instruction from the user for the robot; An output step of outputting a response to the instruction to the user; A saving step of saving the instruction data input from the input step in an input buffer; A processing step of transmitting a prompt including the preconditions, the data read from the input buffer, and the function list to a natural language processing system using a large language model, receiving an answer from the natural language processing system as to which function module among the function list should be executed, and generating an operation instruction for the robot based on the answer. A control program for executing.

Description of Symbols

[0091] 10... Robot system, 11... Control device, 12... Mobile robot, 13... Microphone, 14... Speaker, 15... Robot, 50... Natural language processing system, 111... Processing unit, 112... Storage unit, 112a... Function list storage unit, 112b... Input buffer, 112c... History log storage unit, 112d... Precondition storage unit, 114... Voice input unit, 116... Voice synthesis unit, 121... Robot control unit

Claims

1. A control device (11) for controlling a robot (15) based on an interaction with a user, comprising: a function list storage unit (112a) that stores a function list of function modules, each of a plurality of predetermined operations to be executed by the robot being defined as a function module; a precondition storage unit (112d) that stores preconditions defining an operation policy at each stage by dividing the operation to be performed by the robot into a plurality of stages; an input unit (114) that receives an instruction from the user for the robot; an output unit (116) that outputs a response to the instruction to the user; an input buffer (112b) that stores the instruction data input from the input unit; a processing unit (111) that transmits a prompt including the preconditions, the data read from the input buffer, and the function list to a natural language processing system (50) using a large language model, receives an answer from the natural language processing system as to which function module among the function list should be executed, and generates an operation command for the robot based on the answer.

2. The control device according to claim 1, wherein the processing unit receives data of parameters to be given to the function module together with data specifying the function module from the natural language processing system, and generates an operation command for the robot based on the specified function module and the data of the parameters.

3. The control device according to claim 2, wherein the data of the parameters includes at least data related to coordinates necessary for the operation of the robot or data related to the text to be output to the user.

4. The control device according to claim 1, wherein the preconditions include information on the environment in which the robot operates.

5. The control device according to claim 1, wherein the operation policy is described in UML.

6. A robot system (10) comprising the control device (11) according to any one of claims 1 to 5, and the robot (15).

7. A system (100) comprising the control device (11) according to any one of claims 1 to 5, the robot (15), and the natural language processing system (50).

8. A control method for controlling a robot based on an interaction with a user, comprising: A function list reading step of reading the function list from a function list storage unit that stores a function list of each of a plurality of predetermined operations to be executed by the robot, where each operation is defined as a function module; A precondition reading step of reading the preconditions from a precondition storage unit that stores the preconditions for dividing the operations to be performed by the robot into a plurality of stages and defining the operation policies for each stage; An input step of receiving an instruction from the user for the robot; An output step of outputting a response to the instruction to the user; A saving step of saving the data of the instruction input from the input step in an input buffer; A processing step of transmitting a prompt including the preconditions, the data read from the input buffer, and the function list to a natural language processing system using a large language model, receiving an answer from the natural language processing system as to which function module among the function list should be executed, and generating an operation instruction for the robot based on the answer.

9. A control program for controlling a robot based on interaction with a user, causing a computer to perform a function list reading step of reading the function list from a function list storage unit that stores a function list of each of a plurality of predetermined operations to be executed by the robot, where each operation is defined as a function module; perform a precondition reading step of reading the preconditions from a precondition storage unit that stores the preconditions for dividing the operations to be performed by the robot into a plurality of stages and defining the operation policies for each stage; perform an input step of receiving an instruction from the user for the robot; perform an output step of outputting a response to the instruction to the user; perform a saving step of saving the data of the instruction input from the input step in an input buffer; perform a processing step of transmitting a prompt including the preconditions, the data read from the input buffer, and the function list to a natural language processing system using a large language model, receiving an answer from the natural language processing system as to which function module among the function list should be executed, and generating an operation instruction for the robot based on the answer.

Images