Controller, robot system, system, control method, and control program

Abstract

To provide a controller which causes a robot to act successively considering past actions, a robot system, a system, a control method, and a control program.SOLUTION: In a system 100, a controller 11 controls a robot 15 based on interactions with a user, the controller including: an input unit 114 which receives, from the user, an instruction for a robot; an output unit 116 configured to output, to the user, a reaction to the instruction; an input buffer 112b which saves data of the input instruction; a history log storage unit 112c which accumulates data read from the input buffer and history log; and a processing unit 111 which transmits a prompt to a natural language processing system, the prompt including the data read from the input buffer and the history log, receives a response from the natural language processing system, and generates an action command for the robot based on the response. The processing unit overwrites the data in the input buffer with a result of an action of the robot that acted according to the action command.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present invention relates to a technology for controlling a robot based on interaction with a user.

Background Art

[0002] In recent years, robots that can operate flexibly based on interaction with a user via instructions from the user in natural language have attracted attention.

[0003] Patent Document 1 describes a robot control device that can efficiently cause a robot to perform a target operation by simply instructing the target operation in a form close to natural language.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] However, in the robot control device described in Patent Document 1, a program file for performing a predetermined operation is sorted by component, and an ID describing the operation content in natural language is assigned to each type of component, so that the user can only instruct the operation content to the robot control device. That is, in the robot control device described in Patent Document 1, the operation content can be instructed in a form close to natural language, but the target operation corresponding to the instruction is a predetermined one. Therefore, in the robot control device described in Patent Document 1, the robot cannot be controlled by a user's instruction in natural language, and only a single predetermined operation can be caused to be performed by the robot for a single instruction, and the robot cannot be caused to perform a continuous operation considering the past history.

[0006] In view of the above problems, an object of the present invention is to provide a control device or the like that enables a robot to perform continuous operations considering past circumstances.

Means for Solving the Problems

[0007] The present invention employs the following technical means to solve the above problems. The scope of the claims and the reference numerals in parentheses described in this section are an example showing the correspondence with the specific means described in the embodiments to be described later as one aspect, and do not limit the technical scope of the present invention.

[0008] A control device (11) according to an aspect of the present invention is a control device (11) for controlling a robot (15) based on interaction with a user, comprising an input unit (114) that receives an instruction from the user for the robot, an output unit (116) that outputs a response to the instruction to the user, an input buffer (112b) that stores the instruction data input from the input unit, a history log storage unit (112c) that accumulates a history log consisting of the data read from the input buffer and the answer from a natural language processing system (50) using a large language model corresponding to the read data, and a processing unit (111) that transmits a prompt including the data read from the input buffer and the history log to the natural language processing system, receives the answer from the natural language processing system, and generates an operation command for the robot based on the answer. The processing unit overwrites the operation result from the robot that has operated according to the operation command in the input buffer.

[0009] In this way, the processing unit can generate an operation command for the robot considering past circumstances by transmitting a prompt including the history log to the natural language processing system. Once the user gives an instruction to the robot, the processing unit continuously transmits the prompt to the natural language processing system until the natural language processing system 50 determines that the operation of the robot 15 is completed, enabling the robot to perform continuous operations.

[0010] A robot system (10) according to one aspect of the present invention includes the above-described control device (11) and the above-described robot (15).

[0011] A system (100) according to one aspect of the present invention includes the above-described control device, the above-described robot (15), and the above-described natural language processing system (50).

[0012] A control method as one aspect of the present invention is a control method for controlling a robot based on interaction with a user, the control method including: an input step of receiving an instruction from the user for the robot; an output step of outputting a response to the instruction to the user; a storage step of storing data of the instruction input from the input step in an input buffer; an accumulation step of accumulating a history log including data read from the input buffer and an answer from a natural language processing system using a large language model corresponding to the read data; a processing step of transmitting a prompt including the data read from the input buffer and the history log to the natural language processing system, receiving an answer from the natural language processing system, and generating an operation instruction for the robot based on the answer, wherein the processing step overwrites the input buffer with an operation result from the robot that has operated according to the operation instruction.

[0013] A control program as one aspect of the present invention is a control program for controlling a robot based on interaction with a user, causing a computer to execute an input step of receiving an instruction from the user for the robot, an output step of outputting a response to the instruction to the user, a storage step of storing the instruction data input from the input step in an input buffer, an accumulation step of accumulating a history log including the data read from the input buffer and a response from a natural language processing system using a large language model corresponding to the read data, a processing step of transmitting a prompt including the data read from the input buffer and the history log to the natural language processing system, receiving a response from the natural language processing system, and generating an operation instruction for the robot based on the response, and the processing step overwrites the input buffer with an operation result from the robot operating according to the operation instruction.

Advantages of the Invention

[0014] According to the present invention, it is possible to provide a control device or the like that enables a robot to perform continuous operations considering past circumstances.

Brief Description of the Drawings

[0015]

Figure 1

Figure 2

Figure 3

Figure 4A

Figure 4B

Figure 5

Figure 6

Embodiments for Carrying Out the Invention

[0016] Hereinafter, embodiments of the present invention will be described with reference to the drawings. Note that the embodiments described below show examples of implementing the present invention, and do not limit the present invention to the specific configurations described below. In implementing the present invention, specific configurations according to the embodiments may be appropriately adopted.

[0017] FIG. 1 is a diagram showing an example of the system configuration according to the present embodiment. The system 100 shown in FIG. 1 includes a robot system 10, a natural language processing system 50, and a search server 60. The robot system 10 and the natural language processing system 50 are connected so as to be able to transmit and receive data via a network. The robot system 10 and the search server 60 are connected so as to be able to transmit and receive data via a network.

[0018] The robot system 10 includes a control device 11, a mobile robot 12, a microphone 13, and a speaker 14. In this specification, the mobile robot 12, the microphone 13, and the speaker 14 are collectively referred to simply as the robot 15.

[0019] In the present embodiment, robot control will be described by taking as an example a case where a user orders a drink from a certain table to the robot 15 and the robot 15 brings the ordered drink to the table. When the user orders a drink from the robot 15, the user may ask for a recommended drink according to the weather or the like.

[0020] The natural language processing system 50 is a natural language processing system that utilizes a large language model. A large language model is a deep learning model generated through pre-training based on vast amounts of data, which is called a language model that models human spoken language, which is natural language, according to its probability of occurrence. As a natural language processing system using a large language model, for example, GPT-3, GPT-3.5, and GPT-4 can be used. When the natural language processing system 50 receives a request, it statistically estimates the generation probability of the next word from the text included in the received request using the large language model, and transmits the estimation result to the requester.

[0021] The search server 60 performs a search process based on a search request transmitted from the robot system 10, and provides information corresponding to the search request by transmitting a search result indicating the result of such a search process to the robot system 10.

[0022] In the system 100 of the present embodiment, the robot system 10 transmits a prompt to inquire about the next operation of the robot 50 to the natural language processing system 50, and operates the robot 50 based on the response from the natural language processing system 50.

[0023] The control device 11 is a control device for controlling the robot 15 based on an interaction with a user, and includes a processing unit 111, a storage unit 112, a communication unit 113, a voice input unit 114, a voice recognition unit 115, a voice synthesis unit 116, and a voice output unit 117. These are communicably connected to each other via, for example, a bus. In the present embodiment, the processing unit 111, the voice recognition unit 115, and the voice synthesis unit 116 are each configured separately, but the voice recognition unit 115 and the voice synthesis unit 116 may be part of the functional configuration of the processing unit 111.

[0024] The processing unit 111 is composed of, for example, a CPU (Central Processing Unit) and a RAM (Random Access Memory) used as the main memory device when the CPU executes processing. The CPU realizes various functions corresponding to the program by, for example, reading the program stored in the storage unit 112 into the RAM and executing it.

[0025] The processing unit 111 transmits a prompt to the natural language processing system 50 using, for example, a large language model, receives an answer from the natural language processing system 50, and generates an operation instruction for the robot 15 based on the answer.

[0026] The storage unit 112 is a so-called auxiliary storage device equipped with non-volatile storage circuits such as an HDD (hard disk drive) and an SSD (solid state drive) that store various information. Note that the storage unit 112 may be a drive device or the like that reads and writes various information to and from portable storage media such as CD-ROMs, DVDs, and flash memories.

[0027] The communication unit 113 is realized by, for example, a circuit connected to a network. The communication unit 113 communicates with the natural language processing system 50 via the network.

[0028] The voice input unit 114 is connected to the microphone 13 that collects voice and outputs a voice signal, either wired or wirelessly, and receives the voice signal from the microphone 13.

[0029] The voice recognition unit 115 performs voice recognition processing on the voice signal from the voice input unit 114 and outputs the character information represented by the voice to the processing unit 111. For example, the voice recognition unit 115 can use a voice recognition technology from a voice signal to character information that utilizes deep learning technology.

[0030] The voice synthesis unit 116 synthesizes voice from the operation instruction of the robot 15 generated by the processing unit 111 and outputs it to the voice output unit 117. The voice synthesis unit 116 can use general voice synthesis technology.

[0031] The voice output unit 117 is connected to a speaker 14 that converts a voice signal into voice and outputs it to the outside, either wired or wirelessly, and outputs the voice signal from the voice synthesis unit 116 to the speaker 14.

[0032] The mobile robot 12 includes a robot control unit 121, a right wheel motor 122, a left wheel motor 123, a camera 124, and a sensor 125.

[0033] The robot control unit 121 receives the operation command of the robot 15 generated by the processing unit 111, and controls each part of the mobile robot 12 according to the operation command. Specifically, this robot control device 121 is a computer equipped with hardware such as an arithmetic device such as a CPU, a main storage device such as a semiconductor memory, an auxiliary storage device such as a hard disk, and a communication device.

[0034] The right wheel motor 122 and the left wheel motor 123 are electric actuators that respectively rotate drive the right and left wheels (not shown) provided on the mobile robot. The right wheel motor 122 and the left wheel motor 123 are each independently supplied with electric power from a battery (not shown) via an inverter (not shown), and their rotational speeds change accordingly. Therefore, the rotational speeds of the right and left wheels can be made different, and the mobile robot 12 can change its direction due to the speed difference between the right and left wheels.

[0035] The camera 124 acquires an image of the surroundings of the mobile robot 12. The sensor 125 includes, for example, a sensor that detects the temperature and humidity of the surroundings of the mobile robot 12, an infrared sensor, and the like.

[0036] Hereinafter, the function list storage unit 112a, the input buffer 112b, the history log storage unit 112c, and the prerequisite storage unit 112d, which are storage areas provided in the storage unit 112, will be described.

[0037] (Function List) The memory unit 112 includes a function list memory unit 112a that stores a function list. Here, the function list is a list having a plurality of skill packages obtained by packaging general-purpose operations of the robot 15 in skill units. That is, the function list is a list having a plurality of function modules that define each of a plurality of predetermined operations to be executed by the robot 15 as function modules.

[0038] Currently, when attempting to generate a program (e.g., low-code) for the robot 15 from scratch by sending a request in natural language to the natural language processing system 50, it is difficult to generate a correct program. For this reason, simply sending a request in natural language to the natural language processing system 50 cannot cause the robot 15 to perform the intended operation.

[0039] Therefore, in the present embodiment, in addition to an instruction in natural language from the user, the processing unit 111 sends the natural language processing system 50 including the function list stored in the function list memory unit 112a as a prompt.

[0040] Thereby, the natural language processing system 50 can determine (or identify) a function module to be executed by the robot 15 based on the user's instruction from the function list. Since the function module defines a predetermined operation to be executed by the robot 15 as described above, the processing unit 111 can generate an operation instruction for the intended robot 15 based on the response from the natural language processing system 50 regarding which function module in the function list should be executed. Thus, the robot 15 can appropriately perform the intended operation.

[0041] In addition, there is a functional module that includes at least parameters related to coordinates necessary for the operation of the robot 15 or parameters related to the text to be output to the user. In such a case, the processing unit 111 transmits the parameters used in each functional module to the natural language processing system 50 to include them in the function list. Then, the processing unit 111 receives, from the natural language processing system 50, data for identifying the functional module and data of the parameters to be given to the functional module, and generates an operation command for the robot 15 based on the identified functional module and the data of the parameters to be given to the functional module. Even in this case, since data for identifying the functional module and data of the parameters to be given to the functional module are transmitted as a response from the natural language processing system 50, the operation command for the robot 15 is limited to the functional module, and the robot can be appropriately operated using the data of the parameters corresponding thereto.

[0042] FIG. 2 is a diagram showing an example of a function list. The functional module 2a is a functional module that converts the text designated for the speaker 14 into voice and outputs it. The functional module 2b is a functional module that moves the mobile robot 12 to the designated coordinates (target_location). The functional module 2c is a functional module that checks the ordered table. The functional module 2d is a functional module that performs a waiting operation. The functional module 2e is a functional module for performing an Internet search using the search engine Bing (registered trademark). The functional module 2f is a functional module for performing Azure (registered trademark) Cognitive Search. Thus, there are some functional modules that require specifying parameters.

[0043] (History log) The storage unit 112 further includes an input buffer 112b that stores the input data and a history log storage unit 112c that accumulates the history log.

[0044] The input buffer 112b first stores the data of the instruction from the user input via the voice input unit 114. Next, every time there is a response from the robot 15 that has operated according to the operation instruction generated by the processing unit 111, the response is overwritten by the processing unit 111 in the input buffer 112b. The response from the robot 15 means the operation result (so-called return value) of the robot 15 executing the operation instruction. The history log is log data composed of the data read from the input buffer 112b and the response from the natural language processing system 50 corresponding to the read data. Thus, in the history log storage unit 112c, a set consisting of the input to the natural language processing system 50 and the output for the input is appended as a history log.

[0045] Currently, the natural language processing system 50 can only return one response to one query. Therefore, since the natural language processing system 50 can only return one response to the processing unit 111 when receiving one instruction from the user, the processing unit 111 can only generate one operation instruction for one robot 15 for one instruction from the user. For this reason, with one instruction from the user, the robot 15 cannot be continuously operated.

[0046] Therefore, in the present embodiment, after transmitting an instruction in natural language from the user, the processing unit 111 transmits to the natural language processing system 50 including the history log accumulated in the history log storage unit 112c in addition to the data read from the input buffer 112b as a prompt.

[0047] As a result, a response based on the data read from the input buffer 112b and the history log is sent from the natural language processing system 50, so the processing unit 111 can generate an operation command based on the history log. Consider the case where an instruction is first issued to the robot 15 to make it operate. In this case, the data initially stored in the input buffer 112b is an instruction from the user. At this stage, no data is stored in the history log. Therefore, the natural language processing system 50 outputs the first response to the processing unit 111 based on the instruction of the user read from the input buffer 112b. In the history log, the data read from the input buffer 112b and the response from the natural language processing system 50 corresponding to the read data are added as a set, so the instruction from the user and the first response corresponding to the instruction are added to the history log. The processing unit 111 generates an operation command for the robot 15 based on the first response, and the robot 15 executes an operation according to the operation command. Then, the operation result of the robot 15 is overwritten in the input buffer 112b as a response, and the processing unit 111 sends the data read from the input buffer 112b again and the history log to the natural language processing system 50 including them in the prompt. Then, the natural language processing system 50 outputs the next response to the processing unit 111, and the processing unit 111 generates the next operation command.

[0048] In this way, the processing unit 111 can use the natural language processing system 50 to generate an operation command for the robot 15 considering the past history. Once the user gives an instruction to the robot 15, the processing unit 111 continuously sends the prompt to the natural language processing system 50 until the natural language processing system 50 determines that the operation of the robot 15 is completed, so that the robot 15 can be made to perform continuous operations.

[0049] FIG. 3 is a diagram showing an example of a history log. The first line is log data indicating that the user gave an instruction "Make tea" to robot 15 (specifically, microphone 13). The second line is log data indicating that natural language processing system 50 determined function module 2a from the function list and determined "Please wait a moment" as data of the parameter to be given to function module 2a in response to the instruction "Make tea". Thus, the log data of the first line and the second line are respectively the input and output to natural language processing system 50, and are added as a set to the history log. The third line is log data indicating that function module 2a was executed and the operation result (so-called return value) that speaker 14 output the voice "Please wait a moment" was returned. The fourth line is log data indicating that there was a response from natural language processing system 50 when a prompt including the operation result of the third line was sent to natural language processing system 50. The (n - 1)-th line is log data indicating that function module 2a was executed and the operation result (so-called return value) that speaker 14 output the voice "I have brought the tea. Please take your time" was returned. The last line, the n-th line, is log data indicating that there was a response "Completion" of the operation of robot 15 from natural language processing system 50 when a prompt including the operation result of the (n - 1)-th line was sent to natural language processing system 50. Due to the response "Completion" from natural language processing system 50, the operation of robot 15 is completed. The log data of the third line and the fourth line and the (n - 1)-th line and the n-th line are respectively the input and output to natural language processing system 50, and are added as a set to the history log. Specifically, the "Completion" of the operation means the "Completion" of the flow in the flowchart described in area 4b of the preconditions described below.

[0050] The history log accumulates the history from the first instruction to the robot 15 as a set of inputs and outputs to the natural language processing system 50, enabling the natural language processing system 50 to understand what has been executed and to what extent. Specifically, it can be determined up to which part of the flowchart described in the precondition area 4b, which will be explained next, has been executed. By transmitting this data to the natural language processing system 50, an appropriate response regarding the actions to be taken next can be obtained from the natural language processing system 50, taking into account the past history.

[0051] (Preconditions) The storage unit 112 further includes a precondition storage unit 112d that stores the preconditions for the operation of the robot 15. The preconditions include the work procedure of the robot 15. Specifically, the work procedure of the robot 15 divides the actions to be performed by the robot 15 into multiple stages and defines the operation guidelines for each stage. These operation guidelines are described in the Unified Modeling Language (UML).

[0052] When the work procedure of the robot 15 is given to the natural language processing system 50 as a precondition, if the precondition is described in source code, the natural language processing system 50 will follow the source code, and although an answer with high rigor can be obtained, an answer with flexibility cannot be obtained. On the other hand, if the precondition is described in a text using Japanese, English, etc., there may be ambiguous points in the expression, so an answer that specifies the operation of the robot may not be obtained, and appropriate operation instructions cannot be generated based on such an answer.

[0053] Therefore, in this embodiment, in addition to the instruction in natural language from the user, the processing unit 111 includes the preconditions stored in the precondition storage unit 112d in the prompt and transmits it to the natural language processing system 50.

[0054] As a prerequisite, since the operations to be performed by the robot 15 are divided into multiple stages and the operation guidelines for each stage are defined, the natural language processing system 50 can provide appropriate answers with flexibility according to the operation guidelines. Therefore, based on these answers, the processing unit 111 can generate operation instructions that enable appropriate operations according to the operation guidelines while ensuring the flexibility of the operations of the robot 111.

[0055] FIGS. 4A and 4B are diagrams showing an example of the prerequisite conditions. The prerequisite conditions described in region 4a define the purpose. For example, in region 4a, it is defined to complete the flowchart (the part of "complete the flowchart" in region 4a). In region 4b, the code data representing the flowchart to be completed is described using UML (as an example, Mermaid). This flowchart corresponds to the above operation guidelines. Since it is defined to complete the flowchart in region 4a, the natural language processing system 50 outputs an answer to complete this flowchart in response to an instruction from the user. Therefore, it is possible to provide the processing unit 111 with appropriate answers with flexibility according to the operation guidelines.

[0056] Here, the advantages of UML will be described. For example, in region 4b, there is a description of "Speak out something like ‘Brew coffee’ or ‘A lot of water’ or ‘A little water’". As indicated by "something like", the natural language processing system 50 does not necessarily have to determine any one of "Brew coffee", "A lot of water", and "A little water" as the data of the parameter, but can appropriately consider and determine the data of the parameter by itself in accordance with the purpose described in region 4a. Thereby, the natural language processing system 50 can give flexible answers while following the operation guidelines described in UML.

[0057] In area 4c, Target Location Information is described. This is an example of information about the environment in which robot 15 operates, and corresponds to the data of parameters regarding the coordinates given to functional module 2b. For example, if table 1 is given to functional module 2b, it becomes an operation instruction to move the robot to table 1.

[0058] In area 4d, Remarks are described. Various contents such as conditions and restrictions may be described in the Remarks. For example, the Remarks include information about the region to which the location where robot 15 is used belongs (the part "This is a relaxation area located within a factory of DENSOCorporation in Aichi Prefecture." in area 4d). By including the information about the region to which the location where robot 15 is used belongs as prerequisite information, natural language processing system 50 can acquire various information such as weather information and event information in that region through Internet search. Thereby, a more flexible answer can be provided. Also, the Remarks have the description "Stick to the flowchart." If there is such a description, natural language processing system 50 will surely follow the operation policy. Therefore, an appropriate answer can be efficiently output. Also, the Remarks have the description "Think step by step." If there is such a description, natural language processing system 50 can think step by step, and can return an answer that enables robot 15 to operate appropriately without returning an answer with a logical leap.

[0059] Next, with reference to FIG. 5, the specific operation of processing unit 111 will be described. FIG. 5 is a flowchart for explaining the operation flow of processing unit 111.

[0060] In step S501, the processing unit 111 reads the function list stored in the function list storage unit 112a and the preconditions stored in the precondition storage unit 112d.

[0061] In step S502, the processing unit 111 determines whether there is an input from the voice recognition unit 115, that is, an instruction from the user via the microphone 13. If there is an input from the voice recognition unit 115, the flow proceeds to step S503. If there is no input from the voice recognition unit 115, the flow returns to step S502.

[0062] In step S503, the processing unit 111 overwrites the input from the voice recognition unit 115 in the input buffer 112b.

[0063] In step S504, the processing unit 111 transmits to the natural language processing system 50, in addition to the function list and preconditions read in step S501, the data overwritten in the input buffer 112b and the history log stored in the history log storage unit 112c. Note that if the history log has not been stored yet, empty data will be transmitted.

[0064] In step S505, the processing unit 111 adds the data in the input buffer 112b and the response from the natural language processing system 50 to the history log. Note that the data added to the history log in this step S505 is data in which the input to the natural language processing system 50 and the output for the input are set.

[0065] In step S506, the processing unit 111 branches the process based on the answer from the natural language processing system 50. If the answer from the natural language processing system 50 causes the speaker 14 to output (here, speak) voice (that is, execute the functional module 2a in FIG. 2), the flow proceeds to step S507. If the answer from the natural language processing system 50 causes the mobile robot 12 to perform an operation (here, move or stop) (that is, execute the functional module 2b or 2d in FIG. 2), the flow proceeds to step S509. If the answer from the natural language processing system 50 causes the search server 60 to perform a search (that is, execute the functional module 2e or 2f in FIG. 2), the flow proceeds to step S511. The branching in step S506 is an example, and there may be other cases where the answer from the natural language processing system 50 causes the processing unit 111 to check an ordered table (that is, execute the functional module 2c in FIG. 2). If the answer from the natural language processing system 50 is "completed", the program is terminated.

[0066] In step S507, the processing unit 111 transmits an operation command (here, a speaking command) generated based on the answer from the natural language processing system 50 to the speech synthesis unit 116.

[0067] In step S508, the processing unit 111 overwrites the input buffer 112b with the reply (for example, the result of speaking) from the speech synthesis unit 116. Then the flow returns to step S504, and the processing unit 111 transmits the function list, preconditions, data overwritten in the input buffer 112b, and the history log to the natural language processing system 50. As a result, steps S504 to S510 are looped until the natural language processing system 50 determines that the operation of the robot 15 is completed due to the completion of the flow, and it becomes possible to execute continuous operations on the robot 15.

[0068] In step S509, the processing unit 111 transmits the operation command generated based on the answer from the natural language processing system 50 to the robot control unit 121.

[0069] In step S510, the processing unit 111 overwrites the reply (i.e., the operation result) from the robot control unit 121 in the input buffer 112b. Then the flow returns to step S504, and the processing unit 111 transmits the function list, preconditions, data overwritten in the input buffer 112b, and the history log to the natural language processing system 50. As a result, steps S504 to S510 are looped until the natural language processing system 50 determines that the operation of the robot 15 is completed due to the completion of the flow, and it becomes possible to execute continuous operations on the robot 15.

[0070] In step S511, the processing unit 111 transmits the search command generated based on the answer from the natural language processing system 50 to the search server 60. The search server 60 performs an Internet search using the search engine Bing (registered trademark) and Azure (registered trademark) Cognitive Search based on the search command.

[0071] In step S512, the processing unit 111 overwrites the search result from the search server 60 in the input buffer 112b. Then the flow returns to step S504, and the processing unit 111 transmits the function list, preconditions, data overwritten in the input buffer 112b, and the history log to the natural language processing system 50. As a result, steps S504 to S510 are looped until the natural language processing system 50 determines that the operation of the robot 15 is completed due to the completion of the flow, and it becomes possible to execute continuous operations on the robot 15.

[0072] Note that in FIG. 5, the processing unit 111 has been described as an example of transmitting data in the input buffer, the function list, the history log, and the preconditions to the natural language processing system 50 including them in the prompt. However, the processing unit 111 may transmit at least one of these to the natural language processing system 50 including them in the prompt in addition to the data in the input buffer 112b.

[0073] For example, when all of the data in the input buffer, the function list, the history log, and the preconditions are included in the prompt, it is possible to make the robot 15 perform flexible and appropriate operations by using the preconditions for which the operation policy is defined, and by using the history log, it is possible to loop the process and make the robot 15 perform continuous operations considering the past history. Furthermore, by using the function list, it is possible to make the robot 15 perform the intended operation. Also, by using the function list, an answer as to which function module should be executed will be returned from the natural language processing system 50. Since data specifying a predetermined function module is obtained as an answer, it can be accumulated as a history log in a form that can be used later. Therefore, the history log can be included in the prompt, and an appropriate next answer can be obtained.

[0074] Next, with reference to FIG. 6, the specific operation of the robot control unit 121 will be described. FIG. 6 is a flowchart showing the processing of the robot control unit 121.

[0075] In step S601, the robot control unit 121 determines whether there is an input from the processing unit 111. If there is an input from the processing unit 111, that is, if there is an operation command for the mobile robot 12, the flow proceeds to step S602. If there is no input from the processing unit 111, the flow returns to step S601.

[0076] In step S602, the robot control unit 121 executes the operation instructed in the operation command.

[0077] In step S603, the robot control unit 121 transmits the operation result to the processing unit 111.

[0078] (Other embodiments) Although the embodiments of the present disclosure have been described above, the present disclosure is not construed as being limited to the above embodiments, and can be applied to various embodiments and combinations without departing from the gist of the present disclosure.

[0079] Also, the processing flow described in the above embodiments is also an example, and unnecessary steps may be deleted, new steps may be added, or the processing order may be changed without departing from the gist of the present invention.

[0080] In the above embodiment, an instruction was given to the robot 15 via the microphone 13. However, not only an instruction by voice via the microphone 13 but also an instruction by text information via a keyboard or a touch panel may be given to the robot 15. Further, a display may be used instead of the speaker 14 to provide text information to the user.

[0081] It may also be provided in each of the aspects described below. (Aspect 1) A control device (11) for controlling a robot (15) based on interaction with a user, An input unit (114) that receives an instruction from the user for the robot, An output unit (116) that outputs a response to the instruction to the user, An input buffer (112b) that stores the instruction data input from the input unit, A history log storage unit (112c) that accumulates a history log including the data read from the input buffer and an answer from a natural language processing system (50) using a large language model corresponding to the read data, A processing unit (111) that transmits a prompt including data read from the input buffer and the history log to the natural language processing system, receives an answer from the natural language processing system, and generates an operation instruction for the robot based on the answer. comprising The control device, wherein the processing unit overwrites the input buffer with an operation result from the robot that has operated according to the operation instruction.

[0082] With the above configuration, for example, when an instruction is first given to the robot to make it operate, the data initially stored in the input buffer is the instruction from the user. At this stage, since no data is accumulated in the history log, the processing unit generates an operation instruction based on the first answer from the natural language processing system based on the user's instruction read from the input buffer. In the history log, the data read from the input buffer and the answer from the natural language processing system corresponding to the read data are stored as a set, so the instruction from the user and the first answer corresponding to the instruction are accumulated in the history log. After the processing unit generates an operation instruction for the robot based on the first answer, the robot executes an operation according to the operation instruction. Then, the operation result of the robot is overwritten in the input buffer, and the processing unit transmits to the natural language processing system a prompt including the data read again from the input buffer and the history log. Then, the processing unit generates the next operation instruction based on the next answer from the natural language processing system.

[0083] In this way, by transmitting a prompt including the history log to the natural language processing system, the processing unit can generate an operation instruction for the robot considering the past history. Once the user gives an instruction to the robot, the processing unit continuously transmits the prompt to the natural language processing system until the natural language processing system 50 determines that the operation of the robot 15 is completed, enabling the robot to perform continuous operations.

[0084] (Aspect 2) It further has a precondition storage unit (112d) that stores preconditions for dividing the operations to be performed by the robot into multiple stages and defining the operation guidelines for each stage. The processing unit transmits a prompt including the preconditions, the data read from the input buffer, and the history log to the natural language processing system, receives an answer from the natural language processing system, and generates an operation command for the robot based on the answer. The control device according to Aspect 1.

[0085] (Aspect 3) The operation guidelines are described in UML. The control device according to Aspect 2.

[0086] (Aspect 4) The control device (11) according to any one of Aspects 1 to 3, The robot (15), and a robot system (10) comprising the same.

[0087] (Aspect 5) The control device (11) according to any one of Aspects 1 to 3, The robot (15), The natural language processing system (50), and a system (100) comprising the same.

[0088] (Aspect 6) A control method for controlling a robot based on interaction with a user, An input step of receiving an instruction from the user for the robot, An output step of outputting a response to the instruction to the user, A storage step of storing the data of the instruction input from the input step in an input buffer, An accumulation step of accumulating a history log consisting of the data read from the input buffer and an answer from a natural language processing system using a large language model corresponding to the read data, A processing step of sending a prompt including data read from the input buffer and the history log to the natural language processing system, receiving an answer from the natural language processing system, and generating an operation instruction for the robot based on the answer; comprising: The processing step is a control method of overwriting the input buffer with an operation result from the robot that has operated according to the operation instruction.

[0089] (Aspect 7) A control program for controlling a robot based on interaction with a user, causing a computer to an input step of receiving an instruction from the user for the robot; an output step of outputting a response to the instruction to the user; a saving step of saving the instruction data input from the input step in an input buffer; an accumulating step of accumulating a history log composed of the data read from the input buffer and an answer from a natural language processing system using a large language model corresponding to the read data; executing a processing step of sending a prompt including the data read from the input buffer and the history log to the natural language processing system, receiving an answer from the natural language processing system, and generating an operation instruction for the robot based on the answer; The processing step is a control program for overwriting the input buffer with an operation result from the robot that has operated according to the operation instruction.

Explanation of Reference Numerals

[0090] 10... robot system, 11... control device, 12... mobile robot, 13... microphone, 14... speaker, 15... robot, 50... natural language processing system, 111... processing unit, 112... storage unit, 112a ··· Function list storage unit, 112b ··· Input buffer, 112c ··· History log storage unit, 112d ··· Prerequisite storage unit, 114 ··· Voice input unit, 116 ··· Voice synthesis unit, 121 ··· Robot control unit

Claims

1. A control device (11) for controlling a robot (15) based on an interaction with a user, comprising: an input unit (114) that receives an instruction from the user for the robot; an output unit (116) that outputs a response to the instruction to the user; an input buffer (112b) that stores the data of the instruction input from the input unit; a history log storage unit (112c) that accumulates a history log consisting of the data read from the input buffer and a response from a natural language processing system (50) using a large language model corresponding to the read data; a processing unit (111) that transmits a prompt including the data read from the input buffer and the history log to the natural language processing system, receives a response from the natural language processing system, and generates an operation instruction for the robot based on the response; and comprising: The control device, wherein the processing unit overwrites the input buffer with an operation result from the robot that has operated according to the operation instruction.

2. The control device according to claim 1, further comprising a precondition storage unit (112d) that stores preconditions for dividing the operation to be performed by the robot into a plurality of stages and defining an operation policy for each stage, wherein the processing unit transmits a prompt including the preconditions, the data read from the input buffer, and the history log to the natural language processing system, receives a response from the natural language processing system, and generates an operation instruction for the robot based on the response.

3. The control device according to claim 2, wherein the operation policy is described in UML.

4. A robot system (10) comprising the control device (11) according to any one of claims 1 to 3, and the robot (15).

5. A system (100) comprising the control device (11) according to any one of claims 1 to 3, the robot (15), and the natural language processing system (50).

6. A control method for controlling a robot based on an interaction with a user, comprising: an input step of receiving an instruction from the user for the robot; an output step of outputting a response to the instruction to the user; a storage step of storing the data of the instruction input from the input step in an input buffer; An accumulation step of accumulating a history log consisting of data read from the input buffer and an answer from a natural language processing system using a large language model corresponding to the read data; A processing step of sending a prompt including the data read from the input buffer and the history log to the natural language processing system, receiving an answer from the natural language processing system, and generating an operation instruction for the robot based on the answer; comprising; The processing step is a control method of overwriting the input buffer with an operation result from the robot that has operated according to the operation instruction.

7. A control program for controlling a robot based on interaction with a user, causing a computer to an input step of receiving an instruction from the user for the robot; an output step of outputting a response to the instruction to the user; a saving step of saving the instruction data input from the input step in an input buffer; an accumulation step of accumulating a history log consisting of data read from the input buffer and an answer from a natural language processing system using a large language model corresponding to the read data; a processing step of sending a prompt including the data read from the input buffer and the history log to the natural language processing system, receiving an answer from the natural language processing system, and generating an operation instruction for the robot based on the answer, and The processing step is a control program for overwriting the input buffer with an operation result from the robot that has operated according to the operation instruction.

Images