Control device, control method, and control program

Abstract

To provide a control device, method, and program that enable a conversation with a user during the waiting time of a robot operation.SOLUTION: In a robot system 100, a control device 11 for controlling a robot 10 on the basis of an interaction with a user includes: an input section 114 that receives an instruction from the user; an output section 117 that outputs a response to the instruction; a front processing section 111f that conducts a conversation using a first natural language processing system; and a back processing section 111b that controls the robot using a second natural language processing system. The front processing section outputs a voice based on an answer obtained by transmitting the instruction to the first natural language processing system, and transmits the instruction based on the answer to the back processing section. The back processing section causes the robot to perform an operation based on an answer obtained by transmitting the instruction to the second natural language processing system, and transmits the instruction based on the answer to the front processing section.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to a technology for controlling a robot based on interaction with a user. [Background technology]

[0002] In recent years, robots that can move flexibly based on user interaction via instructions from the user in natural language have been attracting attention.

[0003] Patent Document 1 describes a robot control device that allows a robot to efficiently perform a target action simply by issuing a command for the target action in a form similar to natural language. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Publication No. 08-286708 Summary of the Invention [Problem to be solved by the invention]

[0005] However, the robot control device described in Patent Document 1 cannot make the robot perform one target action while making the robot perform another target action, and can only make the robot perform sequential actions. In particular, if the robot is equipped with a conversation function, the robot cannot have a conversation with the user while waiting for the robot to perform its action.

[0006] In view of the above problems, the present invention aims to enable a conversation with a user while waiting for a robot to operate. [Means for solving the problem]

[0007] The present invention employs the following technical solutions to solve the above problems. The reference symbols in parentheses in the claims and this section are merely examples showing the correspondence with the specific solutions described in the embodiments below as one aspect, and do not limit the technical scope of the present invention.

[0008] A control device (11) according to one aspect of the present invention is a control device (11) for controlling a robot based on an interaction with a user, and includes an input unit (114) that receives instructions from the user for the robot, an output unit (117) that outputs a voice related to a response to the instruction to the user, a front-end processing unit (111f) that uses a first natural language processing system that utilizes a large-scale language model to have a conversation with the user, and a back-end processing unit (111b) that uses a second natural language processing system that utilizes a large-scale language model to control the robot, wherein the front-end processing unit outputs a prompt including data of instructions from the user to the first natural language processing system. The robot is capable of executing an operation module for causing the robot to perform an operation based on the answer from the second natural language processing system, which is obtained by sending a prompt including data of the instruction sent to the back processing unit, to the second natural language processing system, and a front module for sending instructions based on the answer from the second natural language processing system to the front processing unit.

[0009] With the above configuration, the control device divides the tasks of the first and second natural language processing systems into a front processing unit for conversing with the user and a back processing unit for controlling the robot. This allows the control of the robot and search processing to be performed only by the back processing unit, so the user can converse with the user through the front processing unit without waiting for the completion of the control of the robot or the execution of the search processing.

[0010] A control device (21) according to one aspect of the present invention is a control device (21) for controlling robots including a first robot and a second robot based on an interaction with a user, and includes an input unit (114) that receives instructions from the user for the robot, an output unit (117) that outputs a voice related to a response to the instruction to the user, a front processing unit (111f) that conducts a conversation with the user using a first natural language processing system that uses a large-scale language model, a main back processing unit (211b) that manages control of the robot using a second natural language processing system that uses a large-scale language model, a first sub-back processing unit (211ba) that controls the first robot using a third natural language processing system that uses a large-scale language model, and a second sub-back processing unit (211bb) that controls the second robot using a fourth natural language processing system that uses a large-scale language model, and the front processing unit receives a prompt from the first natural language processing system, the prompt including data of the instruction from the user, and the output unit (117) outputs a voice related to a response to the instruction to the user, the front processing unit receives a prompt from the first natural language processing system, the main back processing unit (211b) that manages control of the robot using a second natural language processing system that uses a large-scale language model, and the second sub-back processing unit (211bb) a speech module for outputting a voice based on an answer to the output unit, and a main back module for transmitting an instruction based on the answer from the first natural language processing system to the main back processing unit, wherein the main back processing unit is capable of executing a first sub-back module for transmitting an instruction based on the answer from the second natural language processing system to the first sub-back processing unit, the instruction being obtained by transmitting a prompt including data of the instruction transmitted to the main back processing unit to the second natural language processing system; a second sub-back module for transmitting an instruction based on the answer from the second natural language processing system to the second sub-back processing unit; and a front module for transmitting an instruction based on the answer from the second natural language processing system to the front processing unit, wherein the first sub-back processing unit is capable of executing a first operation module for causing the first robot to perform an operation based on the answer from the third natural language processing system, the instruction being obtained by transmitting a prompt including data of the instruction transmitted to the first sub-back processing unit to the third natural language processing system;and a first main back module that causes an instruction based on the answer from the third natural language processing system to be transmitted to the main back processing unit, and the second sub-back processing unit is capable of executing a second operation module that causes the second robot to perform an operation based on the answer from the fourth natural language processing system, the operation being obtained by transmitting a prompt including data of the instruction transmitted to the second sub-back processing unit to the fourth natural language processing system, and a second main back module that causes an instruction based on the answer from the fourth natural language processing system to be transmitted to the main back processing unit.

[0011] A control method according to one aspect of the present invention is a control method for a control device that includes a front-end processing unit for conducting a conversation with a user using a first natural language processing system that utilizes a large-scale language model to control a robot based on interaction with the user, and a back-end processing unit for controlling the robot using a second natural language processing system that utilizes a large-scale language model, and that includes an input step for accepting instructions from the user for the robot, a front-end processing step for conducting a conversation with the user using the first natural language processing system, and a back-end processing step for controlling the robot using the second natural language processing system, wherein the front-end processing step includes a step for outputting a voice based on a response from the first natural language processing system, which is obtained by sending a prompt including instruction data from the user to the first natural language processing system, to an output unit, and a step for transmitting an instruction based on the response from the first natural language processing system to the back-end processing unit, and the back-end processing step includes a step for causing the robot to perform an action based on a response from the second natural language processing system, which is obtained by sending a prompt including the instruction data transmitted to the back-end processing unit to the second natural language processing system, and a step for transmitting an instruction based on the response from the second natural language processing system to the front-end processing unit.

[0012] A control program according to one aspect of the present invention is a control program for a control device including a front-end processing unit for conducting a conversation with a user using a first natural language processing system that utilizes a large-scale language model in order to control a robot based on interaction with the user, and a back-end processing unit for controlling the robot using a second natural language processing system that utilizes the large-scale language model, the control program causing a computer to execute an input step for receiving instructions from the user for the robot, a front-end processing step for conducting a conversation with the user using the first natural language processing system, and a back-end processing step for controlling the robot using the second natural language processing system, includes a step of causing an output unit to output a voice based on an answer from the first natural language processing system, the voice being obtained by sending a prompt including instruction data from the user to the first natural language processing system, and a step of causing the back processing unit to transmit an instruction based on the answer from the first natural language processing system, wherein the back processing step includes a step of causing the robot to perform an action based on the answer from the second natural language processing system, the action being obtained by sending a prompt including the instruction data transmitted to the back processing unit to the second natural language processing system, and a step of transmitting an instruction based on the answer from the second natural language processing system to the front processing unit. [Effects of the Invention]

[0013] According to the present invention, it is possible to provide a control device etc. that enables a conversation with a user during waiting times for a robot to operate. [Brief explanation of the drawings]

[0014] [Figure 1] FIG. 1 is a diagram illustrating an example of a configuration of a robot system according to a first embodiment. [Figure 2] FIG. 10 is a diagram showing an example of a front function list. [Figure 3] FIG. 10 is a diagram showing an example of a back function list. [Figure 4] FIG. 10 is a diagram showing an example of a front desk history log. [Figure 5] FIG. 10 is a diagram illustrating an example of a back history log. [Figure 6] FIG. 10 is a diagram illustrating an example of a front precondition. [Figure 7] FIG. 10 is a diagram illustrating an example of a back precondition. [Figure 8] 10 is a flowchart illustrating the flow of operations of the front processing unit. [Figure 9] 10 is a flowchart illustrating the flow of operations of the front processing unit. [Figure 10A] 10 is a flowchart illustrating the flow of operations of a back processing unit. [Figure 10B] 10 is a flowchart illustrating the flow of operations of a back processing unit. [Figure 11] 10 is a flowchart showing the processing of a robot control unit. [Figure 12] FIG. 10 is a diagram illustrating an example of a configuration of a robot system according to a second embodiment. [Figure 13] FIG. 10 is a diagram showing an example of a main back function list. [Figure 14] FIG. 10 is a diagram showing an example of a first sub-back function list. [Figure 15] FIG. 10 is a diagram showing an example of a second sub-back function list. [Figure 16] FIG. 10 is a diagram illustrating an example of a main-back precondition. [Figure 17] FIG. 10 is a diagram showing an example of a first sub-back precondition. [Figure 18] FIG. 10 is a diagram showing an example of a second sub-back precondition. [Figure 19] FIG. 10 is a diagram showing an example of a main back history log. [Figure 20] FIG. 10 is a diagram showing an example of a first sub-back history log. [Figure 21] FIG. 10 is a diagram showing an example of a second sub-back history log. DETAILED DESCRIPTION OF THE INVENTION

[0015] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. Note that the embodiment described below shows an example of how the present invention can be implemented, and the present invention is not limited to the specific configuration described below. When implementing the present invention, a specific configuration corresponding to the embodiment may be appropriately adopted.

[0016] (First embodiment) FIG. 1 is a diagram illustrating an example of the configuration of a robot system 100 according to a first embodiment. The robot system 100 illustrated in FIG. 1 includes a robot 10, a first natural language processing system 51, a second natural language processing system 52, and a search server 60. The robot 10 and the first natural language processing system 51 are connected to each other so as to be able to send and receive data via a network. The robot 10 and the second natural language processing system 52 are connected to each other so as to be able to send and receive data via the network. The robot 10 and the search server 60 are connected to each other so as to be able to send and receive data via the network. In this embodiment, the robot 10 and the first natural language processing system 51 and the robot 10 and the second natural language processing system 52 are connected to each other so as to be able to send and receive data via the network. However, a large language model (LLM) (e.g., Calm2) that can be executed locally (i.e., in the front processing unit 111f or the back processing unit 111b described later) without using a network may be used as the first natural language processing system 51 and the second natural language processing system 52.

[0017] The robot 10 includes a control device 11, an arm robot 12, a microphone 13, and a speaker 14.

[0018] In this embodiment, an example of robot control will be described in which a user converses with the robot 10, ordering a drink such as tea from the robot 10, or asking about today's weather. When ordering a drink from the robot 10, the user may ask for recommended drinks based on the weather, etc. The user may also instruct the robot 10 to assemble a hand bell and to ring the assembled hand bell.

[0019] The first natural language processing system 51 and the second natural language processing system 52 are natural language processing systems that use a large-scale language model. A large-scale language model is a deep learning model that is generated by pre-training based on a huge amount of data on what is called a language model, which models natural language, human spoken words, based on their occurrence probability. As natural language processing systems that use large-scale language models, for example, GPT-3, GPT-3.5, and GPT-4 can be used. Upon receiving a request, the first natural language processing system 51 and the second natural language processing system 52 use the large-scale language model to send to the request source a sentence that combines one or more probabilistically estimated words that follow the sentence included in the received request.

[0020] The search server 60 performs a search process based on a search request (an example of an operation command) sent from the robot 10, and provides information corresponding to the search request by sending search results indicating the results of the search process to the robot 10.

[0021] In the robot system 100 of this embodiment, the robot 10 sends prompts to the first natural language processing system 51 and the second natural language processing system 52 to inquire about the operation of the robot 10, and operates the robot 10 based on the responses from the first natural language processing system 51 and the second natural language processing system 52.

[0022] The control device 11 is a control device for controlling the robot 10 based on interactions with a user, and includes a processing unit 111, a memory unit 112, a voice input unit 114, a voice recognition unit 115, a voice synthesis unit 116, and a voice output unit 117. These are communicatively connected to each other, for example, via a bus. In this embodiment, the processing unit 111, the voice recognition unit 115, and the voice synthesis unit 116 are each configured separately, but the voice recognition unit 115 and the voice synthesis unit 116 may be part of the functional configuration of the processing unit 111. The control device 11 also includes a communication unit (not shown). The communication unit is realized, for example, by a circuit connected to a network. The communication unit communicates with external devices such as the first natural language processing system 51, the second natural language processing system 52, the search server 60, and the arm robot 12 via the network.

[0023] The processing unit 111 is configured with, for example, a CPU (Central Processing Unit) and a RAM (Random Access Memory) that is used as a main storage device when the CPU executes processing. The CPU, for example, loads a program stored in the storage unit 112 into the RAM and executes the program, thereby realizing various functions corresponding to the program.

[0024] The processing unit 111 transmits prompts to the first natural language processing system 51 and the second natural language processing system 52, which use, for example, a large-scale language model, receives responses from the first natural language processing system 51 and the second natural language processing system 52, and causes the robot 10 to execute a functional module, described later, identified by the received responses. The processing unit 111 includes, as its functional configuration, a front processing unit 111f and a back processing unit 111b.

[0025] The front processing unit 111f is a processing unit for having a conversation with a user using the first natural language processing system 51. The back processing unit 111b is a processing unit for controlling the robot 10 using the second natural language processing system 52. As an example, GPT-3.5 can be used in the first natural language processing system 51 with an emphasis on speed, and GPT-4 can be used in the second natural language processing system 52 with an emphasis on accuracy. Note that instead of the front processing unit 111f and the back processing unit 111b being functional components of the processing unit 111, the front processing unit 111f and the back processing unit 111b may each be a computer equipped with hardware such as a calculation device such as a CPU, a main storage device such as a semiconductor memory, an auxiliary storage device such as a hard disk, and a communication device.

[0026] The front processing unit 111f transmits to the first natural language processing system 51 a prompt including instruction data from the user input via the microphone 13 or a prompt including instruction data based on a response from the second natural language processing system 52 transmitted from the back processing unit 111b, receives the response from the first natural language processing system 51, and identifies a functional module to be executed from a front function list (described later) based on the received response. As will be described in detail later, if the identified functional module is a speech module, the front processing unit 111f executes the speech module to cause the speech output unit 117 to output a voice based on the received response, and if the identified functional module is a back module, the front processing unit 111f executes the back module to transmit an instruction based on the received response to the back processing unit 111b.

[0027] The back processing unit 111b transmits to the second natural language processing system 52 a prompt including the instruction data transmitted from the front processing unit 111f to the back processing unit 111b, receives a response from the second natural language processing system 52, and identifies a functional module to be executed from a back function list (described later) based on the received response. As will be described in detail later, if the identified functional module is an operation module, the back processing unit 111b executes the operation module to cause the robot 10 to perform an operation based on the received response, and if the identified functional module is a front module, the back processing unit 111b executes the front module to transmit an instruction based on the received response to the front processing unit 111f.

[0028] The front processing unit 111f has a front queue 111f1 in its memory area that temporarily stores data. The back processing unit 111b has a back queue 111b1 in its memory area that temporarily stores data, and a priority memory 111b2. The contents of the data temporarily stored in the front queue 111f1, back queue 111b1, and priority memory 111b2 will be described later.

[0029] The storage unit 112 is a so-called auxiliary storage device that includes a nonvolatile storage circuit such as an HDD (hard disk drive) and an SSD (solid state drive) that store various information. The storage unit 112 may also be a drive or the like that reads and writes various information from and to portable storage media such as CD-RW, DVD-RAM, and flash memory. The storage unit 112 includes a front storage unit 112f for the front processing unit 111f and a back storage unit 112b for the back processing unit 111b.

[0030] The voice input unit 114 is connected by wire or wirelessly to the microphone 13 that collects voice and outputs a voice signal, and receives the voice signal from the microphone 13. In other words, the voice input unit 114 receives instructions from the user to the robot 15.

[0031] The speech recognition unit 115 performs speech recognition processing on the speech signal from the speech input unit 114 and outputs text information represented by the speech to the processing unit 111. For example, the speech recognition unit 115 can use a speech recognition technology that converts speech signals into text information using deep learning technology.

[0032] The voice synthesis unit 116 synthesizes voice from the operation command for the robot 10 generated by the execution of the function module identified by the processing unit 111, and outputs the voice to the voice output unit 117. The voice synthesis unit 116 can use a general voice synthesis technology.

[0033] The audio output unit 117 is connected by wire or wirelessly to the speaker 14, which converts an audio signal into audio and outputs it to the outside, and outputs the audio signal from the audio synthesis unit 116 to the speaker 14. In other words, the audio output unit 117 outputs a response to an instruction from the user to the user.

[0034] The robot 12 includes a robot control unit 121, a first axis L1, a second axis L2, a third axis L3, and a fourth axis L4.

[0035] The robot control unit 121 receives operation commands for the arm robot 12 generated by the back processing unit 111b by executing the functional modules, and controls each part of the arm robot 12 according to the operation commands. Specifically, the robot control unit 121 is a computer equipped with hardware such as a calculation unit such as a CPU, a main storage unit such as a semiconductor memory, an auxiliary storage unit such as a hard disk, and a communication unit.

[0036] The first axis L1, the second axis L2, the third axis L3, and the fourth axis L4 are rotation axes for moving arm joints (not shown) provided on the arm robot 12 using motors (not shown). By independently controlling these multiple axes L1 to L4, the arm robot 12 can be made to perform intended movements.

[0037] Below, we will explain the front function list memory unit 112f1 and the back function list memory unit 112b1, the front history log memory unit 112f2 and the back history log memory unit 112b2, and the front prerequisite memory unit 112f3 and the back prerequisite memory unit 112b3, which are memory areas provided in the front memory unit 112f and the back memory unit 112b of the memory unit 112.

[0038] (Feature List) The storage unit 112 includes function list storage units 112f1 and 112b1 that store function lists of multiple function modules. Here, a functional module is a package of a series of general-purpose operations of the robot 10. The function list is a collection of usage methods for each functional module to be provided as prior information to the natural language processing systems 51 and 52 when the natural language processing systems 51 and 52 are caused to generate functional modules and parameters suitable for task processing. In other words, the function list is a list of multiple functional modules, each of which defines a plurality of predetermined operations to be performed by the robot 10 as a functional module.

[0039] Currently, when trying to generate a program (e.g., low-code) for a robot from scratch by sending natural language requests to a natural language processing system, it is difficult to generate a correct program. For this reason, simply sending natural language requests to a natural language processing system will not allow the robot to perform the intended action.

[0040] Therefore, in this embodiment, the front processing unit 111f includes the front function list stored in the front function list memory unit 112f1 in a prompt in addition to instructions in natural language from the user, and sends the prompt to the first natural language processing system 51, and the back processing unit 111b includes the back function list stored in the back function list memory unit 112b1 in addition to instructions based on the response from the first natural language processing system 51 in a prompt and sends the prompt to the second natural language processing system 52.

[0041] As a result, the first natural language processing system 51 can determine (or identify) a functional module to be executed by the robot 10 from the front function list based on a user's instruction, and the second natural language processing system 52 can determine (or identify) a functional module to be executed by the robot 10 from the back function list based on an instruction based on a response from the first natural language processing system 51. As described above, the functional module defines a predetermined operation to be executed by the robot 10. Therefore, the front processing unit 111f can identify a functional module based on a response from the first natural language processing system 51 regarding which functional module in the front function list should be executed, and can cause the robot 10 to execute the identified functional module. Furthermore, the back processing unit 111b can identify a functional module based on a response from the second natural language processing system 52 regarding which functional module in the back function list should be executed, and can cause the robot 10 to execute the identified functional module. Therefore, the robot 10 can appropriately perform the intended operation.

[0042] Furthermore, the functional modules receive as input parameters related to coordinates and / or time required for the operation of the robot 10, or parameters related to the text to be output to the user. In this case, the processing unit 111 transmits the parameters to be used by each functional module, included in a function list, to the natural language processing systems 51 and 52. The processing unit 111 then receives, from the natural language processing systems 51 and 52, data identifying the functional module along with parameter data to be given to the functional module, and gives the identified functional module the parameter data to be given to the functional module. Even in this case, the natural language processing systems 51 and 52 transmit, as responses, data identifying the functional module and parameter data to be given to the functional module, so that the operation of the robot 10 is limited to the operation defined in the functional module, and the robot can be operated more appropriately using parameter data that matches the function module.

[0043] FIG. 2 is a diagram showing an example of a front function list stored in the front function list storage unit 112f1. The front function list has data on "functional module name," "priority," and "description." "Function module name" is the name of the functional module. "Priority" consists of two types of priority: high priority Immediate and low priority Queue. "Description" is a description of the functional module. By including data on the "description" of the functional module in the front function list, the first natural language processing system 51 can know how to use the functional module and can determine an appropriate functional module for an instruction.

[0044] The front function list storage unit 112f1 stores information on a plurality of functional modules executed by the front processing unit 111f. In the example shown in Fig. 2, the plurality of functional modules are a speech module (Speak) and back modules (ToBack, Weather, WebSearch). The speech module is a functional module for causing the speech output unit 117 to output speech based on a response from the first natural language processing system 51, the speech being obtained by transmitting to the first natural language processing system 51 a prompt including data on instructions from a user or a prompt including data on instructions based on a response from the second natural language processing system 52 transmitted from the back processing unit 111b. The back module is a functional module for causing the back processing unit 111b to transmit an instruction based on a response from the first natural language processing system 51.

[0045] Furthermore, the front function list storage unit 112f1 stores a front function list of a plurality of function modules in association with two types of priority (Immediate, Queue). When the front processing unit 111f executes a function module (Weather, WebSearch) with a higher priority (Immediate) among the back modules (ToBack, Weather, WebSearch), it transmits an instruction based on the response from the first natural language processing system 51 to the back processing unit 111b, and causes the back processing unit 111b to store the instruction in the priority memory 111b2 within the back processing unit 111b (hereinafter simply referred to as "transmitting the instruction based on the response from the first natural language processing system 51 to the priority memory 111b2 of the back processing unit 111b"). On the other hand, when a functional module (ToBack) with a lower priority (Queue) among the back modules (ToBack, Weather, WebSearch) is executed, an instruction based on the response from the first natural language processing system 51 is sent to the back processing unit 111b, and the back processing unit 111b stores the instruction in the back queue 111b1 within the back processing unit 111b (hereinafter, simply referred to as "sending an instruction based on the response from the first natural language processing system 51 to the back queue 111b1 of the back processing unit 111b").

[0046] 2, the functional module Speak is a speech module that converts designated text into speech and outputs it to the speaker 14. The speech module Speak is preferably executed immediately for natural conversation with the user. Specifically, when the response from the first natural language processing system 51 identifies the speech module Speak and another functional module, the execution of the speech module Speak is given top priority.

[0047] The function module ToBack is a function module that transmits a response from the first natural language processing system 51 as is to the back queue 111b1 of the back processing unit 111b. Therefore, if the response from the first natural language processing system 51 is text, the text response is transmitted to the back queue 111b1 of the back processing unit 111b by executing the function module ToBack. If the response from the first natural language processing system 51 is image information or audio information, the image information or audio information is transmitted to the back queue 111b1 of the back processing unit 111b.

[0048] The functional modules Weather, WebSearch, etc. are functional modules that transmit a request based on a response from the first natural language processing system 51, which is generated by the execution of the functional module, to the priority memory 111b2 of the back processing unit 111b. For example, the functional module Weather is a functional module that transmits a weather search request based on a response from the first natural language processing system 51, which is generated by the execution of the functional module, to the priority memory 111b2 of the back processing unit 111b without queuing it. The functional module WebSearch is a functional module that transmits a web search request based on a response from the first natural language processing system 51, which is generated by the execution of the functional module, to the priority memory 111b2 of the back processing unit 111b without queuing it.

[0049] FIG. 3 is a diagram showing an example of a back function list stored in the back function list storage unit 112b1. The back function list has data on "function module name" and "description." "Function module name" is the name of the function module. "Description" is a description of the function module. As with the front function list, by including data on the "description" of the function module in the back function list, the second natural language processing system 52 can know how to use the function module and can determine an appropriate function module for an instruction.

[0050] The back function list storage unit 112b1 stores information on a plurality of function modules executed by the back processing unit 111b. In the example shown in Fig. 3, the plurality of function modules are operation modules (Move, Grab, Give, Place, Assemble, Play, WebSearch, Weather, etc.) for causing the robot 10 to perform an operation based on a response from the second natural language processing system 52, which is obtained by transmitting a prompt including instruction data transmitted from the front processing unit 111f to the back processing unit 111b to the second natural language processing system 52, and a front module (ToFront) for transmitting an instruction based on the response from the second natural language processing system 52 to the front processing unit 111f.

[0051] The functional module ToFront is a functional module that transmits a response from the second natural language processing system 52 to the front processing unit 111f as is, and causes the front processing unit 111f to store the response in the front queue 111f1 of the front processing unit 111f (hereinafter simply referred to as "transmitting the response to the front queue 111f1 in the front processing unit 111f"). As with the functional module ToBack, the response from the second natural language processing system 52 may be in any form, such as text, image information, or audio information.

[0052] The function module Move is a function module that moves the hand of the arm robot 12 to a specified position.

[0053] The function module Grab is a function module that causes the hand of the arm robot 12 to grab a designated item.

[0054] The function module Give is a function module for giving the grabbed item to the user.

[0055] The function module Place is a function module for placing a grabbed item.

[0056] The functional module Assemble is a functional module for assembling handbells.

[0057] The function module Play is a function module for ringing a handbell.

[0058] The functional module WebSearch is a functional module for performing web searches.

[0059] The functional module Weather is a functional module for acquiring weather information.

[0060] In the function lists shown in Figures 2 and 3, the function module name keys and JSON values ​​are written in a simple format because they are complicated, but the function modules can be implemented in JSON format.

[0061] (History log) The storage unit 112 further includes history log storage units 112f2 and 112b2 that store history logs.

[0062] The history log is log data including data transmitted to the natural language processing systems 51, 52 and responses obtained from the natural language processing systems 51, 52 in response to the transmitted data. In this way, a set consisting of an input to the natural language processing systems 51, 52 and an output in response to the input is added as a history log to the history log storage units 112f2, 112b2.

[0063] In this embodiment, after transmitting an instruction in natural language from the user, the front processing unit 111f includes the history log accumulated in the front history log storage unit 112f2 in addition to the data read from the front queue 111f1 in a prompt and transmits the same to the first natural language processing system 51. Furthermore, after transmitting an instruction in natural language from the user, the back processing unit 111b includes the history log accumulated in the back history log storage unit 112b2 in addition to the data read from the back queue 111b1 or the data read from the priority memory 111b2 in a prompt and transmits the same to the second natural language processing system 52.

[0064] As a result, answers based on the history log are sent from the natural language processing systems 51 and 52, and the processing units 111f and 111b can cause the robot 10 to perform successive actions based on past interactions based on the history log.

[0065] 4 is a diagram showing an example of a front desk history log stored in the front desk history log storage unit 112f2. Line No. 1 is log data indicating that the user uttered "Hello" to the robot 10 (specifically, the microphone 13). Line No. 2 is log data indicating that the first natural language processing system 51, in response to the user's utterance of "Hello," selected a function module Speak from the front desk function list and determined "Hello. What can I do for you today?" as parameter data to be given to the function module Speak. In this way, the log data in lines No. 1 and No. 2 are input to the first natural language processing system 51 and output in response to the input, respectively, and are added as a set to the front desk history log storage unit 112f2 as a history log.

[0066] Line No. 3 is log data indicating that the user uttered "Please give me some tea" to the robot 10. Line No. 4 is log data indicating that the first natural language processing system 51, in response to the utterance "Please give me some tea," determined the function module Speak from the front function list, determined "Understood. Please wait a moment" as parameter data to be given to the function module Speak, and also determined the function module ToBack, and determined "Please give me some tea" as the parameter to be given to the function module ToBack.

[0067] Line No. 5 is log data indicating that the user uttered "Can I have some water, too?" to the robot 10. Line No. 6 is log data indicating that the first natural language processing system 51, in response to the utterance "Can I have some water, too," determined the function module Speak from the front function list, determined "Understood. Please wait a moment" as parameter data to be given to the function module Speak, and also determined the function module ToBack, and determined "Please give me some water" as the parameter to be given to the function module ToBack.

[0068] Line No. 7 is log data indicating that the user uttered "Please check the weather" to the robot 10. Line No. 8 is log data indicating that the first natural language processing system 51, in response to the utterance "Please check the weather," determined the functional module Speak from the front function list, and determined "Understood" as parameter data to be given to the functional module Speak, as well as the functional module Weather.

[0069] Line No. 9 is log data indicating that the back processing unit 111b has sent to the front processing unit 111f an instruction (here, the answer itself) based on the answer "The weather is sunny" from the second natural language processing system 52. Line No. 10 is log data indicating that the first natural language processing system 51, in response to the instruction "The weather is sunny," has determined the function module Speak from the front function list and determined "It's sunny today" as parameter data to be given to the function module Speak.

[0070] Line No. 11 is log data indicating that the back processing unit 111b has sent to the front processing unit 111f an instruction (here, the answer itself) based on the answer from the second natural language processing system 52, "I have served you tea. I will serve you water next." Line No. 12 is log data indicating that the first natural language processing system 51, in response to the instruction "I have served you tea. I will serve you water next," has determined the function module Speak from the front function list, and determined "Here is your tea. Please. I will serve you water next." as parameter data to be given to the function module Speak.

[0071] Line No. 13 is log data indicating that the back processing unit 111b has sent to the front processing unit 111f an instruction (here, the answer itself) based on the answer from the second natural language processing system 52, "I've given you some water." Line No. 14 is log data indicating that the first natural language processing system 51, in response to the instruction "I've given you some water," has determined the function module Speak from the front function list, and determined "Here's some water. Please." as parameter data to be given to the function module Speak.

[0072] Fig. 5 is a diagram showing an example of the back history log stored in the back history log storage unit 112b2. The back history log in Fig. 5 has contents corresponding to the front history log in Fig. 4. Note that in Fig. 5, the grayed-out lines do not represent the contents of the back history log, but are shown for ease of understanding.

[0073] Line No. 1 is log data indicating that the front processing unit 111f has executed the functional module ToBack to send an instruction (here, the answer itself) based on the answer from the first natural language processing system 51, "Please give me some tea," to the back queue 111b1 of the back processing unit 111b. Line No. 2 is log data indicating that the second natural language processing system 52, in response to the instruction "Please give me some tea," has determined the functional module Move from the back function list and determined "tea" as parameter data to be given to the functional module Move.

[0074] Line No. 3 is log data indicating that the front processing unit 111f has executed the function module ToBack to send an instruction (here, the answer itself) to the back queue 111b1 of the back processing unit 111b, "Please give me some water," based on the answer from the first natural language processing system 51. Line No. 4 is log data indicating that the back processing unit 111b has queued the instruction "Please give me some water" (that is, has not taken it out of the back queue 111b1), because the operation in response to the instruction "Please give me some tea" sent from the front processing unit 111f in line No. 1 has not been completed.

[0075] Line No. 5 is log data indicating that the front processing unit 111f executed the functional module Weather and sent an instruction to check the weather to the priority memory 111b2 of the back processing unit 111b without queuing. Line No. 6 is log data indicating that the second natural language processing system 52 selected the functional module Weather from the back function list, the back processing unit 111b immediately executed the functional module Weather, and obtained the execution result.

[0076] Line No. 7 is log data indicating that the second natural language processing system 52 determined the function module ToFront based on the execution result of the function module Weather, and determined "The weather is sunny" as the parameter to be given to the function module ToFront.

[0077] Line No. 8 is log data indicating that the back processing unit 111b has completed the execution of the functional module Move(tea) in line No. 2, the second natural language processing system 52 has determined the functional module Grab(tea) based on the execution result, the back processing unit 111b has completed the execution of the functional module Grab(tea), the second natural language processing system 52 has determined the functional module Give(tea) based on the execution result, and the back processing unit 111b has completed the execution of the functional module Give(tea).

[0078] Line No. 9 is log data showing that the second natural language processing system 52 determined the functional module ToFront as a result of executing the functional module Give(tea), and determined that the parameter to be given to the functional module ToFront was "I have given you tea. I will then give you water."

[0079] Row No. 10 is log data indicating that, because the operation for the instruction "Please give me some tea" in row No. 1 has been completed, the back processing unit 111b retrieves the instruction "Please give me some water" in row No. 4, which was stored in the back queue 111b1, from the back queue 111b1, the second natural language processing system 52 determines the functional module Move(water) in response to the instruction, the back processing unit 111b completes execution of the functional module Move(water), the second natural language processing system 52 determines the functional module Grab(water) in response to the execution result, the back processing unit 111b completes execution of the functional module Grab(water), the second natural language processing system 52 determines the functional module Give(water) in response to the execution result, and the back processing unit 111b completes execution of the functional module Give(water).

[0080] Line No. 11 is log data indicating that the second natural language processing system 52 determined the functional module ToFront as a result of executing the functional module Give(water) and determined "I gave you water" as the parameter to be given to the functional module ToFront.

[0081] The history log stores a history from the first instruction to the robot 10 as a set of inputs to the natural language processing systems 51, 52 and outputs from the natural language processing systems 51, 52, and therefore allows the natural language processing systems 51, 52 to understand what has been executed and to what extent. By sending this data to the natural language processing systems 51, 52, the natural language processing systems 51, 52 can obtain an appropriate answer about the next action to be taken, taking into account the past history.

[0082] (Prerequisite) The storage unit 112 further includes precondition storage units 112f3 and 112b3 that store preconditions for the operation of the robot 10.

[0083] The preconditions include the work procedure of the robot 10. Specifically, the work procedure of the robot 10 divides the operations to be performed by the robot 10 into multiple stages and specifies the operation policy for each stage. These operation policies may be written in a natural language such as Japanese or English, or in Unified Modeling Language (UML).

[0084] When the work procedures of the robot 10 are given to the natural language processing systems 51 and 52 as preconditions, if the preconditions are written in source code, the natural language processing systems 51 and 52 will follow the source code, resulting in a highly precise answer but not a flexible answer.

[0085] Therefore, in this embodiment, the front processing unit 111f includes the front preconditions stored in the front precondition storage unit 112f3 in a prompt in addition to an instruction in natural language from the user, and transmits the prompt to the first natural language processing system 51. Furthermore, the back processing unit 111b includes the back preconditions stored in the back precondition storage unit 112b3 in addition to an instruction based on a response from the first natural language processing system 51 in a prompt and transmits the prompt to the second natural language processing system 52.

[0086] The preconditions define the behavior of the robot 10 in multiple stages and define the behavioral policy for each stage, so the natural language processing systems 51 and 52 can provide flexible yet appropriate answers in accordance with the behavioral policies. Therefore, based on these answers, the processing unit 111 can make the robot 10 perform appropriate behavior in accordance with the behavioral policies while ensuring flexibility in the behavior of the robot 10.

[0087] 6 is a diagram showing an example of a front precondition. In area 6a, "About you" is defined as the front precondition. For example, area 6a contains an instruction to the first natural language processing system 51 to generate an answer describing the person as a frank and friendly woman. In other words, a policy for the answer to be generated by the first natural language processing system 51 is defined.

[0088] In area 6b, "your role" is defined as a front prerequisite. For example, area 6b states that you should respond to instructions from the user. It also states that depending on the instructions from the user, you should communicate the instructions to BackAI and cooperate. These define how to respond to instructions from the user, that is, the work procedure for the actions to be performed by the robot 10. Note that BackAI refers to the second natural language processing system 52, and FrontAI refers to the first natural language processing system 51.

[0089] In the area 6c, a "workspace" is defined as a front precondition. This is the workspace in which the robot 10 operates, and various information about objects that the arm robot 12 can grasp is described.

[0090] In the area 6d, "tasks that can be executed by BackAI" are defined as front-end preconditions. In the area 6d, for example, information about function modules that can be executed by the back processing unit 111b is defined.

[0091] FIG. 7 is a diagram showing an example of a back precondition. In area 7a, "Your role" is specified as the back precondition. For example, an instruction to briefly report the result is written, and a policy for the answer to be generated by the second natural language processing system 52 is specified. In area 7b, a "workspace" is specified as the back precondition. In area 7c, a "task" is specified as the back precondition. This is a work procedure for the operation to be performed by the arm robot 12.

[0092] Next, specific operations of the front processing unit 111f will be described with reference to Fig. 8 and Fig. 9. Fig. 8 is a flowchart showing the flow of operations performed by the front processing unit 111f until the front processing unit 111f receives input from the voice recognition unit 115, i.e., an instruction from the user via the microphone 13, and stores it in the front queue 111f1. Fig. 9 is a flowchart for explaining the flow of operations performed by the front processing unit 111f after data has been stored in the front queue 111f1.

[0093] 8, the front processing unit 111f determines whether there is an input from the voice recognition unit 115, that is, whether there is an instruction from the user via the microphone 13. If there is an input from the voice recognition unit 115, the flow proceeds to step S802. If there is no input from the voice recognition unit 115, the flow returns to step S801 and waits for an input from the voice recognition unit 115.

[0094] In step S901 in FIG. 9, the front processing unit 111f reads the front function list stored in the front function list storage unit 112f1 and the front preconditions stored in the front precondition storage unit 112f3.

[0095] In step S902, the front processing unit 111f checks whether data is stored in the front queue 111f1. If data is stored in the front queue 111f1, the flow proceeds to step S903. If data is not stored in the front queue 111f1, the flow returns to step S902 and waits for data to be stored in the front queue 111f1.

[0096] In step S903, the front processing unit 111f transmits the data stored in the front queue 111f1 and the front history log accumulated in the front history log storage unit 112f2, in addition to the front function list and front prerequisites read in step S901, to the first natural language processing system 51. If the front history log has not yet been accumulated, a front history log of empty data will be transmitted.

[0097] In step S904, the front processing unit 111f adds the data of the front queue 111f1 transmitted to the first natural language processing system 51 and the response from the first natural language processing system 51 to the front history log. Note that the data added to the front history log in step S904 includes data that is a set of an input to the first natural language processing system 51 and an output in response to the input.

[0098] In step S905, the front-end processing unit 111f branches the process depending on the priority of the functional module identified by the response from the first natural language processing system 51. If the priority of the functional module identified by the response from the first natural language processing system 51 is Immediate, the flow proceeds to step S906. If the priority of the functional module identified by the response from the first natural language processing system 51 is Queue, the flow proceeds to step S909.

[0099] In step S906, the front processing unit 111f executes a functional module (Speak, WebSearch, or Weather) with an Immediate priority. When the functional module Speak is executed, an operation command is sent to the speech synthesis unit 116. When the functional module WebSearch or Weather is executed, a request (a Web search request in the case of the functional module WebSearch, or a weather search request in the case of the functional module Weather) is sent to the priority memory 111b2 of the back processing unit 111b. Upon receiving the request, the back processing unit 111b immediately includes the request in a prompt and sends it to the second natural language processing system 52, and returns the functional module (WebSearch or Weather) based on the request as a response. Then, the back processing unit 111b executes the functional module (WebSearch or Weather) to obtain the search results, and sends the search results to the front queue 111f1 of the front processing unit 111f using the functional module ToFront.

[0100] In step S907, the front processing unit 111f waits for the completion of execution of the functional module (Speak, WebSearch, or Weather). If the execution of the functional module (Speak, WebSearch, or Weather) is completed, the process proceeds to step S908. Since a functional module with immediate priority is likely not to complete its execution immediately (that is, the functional module Speak, WebSearch, or Weather takes time to complete its execution), the front processing unit 111f must wait for a notification that the execution of the functional module (Speak, WebSearch, or Weather) is completed. The notification that the execution of the functional module Speak is completed is transmitted from the speech synthesis unit 116, which outputs a speech signal in response to the execution of an operation command. Furthermore, the notification that the execution of the functional modules WebSearch and Weather is completed is transmitted from the back processing unit 111b in step S1014, which will be described later.

[0101] In step S908, the front-end processing unit 111f records the execution result of the function module (Speak, WebSearch, or Weather) in the front-end history log, after which the flow returns to step S902.

[0102] In step S909, the front processing unit 111f executes the function module (ToBack) whose priority is "Queue." When the function module (ToBack) is executed, the response from the first natural language processing system 51 is sent as is to the back queue 111b1 of the back processing unit 111b.

[0103] In step S910, the front processing unit 111f records the execution result of the function module (ToBack) in the front history log, after which the flow returns to step S902.

[0104] Next, a specific operation of the back processing unit 111b will be described with reference to Figures 10A and 10B. Figures 10A and 10B are flowcharts for explaining the flow of operation of the back processing unit 111b after data is stored in the back queue 111b1 or the priority memory 111b2.

[0105] In step S1001, the back processing unit 111b reads the back function list stored in the back function list storage unit 112b1 and the back preconditions stored in the back precondition storage unit 112b3.

[0106] In step S1002, the back processing unit 111b checks whether data is stored in the priority memory 111b2. If data is stored in the priority memory 111b2, the flow proceeds to step S1003. If data is not stored in the priority memory 111b2, the flow proceeds to step S1004.

[0107] In step S1003, the back processing unit 111b transmits the data stored in the priority memory 111b2 and the back history log accumulated in the back history log storage unit 112b2, in addition to the back function list and back prerequisites read in step S1001, to the second natural language processing system 52. If a back history log has not yet been accumulated, a back history log of empty data will be transmitted.

[0108] In step S1004, the back processing unit 111b adds to the back history log storage unit 112b2 the data in the priority memory 111b2 transmitted to the second natural language processing system 52 and the response from the second natural language processing system 52. Note that the data added to the back history log in step S1004 includes data that is a set of an input to the second natural language processing system 52 and an output in response to the input.

[0109] In step S1005, the back processing unit 111b checks whether data is stored in the back queue 111b1. If data is stored in the back queue 111b1, the flow proceeds to step S1006. If data is not stored in the back queue 111b1, the flow returns to step S1002 and waits for data to be stored in the priority memory 111b2 or the back queue 111b1.

[0110] In step S1006, the back processing unit 111b transmits to the second natural language processing system 52 the data stored in the back queue 111b1 and the back history log accumulated in the back history log storage unit 112b2, in addition to the back function list and back prerequisites read in step S1001. Note that if a back history log has not yet been accumulated, a back history log of empty data will be transmitted. In this way, when data is stored in the priority memory 111b2, the back processing unit 111b transmits the data stored in the priority memory 111b2 to the second natural language processing system 51, including it in a prompt, in preference to the data stored in the back queue 111b1.

[0111] In step S1007, the back processing unit 111b adds the data of the back queue 111b1 transmitted to the second natural language processing system 52 and the response from the second natural language processing system 52 to the back history log. Note that the data added to the back history log in step S1007 includes data that is a set of an input to the second natural language processing system 52 and an output in response to the input.

[0112] In step S1008, the back processing unit 111b branches the process depending on the response from the second natural language processing system 52. If the response from the second natural language processing system 52 is the function module ToFront, the flow proceeds to step S1009. If the response from the second natural language processing system 52 is the function module WebSearch or Weather, the flow proceeds to step S1011. If the response from the second natural language processing system 52 is a function module (Move, Grab, Give, etc.) related to the operation of the arm robot 12 (abbreviated as robot operation), the flow proceeds to step S1015.

[0113] In step S1009, the back processing unit 111b executes the function module ToFront. When the function module ToFront is executed, the response from the second natural language processing system 52 is sent as is to the front queue 111a1 of the front processing unit 111a.

[0114] In step S1010, the back processing unit 111b records the execution result of the function module ToFront in the back history log, after which the flow returns to step S1002.

[0115] In step S1011, the back processing unit 111b executes the function module WebSearch or Weather.

[0116] In step S1012, the back processing unit 111b stores the search results obtained by executing the functional module WebSearch or Weather in the back queue 111b1. These search results are later sent by the back processing unit 111b to the front queue 111f1 of the front processing unit 111f using the functional module ToFront.

[0117] In step S1013, the back processing unit 111b records the execution result of the function module WebSearch or Weather in the back history log.

[0118] In step S1014, the back processing unit 111b completes the execution of the function module (WebSearch or Weather) executed in step S906, and transmits a notification to the front processing unit 111f that the execution of the function module (WebSearch or Weather) has been completed. Then, the flow returns to step S1002.

[0119] In step S1015, the back processing unit 111b executes a function module (Move, Grab, Give, etc.) related to the robot operation. This generates an operation command for the arm robot 12. The back processing unit 111b transmits the generated operation command to the robot control unit 121.

[0120] In step S1016, the back processing unit 111b stores the execution results of the function modules (Move, Grab, Give, etc.) related to the robot operation in the back queue 111b1.

[0121] In step S1017, the back processing unit 111b records the execution results of the function modules (Move, Grab, Give, etc.) related to the robot operation in the back history log, after which the flow returns to step S1002.

[0122] 9, 10A and 10B illustrate an example in which the processing unit 111 (111f, 111b) includes the data, function list, history log and prerequisites of the queues 111f1, 111b1 and the priority memory 111b2 in a prompt and transmits the prompt to the natural language processing systems 51, 52. However, the processing unit 111 (111f, 111b) may include at least one of these in addition to the data of the queues 111f1, 111b1 and the priority memory 111b2 in a prompt and transmit the prompt to the natural language processing systems 51, 52.

[0123] For example, if a prompt includes all of the data, function list, history log, and preconditions in the queues 111f1, 111b1, and the priority memory 111b2, the robot 10 can be made to perform flexible and appropriate operations by utilizing the preconditions that define the operation policy, the history log can be used to loop processing and make the robot 10 perform continuous operations that take past events into consideration, and the function list can be used to make the robot 10 perform intended operations. Furthermore, by utilizing the function list, the natural language processing systems 51, 52 can return an answer as to which function module should be executed. Since data identifying a predetermined function module is obtained as the answer, it can be accumulated as a history log in a format that can be used later. Therefore, the history log can be included in the prompt, and the next answer can be obtained appropriately.

[0124] Next, a specific operation of the robot control unit 121 will be described with reference to Fig. 11. Fig. 11 is a flowchart showing the processing of the robot control unit 121.

[0125] In step S1101, the robot control unit 121 determines whether or not there is an input from the back processing unit 111b. If there is an input from the back processing unit 111b, that is, an operation command for the arm robot 12, the flow proceeds to step S1102. If there is no input from the back processing unit 111b, the flow returns to step S1101.

[0126] In step S1102, the robot control unit 121 executes the operation instructed in the operation command.

[0127] In step S1103, the robot control unit 121 notifies the back processing unit 111b of the end of the operation. Upon receiving this notification of the end of the operation, the back processing unit 111b stores the execution result of the function module in the back queue 111b1 in step S1016.

[0128] As described above, in the control device 11 according to this embodiment, the tasks of the natural language processing systems 51 and 52 are divided between the front processing unit 111f for conversing with the user and the back processing unit 111b for controlling the robot 10. This allows the control of the robot 10 and search processing to be performed only by the back processing unit 111b, so that the user can converse with the user through the front processing unit 111f without waiting for the control of the robot 10 and the execution of the search processing to be completed.

[0129] Furthermore, when the front processing unit 111f executes a functional module (WebSearch, Weather) with a high priority among the back modules, an instruction generated by the execution of the functional module is sent to the priority memory 111b2 of the back processing unit 111b without being queued, and is immediately executed by the back processing unit 111b. On the other hand, when the front processing unit 111f executes a functional module (ToBack) with a low priority among the back modules, an instruction generated by the execution of the functional module is sent to the back queue 111b1 of the back processing unit 111b, and execution of the instruction is put on hold until it is retrieved from the back queue 111b1. This allows processing with a high priority to be performed first, and enables the user to control the robot 10 while having a natural conversation with the robot 10.

[0130] Furthermore, the instruction generated by the execution of the function module ToBack is the answer itself from the first natural language processing system 51. By executing the function module ToBack, the answer itself from the first natural language processing system 51 is sent to the back queue 111b1 of the back processing unit 111b. Then, the back processing unit 111b sends a prompt including the answer to the second natural language processing system 52, and a function module suitable for the answer is identified. This allows the tasks to be shared by the two natural language processing systems 51 and 52, rather than controlling the robot 10 using only the first natural language processing system 51, and therefore the accuracy of control of the robot 10 can be improved.

[0131] (Second embodiment) Next, a robot system 200 according to a second embodiment will be described. Fig. 12 is a diagram showing an example of the configuration of the robot system 200 according to the second embodiment. The same members as those in the robot system 100 according to the first embodiment will be given the same reference numerals, and their description may be omitted.

[0132] In the robot system 100 according to the first embodiment, the operation of one arm robot 12 is controlled by one back processing unit 111b.

[0133] On the other hand, in a robot system 200 according to a second embodiment, multiple back processing units are used to control the operations of multiple arm robots. The robot system 200 according to this embodiment is characterized by including a first sub-back processing unit 211ba that controls the operation of the left-arm robot 12L and a second sub-back processing unit 211bb that controls the operation of the right-arm robot 12R, as well as a main back processing unit 211b that connects the front processing unit 111f with the first sub-back processing unit 211ba and the second sub-back processing unit 211bb and supervises the control of the robot 20 including the left-arm robot 12L and the right-arm robot 12R.

[0134] By providing the first sub-back processing unit 211ba and the second sub-back processing unit 211bb, it is possible to move the right-arm robot 12R while moving the left-arm robot 12L, for example. Furthermore, if the main back processing unit 211b that manages the overall control of the robot 20 is not provided, it is possible that the right-arm robot 12R is moved by mistake when the intention is to move the left-arm robot 12L. By providing this main back processing unit 211b, such errors can be avoided, and the control of the left-arm robot 12L and the right-arm robot 12R can be made consistent, improving the accuracy of robot control.

[0135] Furthermore, by providing a main back processing unit 211b and a first sub-back processing unit 211ba for the left arm and a second sub-back processing unit 211bb for the right arm to separate the functions, even if, for example, an AGV (Automatic Guided Vehicle) carrying the left arm robot 12L and the right arm robot 12R is to be added to the robot 20, it is only necessary to add a new back processing unit for the AGV, making it possible to achieve a pluggable design.

[0136] 12 includes a robot 20, a first natural language processing system 51, a second natural language processing system 52, a third natural language processing system 53, and a fourth natural language processing system 54. The robot 20 and the natural language processing systems 51, 52, 53, and 54 are connected to each other so as to be able to send and receive data via a network. In this embodiment, as in the first embodiment, the robot 20 and the natural language processing systems 51, 52, 53, and 54 are connected to each other so as to be able to send and receive data via the network. However, a large-scale language model (LLM) (e.g., Calm2) that can be executed locally without using a network may be used as the natural language processing systems 51, 52, 53, and 54.

[0137] The robot 20 includes a control device 21, a left arm robot 12L, a right arm robot 12R, a microphone 13, and a speaker 14.

[0138] In this embodiment, an example of robot control in which a user orders a drink such as tea from the robot 20 while having a conversation with the robot 20 will be described.

[0139] The first natural language processing system 51, the second natural language processing system 52, the third natural language processing system 53, and the fourth natural language processing system 54 are natural language processing systems that utilize a large-scale language model. Upon receiving a request, the first natural language processing system 51, the second natural language processing system 52, the third natural language processing system 53, and the fourth natural language processing system 54 use the large-scale language model to transmit to the request source a sentence that combines one or more probabilistically estimated words that follow the sentence included in the received request.

[0140] In the robot system 200 of this embodiment, the robot 20 transmits prompts to the natural language processing systems 51, 52, 53, and 54 to inquire about the operation of the robot 20, and the robot 20 is operated based on the responses from the natural language processing systems 51, 52, 53, and 54.

[0141] The control device 21 is a control device for controlling the robot 20 including the left-arm robot 12L (first robot) and the right-arm robot 12R (second robot) based on interaction with a user, and includes a processing unit 211, a memory unit 212, a voice input unit 114, a voice recognition unit 115, a voice synthesis unit 116, and a voice output unit 117. These are communicatively connected to each other via, for example, a bus. In this embodiment, the processing unit 211, the voice recognition unit 115, and the voice synthesis unit 116 are configured separately, but the voice recognition unit 115 and the voice synthesis unit 116 may be part of the functional configuration of the processing unit 211. The control device 21 also includes a communication unit (not shown). The communication unit is realized, for example, by a circuit connected to a network. The communication unit communicates with external devices such as natural language processing systems 51, 52, 53, and 54 via the network.

[0142] The processing unit 211 is configured with, for example, a CPU (Central Processing Unit) and a RAM (Random Access Memory) used as a main storage device when the CPU executes processing. The CPU, for example, loads a program stored in the storage unit 212 into the RAM and executes the program, thereby realizing various functions corresponding to the program.

[0143] The processing unit 211, for example, sends prompts to the first natural language processing system 51, the second natural language processing system 52, the third natural language processing system 53, and the fourth natural language processing system 54, which utilize a large-scale language model, receives responses from the first natural language processing system 51, the second natural language processing system 52, the third natural language processing system 53, and the fourth natural language processing system 54, and causes the robot 20 to execute a functional module, described later, identified based on the received responses.

[0144] The processing unit 211 includes, as functional components, a front processing unit 111f, a main back processing unit 211b, a first sub-back processing unit 211ba, and a second sub-back processing unit 211bb.

[0145] The front processing unit 111f is a processing unit for conversing with a user using the first natural language processing system 51. The main back processing unit 111b is a processing unit for managing the control of the robot 20 using the second natural language processing system 52. The first sub-back processing unit 211ba is a processing unit for controlling the left arm robot 12L using the third natural language processing system 53. The second sub-back processing unit 211bb is a processing unit for controlling the right arm robot 12R using the fourth natural language processing system 54. As an example, GPT-3.5 can be used for the first natural language processing system 51 with an emphasis on speed, and GPT-4 can be used for the second natural language processing system 52, the third natural language processing system 53, and the fourth natural language processing system 54 with an emphasis on accuracy.

[0146] The storage unit 212 is a so-called auxiliary storage device that includes a nonvolatile storage circuit such as a hard disk drive (HDD) for storing various information and a solid state drive (SSD). Note that the storage unit 212 may also be a drive or the like that reads and writes various information from and to portable storage media such as CD-RW, DVD-RAM, and flash memory.

[0147] The storage unit 212 includes a front storage unit 112f for the front processing unit 111f, a main back storage unit 212b for the main back processing unit 211b, a first sub-back storage unit 212ba for the first sub-back processing unit 211ba, and a second sub-back storage unit 212bb for the second sub-back processing unit 211bb. Details of the storage unit 212 will be described later.

[0148] The left arm robot 12L includes a robot control unit 121L, a first axis L1, a second axis L2, a third axis L3, and a fourth axis L4.

[0149] The robot control unit 121L receives operation commands for the left arm robot 12L generated by the first sub-back processing unit 211ba by executing the function module, and controls each part of the left arm robot 12L in accordance with the operation commands. Specifically, the robot control unit 121L is a computer equipped with hardware such as a calculation unit such as a CPU, a main storage unit such as a semiconductor memory, an auxiliary storage unit such as a hard disk, and a communication unit.

[0150] The first axis L1, the second axis L2, the third axis L3, and the fourth axis L4 are rotation axes for moving left arm joints (not shown) provided on the left arm robot 12L using motors (not shown). By independently controlling these multiple axes L1 to L4, the left arm robot 12L can be made to perform intended movements.

[0151] The right-arm robot 12R, like the left-arm robot 12L, includes a robot control unit 121R, a first axis R1, a second axis R2, a third axis R3, and a fourth axis R4.

[0152] The robot control unit 121R receives operation commands for the right-arm robot 12R generated by the second sub-back processing unit 211bb by executing the function module, and controls each part of the right-arm robot 12R in accordance with the operation commands. Specifically, the robot control unit 121R is a computer equipped with hardware such as a calculation unit such as a CPU, a main storage unit such as a semiconductor memory, an auxiliary storage unit such as a hard disk, and a communication unit.

[0153] The first axis R1, the second axis R2, the third axis R3, and the fourth axis R4 are rotation axes for moving right arm joints (not shown) provided on the right arm robot 12R using motors (not shown). By independently controlling these multiple axes R1 to R4, the right arm robot 12R can be made to perform intended movements.

[0154] The storage unit 212 will be described in detail below. The front storage unit 112f is similar to the front storage unit 112f according to the first embodiment. That is, the front function list stores information on a plurality of functional modules executed by the front processing unit 111f. The plurality of functional modules are a speech module (Speak) and a main back module (ToBack). The speech module is a functional module that causes the speech output unit 117 to output speech based on a response from the first natural language processing system 51 obtained by transmitting to the first natural language processing system 51 a prompt including instruction data from a user or a prompt including instruction data based on a response from the second natural language processing system 52 transmitted from the main back processing unit 211b. The main back module is a functional module that causes the main back processing unit 211b to transmit an instruction based on a response from the first natural language processing system 51. Note that in this embodiment, there is no search server, and therefore the functional modules Weather and WebSearch in FIG. 2 are not included. However, a search server may be provided and these functional modules may be used.

[0155] The main back storage unit 212b includes a main back function list storage unit 212b1, a main back history log storage unit 212b2, and a main back prerequisite storage unit 212b3.

[0156] The first sub-back storage unit 212ba includes a first sub-back function list storage unit 212ba1, a first sub-back history log storage unit 212ba2, and a first sub-back prerequisite storage unit 212ba3.

[0157] The second sub-back storage unit 212bb includes a second sub-back function list storage unit 212bb1, a second sub-back history log storage unit 212bb2, and a second sub-back prerequisite storage unit 212bb3.

[0158] 13 is a diagram showing an example of a main back function list stored in the main back function list storage unit 212b1. The main back function list storage unit 212b1 stores information on multiple function modules executed by the main back processing unit 211b. In the example shown in FIG. 13, the multiple function modules are first sub-back modules (GiveItemToCustomerL, GiveItemToR_L, TakeItemL, MoveL, PutItemL, ReleaseItemL, etc.), second sub-back modules (GiveItemToCustomerR, GiveItemToL_R, TakeItemR, MoveR, PutItemR, ReleaseItemR, etc.), and a front module (ToUser). The first sub-back module is a function module that causes the second natural language processing system 52 to transmit, to the first sub-back queue 211ba1 of the first sub-back processing unit 211ba, an instruction based on a response from the second natural language processing system 52 obtained by transmitting a prompt including instruction data transmitted from the front processing unit 111f to the main back processing unit 211b. The second sub-back module is a functional module that transmits an instruction based on a response from the second natural language processing system 52 to the second sub-back queue 211bb1 of the second sub-back processing unit 211bb. The front module is a functional module that transmits an instruction based on a response from the second natural language processing system 52 to the front queue 111f1 of the front processing unit 111f.

[0159] The functional module GiveItemToCustomerL is a functional module for instructing the left arm robot 12L to give an item to a customer who is a user.

[0160] The functional module GiveItemToL_R is a functional module for instructing the right-arm robot 12R to give an item to the left-arm robot 12L.

[0161] The functional module GiveItemToR_L is a functional module for instructing the left-arm robot 12L to give an item to the right-arm robot 12R.

[0162] The functional module TakeItemL is a functional module for instructing the left arm robot 12L to take a specified item.

[0163] The functional module TakeItemR is a functional module for instructing the right arm robot 12R to take a specified item.

[0164] The function module MoveL is a function module for instructing the left arm robot 12L to move to a specified location.

[0165] The functional module MoveR is a functional module for instructing the right arm robot 12R to move to a specified location.

[0166] The functional module PutItemL is a functional module for instructing the left arm robot 12L to put down an item.

[0167] The functional module PutItemR is a functional module for instructing the right arm robot 12R to put down an item.

[0168] The function module ReleaseItemL is a function module for instructing the left arm robot 12L to release an item.

[0169] The function module ReleaseItemR is a function module for instructing the right arm robot 12R to release an item.

[0170] The function module ToUser is a function module that transmits the response from the second natural language processing system 52 as is to the front queue 111f1 of the front processing unit 111f.

[0171] FIG. 14 is a diagram showing an example of a first sub-back function list stored in the first sub-back function list storage unit 212ba1. The first sub-back function list storage unit 212ba1 stores information on multiple function modules executed by the first sub-back processing unit 211ba. In the example shown in FIG. 14, the multiple function modules are first operation modules (Move, Grab, Release) and a first main back module (ToUser). The first operation modules are function modules for causing the left arm robot 12L to perform an operation based on a response from the third natural language processing system 53 obtained by transmitting a prompt including instruction data transmitted to the first sub-back processing unit 211ba to the third natural language processing system 53. The first main back module is a function module for transmitting an instruction based on the response from the third natural language processing system 53 to the main back queue 211b1 of the main back processing unit 211b.

[0172] The function module Move is a function module for moving to a specified location.

[0173] The function module Grab is a function module for performing a grabbing action.

[0174] The function module Release is a function module for performing a release operation.

[0175] The function module ToUser is a function module that transmits the response from the third natural language processing system 53 as is to the main back queue 211b1 of the main back processing unit 211b.

[0176] FIG. 15 is a diagram showing an example of a second sub-back function list stored in the second sub-back function list storage unit 212bb1. The second sub-back function list storage unit 212bb1 stores information on multiple function modules executed by the second sub-back processing unit 211bb. In the example shown in FIG. 15, the multiple function modules are second operation modules (Move, Grab, Release) and a second main back module (ToUser). The second operation modules are function modules for causing the right arm robot 12R to perform an operation based on a response from the fourth natural language processing system 54 obtained by transmitting a prompt including instruction data transmitted to the second sub-back processing unit 211bb to the fourth natural language processing system 54. The second main back module is a function module for transmitting an instruction based on the response from the fourth natural language processing system 54 to the main back queue 211b1 of the main back processing unit 211b. The contents of each function module are similar to those of the function modules shown in FIG. 14, and therefore description thereof will be omitted.

[0177] The front-end processing unit 111f transmits to the first natural language processing system 51 a front-end function list included in a prompt including instruction data from the user or a prompt including instruction data based on a response from the second natural language processing system 52 transmitted from the main-back processing unit 211b, receives the response from the first natural language processing system 51, and identifies a functional module to be executed from the front-end function list based on the received response. If the identified functional module is a speech module Speak, the front-end processing unit 111f executes the speech module Speak to output speech based on the response from the speech output unit 117, and if the identified functional module is a main-back module ToBack, the front-end processing unit 111f executes the main-back module ToBack to transmit an instruction based on the response to the main-back queue 211b1 of the main-back processing unit 211b.

[0178] The main back processing unit 211b transmits a main back function list to the second natural language processing system 52 together with a prompt containing instruction data transmitted from the front processing unit 111f to the main back processing unit 211b, receives a response from the second natural language processing system 52, and identifies a functional module to be executed from the main back function list based on the received response. If the identified functional module is a first sub-back module, the main back processing unit 211b executes the first sub-back module to transmit an instruction based on the response to a first sub-back queue 211ba1 of the first sub-back processing unit 211ba, if the identified functional module is a second sub-back module, executes the second sub-back module to transmit an instruction based on the response to a second sub-back queue 211bb1 of the second sub-back processing unit 211bb, and if the identified functional module is a front module ToUser, executes the front module ToUser to transmit an instruction based on the response to the front processing unit 111f.

[0179] The first sub-back processing unit 211ba sends the first sub-back function list to the third natural language processing system 53 along with a prompt containing instruction data sent from the main back processing unit 211b to the first sub-back processing unit 211ba, receives a response from the third natural language processing system 53, and identifies the function module to be executed from the first sub-back function list based on the received response.

[0180] If the identified functional module is the first operation module (Move, Grab, Release), the first sub-back processing unit 211ba executes the first operation module (Move, Grab, Release) to cause the left arm robot 12L to perform an operation based on the answer, and if the identified functional module is the first main back module ToUser, the first sub-back processing unit 211ba executes the first main back module ToUser to send an instruction based on the answer to the main back queue 211b1 of the main back processing unit 211b.

[0181] The second sub-back processing unit 211bb sends to the fourth natural language processing system 54 a prompt containing instruction data sent from the main back processing unit 211b to the second sub-back processing unit 211bb, including the second sub-back function list, receives a response from the fourth natural language processing system 54, and identifies the function module to be executed from the second sub-back function list based on the received response.

[0182] If the identified functional module is the second operation module (Move, Grab, Release), the second sub-back processing unit 211bb executes the second operation module (Move, Grab, Release) to cause the right arm robot 12R to perform an operation based on the answer, and if the identified functional module is the second main back module ToUser, the second sub-back processing unit 211bb executes the second main back module ToUser to send an instruction based on the answer to the main back queue 211b1 of the main back processing unit 211b.

[0183] 16 is a diagram showing an example of the main back preconditions stored in the main back precondition storage unit 212b3. As shown in FIG. 16, the main back preconditions include "About you," "Your role," "About the user," "About robot L and robot R," "About the workspace," "Tasks that robot L can execute," "Tasks that robot R can execute," "Output format," etc.

[0184] FIG. 17 is a diagram showing an example of the first sub-back prerequisites stored in the first sub-back prerequisites storage unit 211ba3. FIG. 18 is a diagram showing an example of the second sub-back prerequisites stored in the second sub-back prerequisites storage unit 211bb3. As shown in FIGS. 17 and 18, the first sub-back prerequisites and second sub-back prerequisites include "About you," "Your role," "About the user," "About robot L and robot R," "About the workspace," "Tasks that robot L can perform," "Tasks that robot R can perform," "Examples," "Output format," etc.

[0185] FIG. 19 is a diagram showing an example of the main back history log stored in the main back history log storage unit 212b2.

[0186] Line No. 1 is log data indicating that the front processing unit 111f executed the function module ToBack, causing the instruction "Please serve tea" to be sent to the main back queue 211b1 of the main back processing unit 211b and stored in the main back queue 211b1. Line No. 2 is log data indicating that the second natural language processing system 52 selected the function module TakeItemL (tea) from the main back function list in response to the instruction "Please serve tea."

[0187] Line No. 3 is log data showing that execution of the functional module TakeItemL (tea) generates an instruction for the left arm robot 12L to take tea, and in response to that instruction, the third natural language processing system 53 determines one by one the functional modules for executing the series of actions up to the left arm robot 12L taking the tea, which are executed one by one by the first sub-back processing unit 211ba, and finally the third natural language processing system 53 determines the functional module ToUser (Done).

[0188] Line No. 4 is log data indicating that the second natural language processing system 52 determined the function module GiveItemToCustomer from the main back function list in response to the instruction "Done" sent to the main back queue 211b1 of the main back processing unit 211b by executing the function module ToUser(Done).

[0189] In this way, the main back history log stores a history log of when the user first orders tea from the robot 20 and then black tea.

[0190] Fig. 20 is a diagram showing an example of a first sub-back history log stored in the first sub-back history log storage unit 212ba2. Fig. 21 is a diagram showing an example of a second sub-back history log stored in the second sub-back history log storage unit 212bb2. The first sub-back history log in Fig. 20 and the second sub-back history log in Fig. 21 correspond to the main back history log in Fig. 19, so a description thereof will be omitted.

[0191] Moreover, the operation of the front processing unit 111f according to this embodiment is similar to the operation of the front processing unit 111f according to the first embodiment, and therefore a description thereof will be omitted.

[0192] In addition, the operations of the main back processing unit 211b, the first sub-back processing unit 211ba, and the second sub-back processing unit 211bb in this embodiment are the same as the operations of the back processing unit 111b in the first embodiment, except for the difference in functional modules, so explanations will be omitted.

[0193] Moreover, the operations of the robot control units 121L and 121R according to this embodiment are also similar to the operations of the robot control unit 121 according to the first embodiment, and therefore, description thereof will be omitted.

[0194] As described above, in the control device 21 according to this embodiment, the tasks of the natural language processing systems 51, 52, 53, and 54 are divided among the front processing unit 111f for conversing with the user, the main back processing unit 211b for managing the control of the robot 20, the first sub-back processing unit 211ba for controlling the left-arm robot 12L, and the second sub-back processing unit 211bb for controlling the right-arm robot 12R. This allows the main back processing unit 211b, the first sub-back processing unit 211ba, and the second sub-back processing unit 211bb to control the robot 20, so that the user can converse with the user through the front processing unit 111f without waiting for the completion of execution of control of the robot 20.

[0195] (Other embodiments) The above describes embodiments of the present disclosure, but the present disclosure should not be construed as being limited to the above embodiments, and can be applied to various embodiments and combinations within the scope that does not deviate from the gist of the present disclosure.

[0196] Furthermore, the processing flow described in the above embodiment is also an example, and unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged within the scope of the present invention.

[0197] It may be provided in the following manner. (Aspect 1) A control device (11) for controlling a robot based on an interaction with a user, an input unit (114) that receives instructions from the user for the robot; an output unit (117) that outputs a voice related to a response to the instruction to the user; a front-end processing unit (111f) for conducting a conversation with the user using a first natural language processing system that utilizes a large-scale language model; a back processing unit (111b) for controlling the robot using a second natural language processing system that utilizes a large-scale language model; Equipped with the front processing unit is capable of executing a speech module for causing the output unit to output a speech based on a response from the first natural language processing system, the speech being obtained by sending a prompt including data of an instruction from the user to the first natural language processing system, and a back module for causing the back processing unit to transmit an instruction based on the response from the first natural language processing system, The back processing unit is a control device that is capable of executing an operation module for causing the robot to perform an operation based on a response from the second natural language processing system, the operation module being obtained by sending a prompt including the instruction data sent to the back processing unit to the second natural language processing system, and a front module for sending an instruction based on the response from the second natural language processing system to the front processing unit.

[0198] (Aspect 2) a front function list storage unit that stores a front function list of a plurality of function modules that are executed by the front processing unit; a back function list storage unit that stores a back function list of a plurality of function modules that are executed by the back processing unit; the front-end processing unit transmits the prompt including the front-end function list to the first natural language processing system; 2. The control device according to aspect 1, wherein the back processing unit transmits the prompt to the second natural language processing system together with the back function list.

[0199] (Aspect 3) a front-end function list storage unit that stores a front-end function list in which a plurality of function modules executed by the front-end processing unit are associated with two types of priorities; The front processing unit includes: When a functional module having a higher priority among the plurality of back modules is executed, an instruction based on the answer is transmitted to the back processing unit, and the back processing unit is caused to store the instruction based on the answer in a priority memory within the back processing unit; When a functional module having a low priority among the plurality of back modules is executed, an instruction based on the answer is sent to the back processing unit, and the back processing unit is caused to store the instruction based on the answer in a back queue within the back processing unit; The control device according to aspect 1 or 2, wherein, when the instruction data is stored in the priority memory, the back processing unit includes the instruction data stored in the priority memory in the prompt and transmits it to the second natural language processing system in priority over the instruction data stored in the back queue.

[0200] (Aspect 4) A control device according to aspect 3, wherein a functional module with a low priority among the back modules is a functional module that sends the answer generated by execution of the functional module to the back processing unit and causes the back processing unit to store the answer in the back queue within the back processing unit.

[0201] (Aspect 5) A control device according to aspect 3 or 4, wherein a functional module with a high priority among the back modules is a functional module that transmits a request generated by execution of the functional module to the back processing unit and causes the back processing unit to store the request in the priority memory within the back processing unit.

[0202] (Aspect 6) a front precondition storage unit that stores a front precondition that defines information about a function module that can be executed by the back processing unit; 6. The control device according to any one of aspects 1 to 5, wherein the front-end processing unit includes the front-end precondition in the prompt to be sent to the first natural language processing system.

[0203] (Aspect 7) 7. The control device according to aspect 6, wherein the front-end precondition defines a policy regarding the answer generated by the first natural language processing system.

[0204] (Aspect 8) A control device according to aspect 6 or 7, wherein the front preconditions specify information about a workspace in which the robot operates.

[0205] (Aspect 9) a back precondition storage unit that stores back preconditions that define a work procedure for an operation to be performed by the robot; 9. The control device according to any one of aspects 1 to 8, wherein the back processing unit includes the back precondition in the prompt to be sent to the natural language processing system.

[0206] (Aspect 10) A control device as described in aspect 9, wherein the back precondition specifies information about a workspace in which the robot operates.

[0207] (Aspect 11) 11. The control device according to aspect 9 or 10, wherein the back precondition defines a policy regarding the answer generated by the second natural language processing system.

[0208] (Aspect 12) A control device (21) for controlling robots including a first robot and a second robot based on an interaction with a user, an input unit (114) that receives instructions from the user for the robot; an output unit (117) that outputs a voice related to a response to the instruction to the user; a front-end processing unit (111f) for conducting a conversation with the user using a first natural language processing system that utilizes a large-scale language model; a main back processing unit (211b) for controlling the robot using a second natural language processing system that utilizes a large-scale language model; a first sub-back processing unit (211ba) for controlling the first robot using a third natural language processing system that utilizes a large-scale language model; a second sub-back processing unit (211bb) for controlling the second robot using a fourth natural language processing system that utilizes a large-scale language model; Equipped with The front processing unit includes: a speech module for causing the output unit to output a speech based on a response from the first natural language processing system, the speech being obtained by transmitting a prompt including instruction data from the user to the first natural language processing system; and a main back module for transmitting an instruction based on the response from the first natural language processing system to the main back processing unit, the main back processing unit is capable of executing a first sub-back module that causes the second natural language processing system to transmit to the first sub-back processing unit an instruction based on a response from the second natural language processing system, the instruction being obtained by transmitting a prompt including data of the instruction transmitted to the main back processing unit; a second sub-back module that causes the second sub-back processing unit to transmit the instruction based on the response from the second natural language processing system; and a front module that causes the front processing unit to transmit the instruction based on the response from the second natural language processing system, the first sub-back processing unit is capable of executing a first operation module for causing the first robot to perform an operation based on a response from the third natural language processing system, the response being obtained by transmitting a prompt including data of the instruction transmitted to the first sub-back processing unit to the third natural language processing system, and a first main back module for causing the first robot to transmit an instruction based on the response from the third natural language processing system to the main back processing unit; The second sub-back processing unit is a control device that is capable of executing a second operation module for causing the second robot to perform an operation based on an answer from the fourth natural language processing system obtained by sending a prompt including the instruction data sent to the second sub-back processing unit to the fourth natural language processing system, and a second main back module for sending an instruction based on the answer from the fourth natural language processing system to the main back processing unit.

[0209] (Aspect 13) a front function list storage unit that stores a front function list of a plurality of function modules that are executed by the front processing unit; a main back function list storage unit that stores a main back function list of a plurality of function modules that are executed by the main back processing unit; a first sub-back function list storage unit that stores a first sub-back function list of a plurality of function modules that are executed by the first sub-back processing unit; a second sub-back function list storage unit that stores a second sub-back function list of a plurality of function modules executed by the second sub-back processing unit; the front-end processing unit transmits the prompt including the front-end function list to the first natural language processing system; the main back processing unit transmits the prompt including the main back function list to the second natural language processing system; the first sub-back processing unit transmits the prompt including the first sub-back function list to the third natural language processing system; 13. The control device according to aspect 12, wherein the second sub-back processing unit transmits the prompt to the fourth natural language processing system together with the second sub-back function list.

[0210] (Aspect 14) A control method for a control device including a front-end processing unit for conducting a conversation with a user using a first natural language processing system utilizing a large-scale language model, for controlling a robot based on an interaction with the user, and a back-end processing unit for controlling the robot using a second natural language processing system utilizing the large-scale language model, an input step of receiving an instruction from the user to the robot; a front-end processing step for conducting a conversation with the user using the first natural language processing system; a back processing step for controlling the robot using the second natural language processing system; Equipped with the front-end processing step includes a step of causing an output unit to output a voice based on a response from the first natural language processing system, the voice being obtained by transmitting a prompt including data of an instruction from the user to the first natural language processing system, and a step of causing the back-end processing unit to transmit an instruction based on the response from the first natural language processing system, The back processing step includes a step of causing the robot to perform an action based on an answer from the second natural language processing system obtained by sending a prompt including data of the instruction sent to the back processing unit to the second natural language processing system, and a step of causing the robot to send an instruction based on the answer from the second natural language processing system to the front processing unit.

[0211] (Aspect 15) A control program for a control device, comprising: a front-end processing unit for conducting a conversation with a user using a first natural language processing system utilizing a large-scale language model, for controlling a robot based on an interaction with the user; and a back-end processing unit for controlling the robot using a second natural language processing system utilizing the large-scale language model, On the computer, an input step of receiving an instruction from the user to the robot; a front-end processing step for conducting a conversation with the user using the first natural language processing system; a back processing step for controlling the robot using the second natural language processing system; Execute the front-end processing step includes a step of causing an output unit to output a voice based on a response from the first natural language processing system, the voice being obtained by transmitting a prompt including data of an instruction from the user to the first natural language processing system, and a step of causing the back-end processing unit to transmit an instruction based on the response from the first natural language processing system, The back processing step includes a step of causing the robot to perform an action based on an answer from the second natural language processing system obtained by sending a prompt including the instruction data sent to the back processing unit to the second natural language processing system, and a step of causing the robot to send an instruction based on the answer from the second natural language processing system to the front processing unit. [Explanation of symbols]

[0212] 100, 200... Robot system, 10, 20... Robot, 11, 21: Control device, 12: Arm robot, 12L···Left arm robot, 12R···Right arm robot, 13. Microphone, 14. Speaker, 51. First natural language processing system, 52. Second natural language processing system, 53. Third natural language processing system, 54···Fourth natural language processing system, 111, 211···Processing unit 111f: Front processing section, 111b: Back processing section, 111f1···Front cue, 111b1···Back cue, 111b2···Priority memory, 211b···Main back processing unit, 211ba: First sub-back processing unit, 211bb: Second sub-back processing unit, 211b1: Main back queue, 211ba1: First sub-back queue, 211bb1: Second sub-back queue, 112, 212: Storage unit, 112f1···Front function list memory section, 112f2: Front history log storage unit, 112f3: Front precondition storage unit, 112b1: Back function list storage unit, 112b2: Back history log storage unit, 112b3···Back precondition storage section, 212b1···Main back function list storage unit, 212b2 ···Main back history log storage section, 212b3···Main back prerequisite memory section, 212ba1: First sub-back function list storage unit, 212ba2: First sub-back history log storage unit, 212ba3···First sub-back prerequisite memory unit, 212bb1···Second sub-back function list storage unit, 212bb2: Second sub-back history log storage unit, 212bb3: Second sub-back prerequisite memory unit, 114: Voice input unit, 116: Speech synthesis unit, 121, 121L, 221R: Robot control unit

Claims

1. A control device (11) for controlling a robot based on an interaction with a user, comprising: an input unit (114) that receives instructions from the user for the robot; an output unit (117) that outputs a voice related to a response to the instruction to the user; a front-end processing unit (111f) for conducting a conversation with the user using a first natural language processing system utilizing a large-scale language model; a back processing unit (111b) for controlling the robot using a second natural language processing system utilizing a large-scale language model; Equipped with the front processing unit is capable of executing a speech module for causing the output unit to output a speech based on a response from the first natural language processing system, the speech being obtained by sending a prompt including data of an instruction from the user to the first natural language processing system, and a back module for causing the back processing unit to transmit an instruction based on the response from the first natural language processing system, The back processing unit is a control device that is capable of executing an operation module for causing the robot to perform an operation based on an answer from the second natural language processing system, which is obtained by sending a prompt including the instruction data sent to the back processing unit to the second natural language processing system, and a front module for sending an instruction based on the answer from the second natural language processing system to the front processing unit.

2. a front function list storage unit that stores a front function list of a plurality of function modules that are executed by the front processing unit; a back function list storage unit that stores a back function list of a plurality of function modules that are executed by the back processing unit; The front processing unit includes: sending the prompt including the front-end function list to the first natural language processing system; The back processing unit includes: The control device of claim 1 , further comprising: transmitting the prompt to the second natural language processing system including the back function list.

3. a front-end function list storage unit that stores a front-end function list in which a plurality of function modules executed by the front-end processing unit are associated with two types of priority; The front processing unit includes: When a functional module having a higher priority among the plurality of back modules is executed, an instruction based on the answer is transmitted to the back processing unit, and the back processing unit is caused to store the instruction based on the answer in a priority memory within the back processing unit; When a functional module having a low priority among the plurality of back modules is executed, an instruction based on the answer is sent to the back processing unit, and the back processing unit is caused to store the instruction based on the answer in a back queue within the back processing unit; 2. The control device according to claim 1, wherein, when the instruction data is stored in the priority memory, the back processing unit includes the instruction data stored in the priority memory in the prompt and transmits the instruction data stored in the priority memory to the second natural language processing system in priority over the instruction data stored in the back queue.

4. 4. The control device according to claim 3, wherein a functional module having a low priority among the back modules is a functional module that transmits the answer generated by execution of the functional module to the back processing unit and causes the back processing unit to store the answer in the back queue within the back processing unit.

5. 4. The control device according to claim 3, wherein a functional module having a high priority among the back modules is a functional module that transmits a request generated by execution of the functional module to the back processing unit and causes the back processing unit to store the request in the priority memory within the back processing unit.

6. a front precondition storage unit that stores a front precondition that defines information about a function module that can be executed by the back processing unit; The control device according to claim 1 , wherein the front-end processing unit includes the front-end precondition in the prompt sent to the first natural language processing system.

7. The control device according to claim 6 , wherein the front-end precondition defines a policy for the answer generated by the first natural language processing system.

8. The control device according to claim 6 , wherein the front precondition defines information about a workspace in which the robot operates.

9. a back precondition storage unit that stores back preconditions that define a work procedure for an operation to be performed by the robot; The control device according to claim 1 , wherein the back processing unit includes the back precondition in the prompt sent to the natural language processing system.

10. The control device according to claim 9 , wherein the back precondition defines information about a workspace in which the robot operates.

11. The control device according to claim 9 , wherein the back precondition defines a policy for the answer generated by the second natural language processing system.

12. A control device (21) for controlling robots including a first robot and a second robot based on an interaction with a user, an input unit (114) that receives instructions from the user for the robot; an output unit (117) that outputs a voice related to a response to the instruction to the user; a front-end processing unit (111f) for conducting a conversation with the user using a first natural language processing system utilizing a large-scale language model; a main back processing unit (211b) for controlling the robot using a second natural language processing system that utilizes a large-scale language model; a first sub-back processing unit (211ba) for controlling the first robot using a third natural language processing system that uses a large-scale language model; a second sub-back processing unit (211bb) for controlling the second robot using a fourth natural language processing system that uses a large-scale language model; Equipped with The front processing unit includes: a speech module for causing the output unit to output a speech based on a response from the first natural language processing system, the speech being obtained by transmitting a prompt including instruction data from the user to the first natural language processing system; and a main back module for transmitting an instruction based on the response from the first natural language processing system to the main back processing unit, the main back processing unit is capable of executing a first sub-back module that causes the second natural language processing system to transmit to the first sub-back processing unit an instruction based on a response from the second natural language processing system, the instruction being obtained by transmitting a prompt including data of the instruction transmitted to the main back processing unit; a second sub-back module that causes the second sub-back processing unit to transmit the instruction based on the response from the second natural language processing system; and a front module that causes the front processing unit to transmit the instruction based on the response from the second natural language processing system, the first sub-back processing unit is capable of executing a first operation module for causing the first robot to perform an operation based on a response from the third natural language processing system, the response being obtained by transmitting a prompt including data of the instruction transmitted to the first sub-back processing unit to the third natural language processing system, and a first main back module for causing the first robot to transmit an instruction based on the response from the third natural language processing system to the main back processing unit; The second sub-back processing unit is a control device that is capable of executing a second operation module for causing the second robot to perform an operation based on an answer from the fourth natural language processing system, obtained by sending a prompt including the instruction data sent to the second sub-back processing unit to the fourth natural language processing system, and a second main back module for sending an instruction based on the answer from the fourth natural language processing system to the main back processing unit.

13. a front function list storage unit that stores a front function list of a plurality of function modules that are executed by the front processing unit; a main back function list storage unit that stores a main back function list of a plurality of function modules that are executed by the main back processing unit; a first sub-back function list storage unit that stores a first sub-back function list of a plurality of function modules that are executed by the first sub-back processing unit; a second sub-back function list storage unit that stores a second sub-back function list of a plurality of function modules executed by the second sub-back processing unit, the front-end processing unit transmits the prompt including the front-end function list to the first natural language processing system; the main back processing unit transmits the prompt including the main back function list to the second natural language processing system; the first sub-back processing unit transmits the prompt including the first sub-back function list to the third natural language processing system; The control device according to claim 12 , wherein the second sub-back processing unit transmits the prompt to the fourth natural language processing system together with the second sub-back function list.

14. A control method for a control device including a front-end processing unit for conducting a conversation with a user using a first natural language processing system utilizing a large-scale language model, for controlling a robot based on an interaction with the user, and a back-end processing unit for controlling the robot using a second natural language processing system utilizing the large-scale language model, an input step of receiving an instruction from the user to the robot; a front-end processing step for conducting a conversation with the user using the first natural language processing system; a back processing step for controlling the robot using the second natural language processing system; Equipped with the front-end processing step includes a step of causing an output unit to output a voice based on a response from the first natural language processing system, the voice being obtained by transmitting a prompt including data of an instruction from the user to the first natural language processing system, and a step of causing the back-end processing unit to transmit an instruction based on the response from the first natural language processing system, The back processing step includes a step of causing the robot to perform an action based on an answer from the second natural language processing system, the answer being obtained by sending a prompt including data of the instruction sent to the back processing unit to the second natural language processing system, and a step of causing the robot to send an instruction based on the answer from the second natural language processing system to the front processing unit.

15. A control program for a control device, the control program comprising: a front-end processing unit for conducting a conversation with a user using a first natural language processing system utilizing a large-scale language model, for controlling a robot based on an interaction with the user; and a back-end processing unit for controlling the robot using a second natural language processing system utilizing the large-scale language model, On the computer, an input step of receiving an instruction from the user to the robot; a front-end processing step for conducting a conversation with the user using the first natural language processing system; a back processing step for controlling the robot using the second natural language processing system; Execute the front-end processing step includes a step of causing an output unit to output a voice based on a response from the first natural language processing system, the voice being obtained by transmitting a prompt including data of an instruction from the user to the first natural language processing system, and a step of causing the back-end processing unit to transmit an instruction based on the response from the first natural language processing system, The back processing step includes a step of causing the robot to perform an action based on an answer from the second natural language processing system obtained by sending a prompt including the instruction data sent to the back processing unit to the second natural language processing system, and a step of causing the robot to send an instruction based on the answer from the second natural language processing system to the front processing unit.

Images