Control device, robot system, system, control method and control program

Abstract

To speed up task processing in operating a robot.SOLUTION: A control device (11), which controls a robot (15) on the basis of interaction with a user, comprises: an input unit (114) that receives an instruction from the user to the robot; an output unit (116) that outputs response to the instruction to the user; a function list memorizing unit (112a) memorizing a function list of function modules; an input buffer (112b) that stores data on the instruction inputted from the input unit; and a processing unit (111) that transmits a prompt including the function list and the data stored in the input buffer to a natural language processing system (50), receives a response from the natural language processing system, and makes the robot execute a function module identified on the basis of the response. When an execution result of the function module properly executed exists as an estimation result in the function list memorizing unit, the processing unit overwrites the estimation result on the input buffer.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to a technology for controlling a robot based on interaction with a user. [Background technology]

[0002] In recent years, robots that can move flexibly based on user interaction via instructions from the user in natural language have been attracting attention.

[0003] Patent Document 1 describes a workflow management method in which the next task is derived after one task is completed. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Laid-Open No. 2010-170230 Summary of the Invention [Problem to be solved by the invention]

[0005] However, when the workflow management method described in Patent Document 1 is combined with a natural language processing system that uses a large-scale language model that enables interaction through natural language, the natural language processing system takes time to derive tasks, which results in a waiting time each time a task for robot operation is deduced, and therefore it takes time to process the task for robot operation.

[0006] In view of the above problems, an object of the present invention is to speed up task processing for robot operation. [Means for solving the problem]

[0007] The present invention employs the following technical solutions to solve the above problems. The reference symbols in parentheses in the claims and this section are merely examples showing the correspondence with the specific solutions described in the embodiments below as one aspect, and do not limit the technical scope of the present invention.

[0008] A control device (11, 21) according to one aspect of the present invention is a control device for controlling a robot (15, 25) based on an interaction with a user, and includes an input unit (114) that receives instructions from the user for the robot, an output unit (116) that outputs a response to the instructions to the user, a function list storage unit (112a) in which each of a plurality of predetermined operations to be executed by the robot is defined as a function module and a function list of the function modules is stored, an input buffer (112b) that saves data of the instructions input from the input unit, and a natural language processing unit (112a) that uses a large-scale language model. and a processing unit (111, 211) that sends a prompt including the function list and data read from the input buffer to the system (50), receives a response from the natural language processing system, and causes the robot to execute a function module identified by the response, wherein the function list storage unit stores an execution result that is returned when the function module is normally executed as an estimated result, corresponding to the function module, and when the estimated result of the function module identified by the response exists in the function list storage unit, the processing unit overwrites the input buffer with the estimated result.

[0009] With the above configuration, the processing unit can execute a functional module identified by a response from the natural language processing system, thereby causing the natural language processing system to infer the next response using the inference result of the functional module, without actually operating the robot. As a result, if the inference result of the functional module exists in the function list storage unit, the processing unit can infer the functional module on the assumption that the execution result of the functional module is normal, and can infer the robot's next operation without waiting for the completion of execution of the functional module. This can therefore speed up task processing of robot operation.

[0010] A robot system (10) according to one embodiment of the present invention includes the above-described control device (11), processing devices (16a, 16b) including a functional module execution unit that executes the functional module, and a robot (15) that is controlled by operation commands generated by the functional module execution unit executing the functional module. A system (100) according to one aspect of the present invention includes the robot system (10) and the natural language processing system (50).

[0011] A robot system (20) according to one aspect of the present invention includes the above-described control device (21) and the above-described robot (25).

[0012] A system (200) according to one aspect of the present invention includes the above-described control device (21), the above-described robot (25), and the above-described natural language processing system (50).

[0013] a function list read step of reading a function list from a function list storage unit storing a function list of the functional modules, each of which is defined as a functional module; a storage step of saving data of the instructions input by the input step in an input buffer; and a processing step of sending a prompt including the function list and the data read from the input buffer to a natural language processing system using a large-scale language model, receiving a response from the natural language processing system, and causing the robot to execute a functional module identified by the response. The function list storage unit stores an execution result returned when the functional module is executed normally as an estimated result, corresponding to the functional module. The processing step overwrites the input buffer with the estimated result if the estimated result of the functional module identified by the response exists in the function list storage unit.

[0014] a function list read step of reading a function list from a function list storage unit storing a function list of the functional modules, each of which is defined as a functional module; a storage step of saving data of the instructions input by the input step in an input buffer; and a processing step of sending a prompt including the function list and the data read from the input buffer to a natural language processing system using a large-scale language model, receiving a response from the natural language processing system, and causing the robot to execute a functional module identified by the response. The function list storage unit stores an execution result returned when the functional module is executed normally as an estimated result, corresponding to the functional module. The processing step overwrites the input buffer with the estimated result if the estimated result of the functional module identified by the response exists in the function list storage unit. [Effects of the Invention]

[0015] According to the present invention, it is possible to provide a control device and the like that can speed up task processing of robot operation. [Brief explanation of the drawings]

[0016] [Figure 1] FIG. 1 illustrates an example of a system configuration according to a first embodiment. [Figure 2] FIG. 10 is a diagram showing an example of a function list. [Figure 3] FIG. 10 is a diagram illustrating an example of a history log. [Figure 4A] FIG. 10 is a diagram illustrating an example of a precondition. [Figure 4B] FIG. 10 is a diagram illustrating an example of a precondition. [Figure 5]4 is a flowchart illustrating the flow of operations of an inference unit according to the first embodiment. [Figure 6] 10 is a flowchart showing a process of a queue transmission unit according to the first embodiment. [Figure 7] 5 is a flowchart showing the processing of a function module execution unit according to the first embodiment; [Figure 8] 10 is a flowchart showing the processing of a robot control unit. [Figure 9] FIG. 10 illustrates an example of a system configuration according to a second embodiment. [Figure 10] FIG. 10 is a diagram illustrating an example of a thread list. [Figure 11] 10 is a flowchart illustrating the flow of operations of a processing unit according to the second embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0017] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. Note that the embodiment described below shows an example of how the present invention can be implemented, and the present invention is not limited to the specific configuration described below. When implementing the present invention, a specific configuration corresponding to the embodiment may be appropriately adopted.

[0018] (First embodiment) Fig. 1 is a diagram showing an example of the configuration of a system 100 according to a first embodiment. The system 100 shown in Fig. 1 includes a robot system 10, a natural language processing system 50, and a search server 60. The robot system 10 and the natural language processing system 50 are connected to each other so as to be able to send and receive data via a network. The robot system 10 and the search server 60 are connected to each other so as to be able to send and receive data via the network.

[0019] The robot system 10 includes a control device 11, a mobile robot 12, a microphone 13, a speaker 14, and processing units 16a and 16b. In this specification, the mobile robot 12, the microphone 13, and the speaker 14 are collectively referred to as a robot 15.

[0020] In this embodiment, an example of robot control will be described in which a user orders a drink from the robot 15 at a certain table and the robot 15 brings the ordered drink to the table. When the user orders a drink from the robot 15, the user may ask the robot 15 for recommended drinks based on the weather, etc.

[0021] The natural language processing system 50 is a natural language processing system that uses a large-scale language model (LLM). A large-scale language model is a deep learning model that is generated by pre-training based on a huge amount of data on what is called a language model that models natural language, spoken by humans, based on its occurrence probability. As the natural language processing system 50 that uses a large-scale language model, for example, GPT-3, GPT-3.5, or GPT-4 can be used. When the natural language processing system 50 receives a request, it uses the large-scale language model to statistically estimate the probability of generating the next word from the sentence included in the received request, and transmits the estimation result to the request source.

[0022] The search server 60 performs a search process based on a search request (an example of an operation command) sent from the robot system 10, and provides information corresponding to the search request by sending a search result (an example of an execution result) indicating the result of the search process to the robot system 10.

[0023] In the system 100 of this embodiment, the robot system 10 transmits a prompt to the natural language processing system 50 inquiring about the next action of the robot 15, and operates the robot 15 based on the response from the natural language processing system 50.

[0024] The control device 11 is a control device for controlling the robot 15 based on interactions with a user, and includes a processing unit 111, a memory unit 112, a voice input unit 114, a voice recognition unit 115, a voice synthesis unit 116, a voice output unit 117, a mobile robot queue 118, and a search queue 119. These are communicatively connected to each other, for example, via a bus. In this embodiment, the processing unit 111, the voice recognition unit 115, and the voice synthesis unit 116 are each configured separately, but the voice recognition unit 115 and the voice synthesis unit 116 may be part of the functional configuration of the processing unit 111. The control device 11 also includes a communication unit (not shown). The communication unit is realized, for example, by a circuit connected to a network. The communication unit communicates with external devices such as the natural language processing system 50, the processing unit 16a, and the processing unit 16b via the network.

[0025] The processing unit 111 is configured with, for example, a CPU (Central Processing Unit) and a RAM (Random Access Memory) that is used as a main storage device when the CPU executes processing. The CPU, for example, loads a program stored in the storage unit 112 into the RAM and executes the program, thereby realizing various functions corresponding to the program.

[0026] The processing unit 111 transmits a prompt to a natural language processing system 50 that uses a large-scale language model, receives a response from the natural language processing system 50, and causes the robot 15 to execute a function module (described later) identified based on the received response. The processing unit 111 includes, as functional components, an inference unit 111a and a queue transmission unit 111b. The operations of the inference unit 111a and the queue transmission unit 111b will be described in detail below.

[0027] The storage unit 112 is a so-called auxiliary storage device that includes a nonvolatile storage circuit such as a hard disk drive (HDD) for storing various information and a solid state drive (SSD). Note that the storage unit 112 may also be a drive or the like that reads and writes various information from and to portable storage media such as a CD-ROM, a DVD, and a flash memory.

[0028] The voice input unit 114 is connected by wire or wirelessly to the microphone 13 that collects voice and outputs a voice signal, and receives the voice signal from the microphone 13. In other words, the voice input unit 114 receives instructions from the user to the robot 15.

[0029] The speech recognition unit 115 performs speech recognition processing on the speech signal from the speech input unit 114 and outputs text information represented by the speech to the processing unit 111. For example, the speech recognition unit 115 can use a speech recognition technology that converts speech signals into text information using deep learning technology.

[0030] The voice synthesis unit 116 synthesizes voice from the operation command for the robot 15 generated by the execution of the function module identified by the processing unit 111, and outputs the voice to the voice output unit 117. The voice synthesis unit 116 can use a general voice synthesis technology.

[0031] The audio output unit 117 is connected by wire or wirelessly to the speaker 14, which converts an audio signal into audio and outputs it to the outside, and outputs the audio signal from the audio synthesis unit 116 to the speaker 14. In other words, the audio output unit 117 outputs a response to an instruction from the user to the user.

[0032] The mobile robot queue 118 includes, as functional components, an instruction queue 118a and a reply queue 118b. The instruction queue 118a receives from the processing unit 111 an execution instruction for a functional module (functional modules Move and Sleep described later in FIG. 2) identified by the processing unit 111 in response to a response from the natural language processing system 50, and temporarily stores the received execution instruction. The reply queue 118b receives from the functional module execution unit 161a the execution result of the functional module executed by the functional module execution unit 161a described later, and temporarily stores the received execution result.

[0033] Similar to the mobile robot queue 118, the search queue 119 includes an instruction queue 119a and a reply queue 119b as functional components. The instruction queue 119a receives from the processing unit 111 an execution instruction for a functional module (functional module Search in FIG. 2 described later) identified by the processing unit 111 based on a response from the natural language processing system 50, and temporarily stores the received execution instruction. The reply queue 119b receives from the functional module execution unit 161b the execution result of a functional module executed by the functional module execution unit 161b described later, and temporarily stores the received execution result. The reply queue 119b receives from the functional module execution unit 161b the execution result of a functional module executed by the functional module execution unit 161b, and temporarily stores the received execution result.

[0034] The processing device 16a includes a functional module executing unit 161a. The functional module executing unit 161a acquires an execution instruction from the mobile robot queue 118 (instruction queue 118a) in which the execution instruction of the functional module identified by the processing unit 111 is temporarily stored, generates an operation command for the robot 15 by executing the functional module described in the acquired execution instruction, and transmits the generated operation command to the mobile robot 12 and also transmits the execution result of the executed functional module to the mobile robot queue 118 (reply queue 118b). Specifically, the functional module executing unit 161a is a computer equipped with hardware such as a calculation device such as a CPU, a main storage device such as a semiconductor memory, an auxiliary storage device such as a hard disk, and a communication device.

[0035] The processing device 16b has the same configuration as the processing device 16a, and includes a functional module execution unit 161b. The functional module execution unit 161b acquires an execution instruction from the search queue 119 (instruction queue 119a) in which an execution instruction for a functional module identified by the processing unit 111 is temporarily stored, generates a search request by executing the functional module described in the acquired execution instruction, and causes the search server 60 to perform search processing. The functional module execution unit 161b also receives search results indicating the results of the search processing from the search server 60 and transmits the search results to the search queue 119 (reply queue 119b). Specifically, the functional module execution unit 161b is a computer including hardware such as a calculation device such as a CPU, a main storage device such as a semiconductor memory, an auxiliary storage device such as a hard disk, and a communication device.

[0036] The mobile robot 12 includes a robot control unit 121 , a right wheel motor 122 , a left wheel motor 123 , a camera 124 , and a sensor 125 .

[0037] The robot control unit 121 receives the operation commands for the robot 15 generated by the function module execution unit 161a, and controls each unit of the mobile robot 12 in accordance with the operation commands. Specifically, the robot control unit 121 is a computer equipped with hardware such as a calculation unit such as a CPU, a main storage unit such as a semiconductor memory, an auxiliary storage unit such as a hard disk, and a communication unit.

[0038] The right wheel motor 122 and the left wheel motor 123 are electric actuators that respectively rotate and drive the right and left wheels (not shown) of the mobile robot. The rotation speeds of the right wheel motor 122 and the left wheel motor 123 are independently changed depending on the amount of power supplied from a battery (not shown) via an inverter (not shown). This allows the right wheel and the left wheel to rotate at different speeds, and this speed difference between the right wheel and the left wheel enables the mobile robot 12 to change direction.

[0039] The camera 124 captures images of the surroundings of the mobile robot 12. The sensor 125 includes various sensors, such as a sensor for detecting the temperature and humidity around the mobile robot 12, an infrared sensor, and the like.

[0040] The following describes the storage areas provided in storage unit 112: function list storage unit 112a, input buffer 112b, history log storage unit 112c, and prerequisite storage unit 112d.

[0041] (Feature List) The storage unit 112 includes a function list storage unit 112a that stores a function list. Here, a functional module is a package of a series of general-purpose operations of the robot 15. The function list is a collection of usage methods for each functional module, which is provided to the natural language processing system 50 as prior information when the natural language processing system 50 generates functional modules and parameters suitable for task processing. In other words, the function list is a list having multiple functional modules, each of which defines as a functional module a plurality of predetermined operations to be performed by the robot 15.

[0042] Currently, it is difficult to generate a correct program (e.g., low code) for the robot 15 from scratch by sending a request in natural language to the natural language processing system 50. For this reason, simply sending a request in natural language to the natural language processing system 50 will not allow the robot 15 to perform the intended action.

[0043] Therefore, in this embodiment, processing unit 111 transmits to natural language processing system 50 a prompt containing a function list stored in function list storage unit 112a in addition to a natural language instruction from the user.

[0044] This allows the natural language processing system 50 to determine (or identify) from the function list the functional module to be executed by the robot 15 based on the user's instruction. As described above, the functional module defines a predetermined operation to be executed by the robot 15, so the processing unit 111 can identify the functional module based on the response from the natural language processing system 50 regarding which functional module in the function list should be executed, and can cause the robot 15 to execute the identified functional module. Therefore, the robot 15 can appropriately perform the intended operation.

[0045] Furthermore, the functional modules receive as input parameters related to coordinates and / or time required for the operation of the robot 15, or parameters related to the text to be output to the user. In this case, the processing unit 111 transmits the parameters to be used by each functional module, included in a function list, to the natural language processing system 50. The processing unit 111 then receives, from the natural language processing system 50, data identifying a functional module as well as parameter data to be given to the functional module, and provides the identified functional module with the parameter data to be given to the functional module. Even in this case, the natural language processing system 50 transmits, as a response, the data identifying a functional module and the parameter data to be given to the functional module, so that the operation of the robot 15 is limited to the operation defined in the functional module, and the robot can be operated more appropriately using parameter data that matches the function module.

[0046] FIG. 2 is a diagram showing an example of a function list. The function list has data such as "function module name," "parameter," "type," "description," and "estimated result." "Function module name" is the name of the function module. "Parameter" is the parameter given to the function module as described above. "Type" is the data type of the parameter. "Description" is a description of the function module. "Estimated result" is the execution result returned when the function module is executed normally. In this embodiment, function list storage unit 112a stores the execution result returned when the function module is executed normally as an estimated result, corresponding to the function module.

[0047] The functional module Speaker is a functional module that converts designated text into voice and outputs it to the speaker 14. The functional module Move is a functional module that moves the mobile robot 12 to designated coordinates (location). The functional module Sleep is a functional module that causes the mobile robot 12 to wait for a designated time (seconds). The functional module Search is a functional module that performs an internet search using designated text, for example, using the search engine Bing (registered trademark).

[0048] For the functional modules Speaker, Move, and Sleep, estimated results (Finished to speak, Finished to move, Finished to sleep) that are returned when each of these functional modules is executed normally are stored in function list storage unit 112a. For example, when the functional module Speaker is executed, the specified text is converted into speech and output via speaker 14. In other words, assuming that the functional module Speaker is executed normally, the execution result is known in advance without even being executed. In such cases, the execution result that is returned when the functional module is executed normally is stored in function list storage unit 112a as an estimated result, associated with the functional module.

[0049] On the other hand, in the case of the function module Search, for example, when search server 60 is asked to search for today's weather, the search results returned will be "sunny," "cloudy," "rainy," "snowy," "sunny then cloudy," etc., depending on the actual weather. In this way, even if the function module Search is executed normally with a specified text (e.g., "What's the weather today?"), the execution result cannot be determined unless it is actually executed, and the execution result cannot be stored in advance as an estimated result in function list storage unit 112a. Therefore, such a function module (for example, the function module Search) is associated with an estimated result "Unable to expect" and stored in function list storage unit 112a. In the following description, the estimated result of a function module being "Unable to expect" is considered to be synonymous with the fact that the estimated result of the function module does not exist in function list storage unit 112a.

[0050] In the function list shown in Figure 2, the function name keys and JSON values ​​are written in a simple format because they are complicated, but function modules can be implemented in JSON format.

[0051] (History log) The storage unit 112 further includes an input buffer 112b for storing input data, and a history log storage unit 112c for accumulating a history log.

[0052] Input buffer 112b first stores data of instructions from the user input via voice input unit 114. Next, if an estimation result of a functional module identified by a response from natural language processing system 50 exists in function list storage unit 112a, input buffer 112b is overwritten with the estimation result. If an estimation result of a functional module identified by a response from natural language processing system 50 does not exist in function list storage unit 112a, the execution result of the functional module when it is executed is overwritten.

[0053] The history log is log data including data read from the input buffer 112b and responses obtained from the natural language processing system 50 in response to the read data. In this way, a set consisting of an input to the natural language processing system 50 and an output corresponding to the input is added as a history log to the history log storage unit 112c.

[0054] The processing unit 111, which receives a response to one query from the natural language processing system 50, causes the robot 15 to execute one function module included in the response. In this case, a task that requires the execution of multiple function modules cannot be completed.

[0055] Therefore, in this embodiment, after transmitting an instruction in natural language from the user, the processing unit 111 includes the history log accumulated in the history log storage unit 112c in addition to the data read from the input buffer 112b in a prompt and transmits the prompt to the natural language processing system 50.

[0056] As a result, the natural language processing system 50 sends a response based on the data read from the input buffer 112b and the history log, and the processing unit 111 can cause the robot 15 to perform continuous actions based on past interactions based on the history log.

[0057] Consider an example in which an instruction is issued to the robot 15 for the first time to cause it to operate. In this case, the data initially stored in the input buffer 112b is an instruction from the user. At this stage, no data has been accumulated in the history log. Therefore, the natural language processing system 50 outputs a first response to the processing unit 111 based on the user's instruction read from the input buffer 112b. The data read from the input buffer 112b and the response from the natural language processing system 50 corresponding to the read data are added to the history log as a set. Therefore, the instruction from the user and the first response corresponding to the instruction are added to the history log. The processing unit 111 identifies a functional module based on the first response, and if an estimation result of the identified functional module exists in the function list storage unit 112a, the processing unit 111 overwrites the estimation result in the input buffer 112b. Note that if an estimation result of the identified functional module does not exist in the function list storage unit 112a, the processing unit 111 waits for the functional module to be executed and then overwrites the execution result in the input buffer 112b. Then, the processing unit 111 again includes the data read from the input buffer 112b and the history log in a prompt and sends the prompt to the natural language processing system 50. Then, the natural language processing system 50 outputs the next answer to the processing unit 111, and the processing unit 111 identifies the next functional module.

[0058] In this way, the processing unit 111 can use the natural language processing system 50 to cause the robot 15 to execute a functional module that takes into account past history. Once the user gives an instruction to the robot 15, the processing unit 111 continues the process of sending prompts to the natural language processing system 50 until the natural language processing system 50 determines that the robot 15 has completed its operation, receiving an answer as to the operation the robot 15 should perform, and causing the robot 15 to operate, thereby making it possible for the robot 15 to perform continuous operations.

[0059] FIG. 3 is a diagram showing an example of a history log. The first line is log data indicating that the user has issued an instruction to the robot 15 (specifically, the microphone 13) to "make tea that suits today's weather." The second line is log data indicating that the natural language processing system 50, in response to the instruction to "make tea that suits today's weather," has selected the functional module Speaker from the function list and selected "Please wait a moment" as parameter data to be given to the functional module Speaker. As such, the log data on the first and second lines are input to the natural language processing system 50 and output from the natural language processing system 50, respectively, and are added as a set to the history log storage unit 112c as a history log. The third line is log data indicating the estimation result "Finished to speak" of the functional module Speaker that causes the speaker 14 to output the voice "Please wait a moment." The fourth line is log data indicating that when a prompt including the estimation result of the third line is sent to the natural language processing system 50, the natural language processing system 50 determines the functional module Search from the function list and determines "What's the weather like today?" as a parameter to be given to the functional module Search. The fifth line is log data indicating that the functional module Search is executed and the search result "The weather is sunny today" is returned from the search server 60. As described above, since the estimation result of the functional module Search does not exist in the function list storage unit 112a (i.e., the estimation result of the functional module is "Unable to expect"), the estimation result is not added as log data, and the search result is added as log data. The n-1th line is log data indicating the estimation result "Finished to speak" of the functional module Speaker determined in the n-2th line, which causes the speaker 14 to output a voice saying "I've brought you some tea. Please enjoy your stay." The final line, line n, is log data indicating that when a prompt including the estimation result on line n-1 was sent to the natural language processing system 50, the natural language processing system 50 responded that the operation of the robot 15 was "completed." The response of "completed" from the natural language processing system 50 completes the operation of the robot 15.The log data on the third and fourth lines, the fifth and sixth lines, and the (n-1)th and nth lines are respectively input and output to the natural language processing system 50, and are added as a set to the history log. Specifically, the "completion" of an operation refers to the "completion" of the flow in the flowchart described in the prerequisites area 4b, which will be explained next.

[0060] The history log stores a history of inputs and outputs to the natural language processing system 50 as a set, from the first instruction to the robot 15, and therefore allows the natural language processing system 50 to understand what has been executed and to what extent. Specifically, it shows how far the flow chart described in the prerequisites area 4b, which will be explained next, has been executed. By sending this data to the natural language processing system 50, the natural language processing system 50 can obtain an appropriate answer regarding the next action to be taken, taking into account the past history.

[0061] (Prerequisite) The storage unit 112 further includes a prerequisite storage unit 112d that stores prerequisites for the operation of the robot 15. The prerequisites include the work procedures of the robot 15. Specifically, the work procedures of the robot 15 divide the operations to be performed by the robot 15 into multiple stages and specify the operation policies for each stage. These operation policies are written in Unified Modeling Language (UML).

[0062] When the work procedures of the robot 15 are given to the natural language processing system 50 as preconditions, if the preconditions are written in source code, the natural language processing system 50 will follow the source code and will obtain a highly precise answer but will not obtain a flexible answer. On the other hand, if the preconditions are written in sentences using Japanese or English, the expression will be ambiguous, and it may not be possible to obtain an answer that specifies the robot's operation, and the robot cannot be operated appropriately based on such an answer.

[0063] Therefore, in this embodiment, the processing unit 111 transmits to the natural language processing system 50 a prompt containing preconditions stored in the precondition storage unit 112d in addition to instructions in natural language from the user.

[0064] The preconditions define the behavior of the robot 15 in multiple stages and define the behavioral policy for each stage, so the natural language processing system 50 can provide flexible yet appropriate answers in accordance with the behavioral policies. Therefore, based on these answers, the processing unit 111 can make the robot 15 perform appropriate behavior in accordance with the behavioral policy while ensuring flexibility in the behavior of the robot 15.

[0065] 4A and 4B are diagrams showing examples of preconditions. The preconditions written in area 4a define a purpose. For example, area 4a defines that a flowchart should be completed (the "complete the flowchart" part in area 4a). Area 4b defines code data representing the flowchart to be completed using UML (for example, Mermaid). This flowchart corresponds to the above-mentioned operational policy. Since area 4a defines that the flowchart should be completed, the natural language processing system 50 outputs a response to a user instruction to complete the flowchart. Therefore, the natural language processing system 50 can provide the processing unit 111 with a flexible yet appropriate response that conforms to the operational policy.

[0066] Here, the advantages of UML will be explained. For example, area 4b contains the statement "Speak out something like 'Make me a coffee' or 'Please give me lots of water' or 'Please give me a little water'." As indicated by the "something like" in the statement, the natural language processing system 50 does not necessarily need to determine whether the parameter data should be "Make me a coffee," "Please give me lots of water," or "Please give me a little water." Rather, the natural language processing system 50 can independently determine the parameter data as appropriate in accordance with the purpose described in area 4a. This allows the natural language processing system 50 to provide flexible responses while following the operational guidelines described in UML.

[0067] Area 4c contains target location information. This is an example of information about the environment in which the robot 15 operates, and corresponds to parameter data related to coordinates given to the function module Move. For example, giving table 1 to the function module Move results in an operation command to move the robot to table 1.

[0068] Area 4d contains remarks. Remarks may contain various information, such as conditions and restrictions. For example, the remarks include information about the region where the robot 15 is used (the portion of area 4d that reads, "This is a relaxation area located within a factory of DENSO Corporation in Aichi Prefecture."). By including information about the region where the robot 15 is used in the premise information, the natural language processing system 50 can obtain various information, such as weather information and event information, in the region through an Internet search. This allows for more flexible responses to be provided. The remarks also include a statement such as "Stick to the flowchart." Such a statement ensures that the natural language processing system 50 follows the operating policy. Therefore, appropriate responses can be output efficiently. The remarks also include a statement such as "Think step by step." Such a statement enables the natural language processing system 50 to think step by step and return an answer that allows the robot 15 to operate appropriately without making logical leaps.

[0069] Next, a specific operation of the inference unit 111a in the processing unit 111 will be described with reference to Fig. 5. Fig. 5 is a flowchart for explaining the flow of the operation of the inference unit 111a in the processing unit 111.

[0070] In step S501, the inference unit 111a reads the function list stored in the function list storage unit 112a and the prerequisites stored in the prerequisite storage unit 112d.

[0071] In step S502, the inference unit 111a determines whether there is an input from the voice recognition unit 115, that is, whether there is an instruction from the user via the microphone 13. If there is an input from the voice recognition unit 115, the flow proceeds to step S503. If there is no input from the voice recognition unit 115, the flow returns to step S502 and waits for an input from the voice recognition unit 115.

[0072] In step S503, the inference unit 111a overwrites the input from the speech recognition unit 115 in the input buffer 112b.

[0073] In step S504, the inference unit 111a transmits the function list and prerequisites read in step S501, as well as the data overwritten in the input buffer 112b and the history log accumulated in the history log storage unit 112c, to the natural language processing system 50. If no history log has been accumulated yet, a history log of empty data will be transmitted.

[0074] In step S505, the inference unit 111a adds to the history log the data in the input buffer 112b and the response from the natural language processing system 50. Note that the data added to the history log in step S505 is a set of data including an input to the natural language processing system 50 and an output corresponding to the input.

[0075] In step S506, the inference unit 111a branches the process depending on the response from the natural language processing system 50. If the response from the natural language processing system 50 is to have the speaker 14 output (speak in this case) a voice (i.e., execute the functional module Speaker in FIG. 2), the flow proceeds to step S507. If the response from the natural language processing system 50 is something else, for example, to have the mobile robot 12 perform an action (move or standstill in this case) (i.e., execute the functional module Move or Sleep in FIG. 2), the flow proceeds to step S509. If the response from the natural language processing system 50 is to have the search server 60 perform a search (i.e., execute the functional module Search in FIG. 2), the flow also proceeds to step S509. If the response from the natural language processing system 50 is "completed," the flow returns to step S502 and waits for the next input from the speech recognition unit 115.

[0076] In step S507, the inference unit 111a transmits an execution instruction for the functional module (here, the functional module Speaker) identified based on the response from the natural language processing system 50 to the speech synthesis unit 116. In this case, the speech synthesis unit 116 executes the functional module to generate an operation command. Note that in step 507, the processing unit 111 may transmit the operation command generated by executing the functional module to the speech synthesis unit 116.

[0077] In step S508, the inference unit 111a overwrites the input buffer 112b with the estimation result (here, "Finished to speak") of the function module that the inference unit 111a has instructed the speech synthesis unit 116 to execute. Then, the flow returns to step S504, and the inference unit 111a transmits the function list, the prerequisites, the data overwritten in the input buffer 112b, and the history log to the natural language processing system 50.

[0078] In step S509, the inference unit 111a transmits an instruction to execute the function module identified based on the response from the natural language processing system 50 to the queue transmission unit 111b.

[0079] In step S510, the inference unit 111a checks whether the inference result (here, "Finished to move" or "Finished to sleep") of the functional module that sent the execution instruction to the queue transmission unit 111b in step S509 exists in the function list storage unit 112a. If the inference result exists in the function list storage unit 112a, the flow proceeds to step S508, where the inference result of the functional module that sent the execution instruction to the queue transmission unit 111b is overwritten in the input buffer 112b. Thereafter, the flow returns to step S504, where the inference unit 111a sends the function list, prerequisites, the data overwritten in the input buffer 112b, and the history log to the natural language processing system 50. In this way, the inference unit 111a overwrites the inference result of the functional module in the input buffer 112b independently of the execution state of the functional module. If the estimation result of the functional module that sent the execution instruction to the queue sending unit 111b is not present in the function list storage unit 112a (i.e., if the estimation result of the functional module is "Unable to expect"), the flow proceeds to step S511. In step S511, the inference unit 111a waits for a response from the queue sending unit 111b. The queue sending unit 111b sends, as a response, the execution result obtained when the execution of the functional module that sent the execution instruction in step S509 is completed. The inference unit 111a then overwrites the execution result in the input buffer 112b. The flow then returns to step S504, where the inference unit 111a sends the function list, prerequisites, the data overwritten in the input buffer 112b, and the history log to the natural language processing system 50. As a result, steps S504 to S511 are looped until the natural language processing system 50 determines that the operation of the robot 15 has been completed due to the completion of the flow, making it possible for the robot 15 to perform continuous operations.

[0080] In addition, in Figure 5, an example has been described in which the inference unit 111a includes the data in the input buffer, the function list, the history log, and the prerequisites in the prompt and transmits it to the natural language processing system 50. However, the inference unit 111a may also include at least one of these in addition to the data in the input buffer 112b in the prompt and transmit it to the natural language processing system 50.

[0081] For example, if the prompt includes all of the input buffer data, function list, history log, and preconditions, the robot 15 can be made to perform flexible and appropriate operations by utilizing the preconditions that define the operating policy, the history log can be used to loop processing and make the robot 15 perform continuous operations that take past events into consideration, and the function list can be used to make the robot 15 perform intended operations. Furthermore, by utilizing the function list, the natural language processing system 50 can return an answer as to which function module should be executed. Since data identifying a predetermined function module is obtained as an answer, it can be accumulated as a history log in a format that can be used later. Therefore, the history log can be included in the prompt, and the next answer can be obtained appropriately.

[0082] Next, a specific operation of the queue transmitting unit 111b in the processing unit 111 will be described with reference to Fig. 6. Fig. 6 is a flowchart for explaining the flow of the operation of the queue transmitting unit 111b in the processing unit 111.

[0083] In step S601, the queue transmitting unit 111b checks the function module name of the function module for which an execution instruction was sent from the inference unit 111a in step S509, and checks whether the inference result of this function module exists in the function list storage unit 112a.

[0084] In step S602, the queue transmitting unit 111b selects a queue from the mobile robot queue 118 or the search queue 119 to which an instruction to execute the functional module is to be transmitted, according to the functional module name confirmed in step S601, and transmits the instruction to execute the functional module to the instruction queue 118a or 119a of the selected queue. Specifically, if the functional module is a functional module for operating the mobile robot 12 (e.g., Move or Sleep), the queue transmitting unit 111b selects the mobile robot queue 118 and transmits the instruction to execute the functional module to the instruction queue 118a of the mobile robot queue 118. On the other hand, if the functional module is a functional module for causing the search server 60 to perform a search (e.g., Search), the queue transmitting unit 111b selects the search queue 119 and transmits the instruction to execute the functional module to the instruction queue 119a of the search queue 119.

[0085] In step S603, the queue sending unit 111b acquires (or receives) the execution result of the functional module for which an execution instruction was sent from the inference unit 111a in step S509, from the reply queue 118b of the mobile robot queue 118 or the reply queue 119b of the search queue 119.

[0086] In step S604, if the estimated result of the functional module confirmed in step S601 exists in the function list storage unit 112a, the queue transmission unit 111b checks whether this estimated result matches the execution result of the functional module acquired in step S603. If the estimated result matches the execution result, the queue transmission unit 111b determines that the processing is normal. On the other hand, if the estimated result does not match the execution result, the queue transmission unit 111b determines that some abnormality has occurred in the processing and performs error processing.

[0087] In step S605, if the estimated result of the functional module confirmed in step S601 does not exist in the function list storage unit 112a, the queue transmission unit 111b transmits the execution result of the functional module acquired in step S603 to the inference unit 111a. The inference unit 111a overwrites the input buffer with the execution result of the functional module transmitted in step S605 in step S511 of Fig. 5. Thereafter, the flow returns to step S601.

[0088] Next, the operation of the functional module execution unit 161a in the processing device 16a will be described with reference to Fig. 7. Fig. 7 is a flowchart for explaining the flow of operation of the functional module execution unit 161a in the processing device 16a. The operation of the functional module execution unit 161b in the processing device 16b is similar to the operation of the functional module execution unit 161a in the processing device 16a, and therefore a description thereof will be omitted. That is, the functional module execution unit 161b in the processing device 16b operates in accordance with the flowchart in Fig. 7.

[0089] In step S701, it is confirmed whether an execution instruction for a functional module temporarily stored in the instruction queue 118a of the mobile robot queue 118 exists in the instruction queue 118a. If an execution instruction does not exist, the confirmation is repeated until an execution instruction is received. If an execution instruction exists, the flow proceeds to step S702.

[0090] In step S702, the functional module execution unit 161a acquires an execution instruction for a functional module temporarily stored in the instruction queue 118a of the mobile robot queue 118. The functional module execution unit 161b acquires an execution instruction for a functional module temporarily stored in the instruction queue 119a of the search queue 119.

[0091] In step S703, the functional module execution unit 161a executes the functional module for which the execution instruction was obtained in step S702, thereby generating an operation command for the mobile robot 12. The functional module execution unit 161b executes the functional module for which the execution instruction was obtained in step S702, thereby generating a search request, which is an example of an operation command for the search server 60.

[0092] In step S704, the functional module execution unit 161a transmits the operation command generated in step S703 to the mobile robot 12. The functional module execution unit 161b transmits the search request generated in step S703 to the search server 60.

[0093] In step S705, the functional module execution unit 161a transmits the execution result of the functional module executed in step S703 to the reply queue 118b of the mobile robot queue 118. The queue transmission unit 111b receives the execution result of the functional module transmitted to the reply queue 118b in step S705 from the reply queue 113b in step S603 of Fig. 6. Thereafter, the flow returns to step S701. Note that the functional module execution unit 161b transmits the search result, which is the execution result of the functional module executed in step S703, to the reply queue 119b of the search queue 119.

[0094] Next, a specific operation of the robot control unit 121 will be described with reference to Fig. 8. Fig. 8 is a flowchart showing the processing of the robot control unit 121.

[0095] In step S801, the robot control unit 121 determines whether there is an input from the functional module execution unit 161a. If there is an input from the functional module execution unit 161a, that is, an operation command for the mobile robot 12, the flow proceeds to step S802. If there is no input from the functional module execution unit 161a, the flow returns to step S801.

[0096] In step S802, the robot control unit 121 executes the operation instructed in the operation command.

[0097] In step S803, the robot control unit 121 notifies the functional module execution unit 161a of the end of operation. Upon receiving this notification of the end of operation, the functional module execution unit 161a transmits the execution result of the functional module to the queue 113 in step S705. Note that the functional module execution unit 161a may transmit the execution result of the functional module to the queue 113 at the time of completing the execution of the functional module, without waiting for this notification of the end of operation.

[0098] As described above, the control device 11 according to this embodiment executes a functional module identified by a response from the natural language processing system 50, thereby making the natural language processing system 50 infer the next response using the estimation result of the functional module, without actually operating the robot 15. As a result, if the estimation result of the functional module exists in the function list storage unit 112a, the functional module can be inferred on the assumption that the execution result of the functional module is normal, and the next operation of the robot 15 can be inferred without waiting for the completion of execution of the functional module. Therefore, task processing of the robot operation can be accelerated.

[0099] (Second embodiment) Next, a system 200 according to the second embodiment will be described. In the system 100 according to the first embodiment, the functional module execution units 161a and 161b that execute the functional modules are separated from the processing unit 111 that infers the functional modules, and the functional module execution units 161a and 161b and the processing unit 111 are configured to cooperate with each other using a queue 113, thereby enabling the inference processing of the functional modules and the execution processing of the functional modules to be processed in parallel, thereby speeding up task processing of robot operation.

[0100] On the other hand, in the system 200 according to the second embodiment, instead of separating the functional module execution unit that executes the functional module from the processing unit, the processing unit uses multithreading to perform functional module inference processing and functional module execution processing in parallel. That is, while the inference processing of the functional module is being performed in one thread, the execution processing of the functional module is being performed using another thread. As a result, the system 200 according to the second embodiment can also perform functional module inference processing and functional module execution processing in parallel, thereby speeding up task processing of robot operation.

[0101] FIG. 9 is a diagram showing an example of the configuration of a system 200 according to the second embodiment. The same components as those in the system 100 according to the first embodiment are given the same reference numerals, and their description may be omitted. The system 200 shown in FIG. 9 includes a robot system 20, a natural language processing system 50, and a search server 60. The robot system 20 and the natural language processing system 50 are connected to each other so as to be able to send and receive data via a network. The robot system 20 and the search server 60 are connected to each other so as to be able to send and receive data via the network.

[0102] The robot system 20 includes a control device 21, a left arm robot 22L, a right arm robot 22R, a microphone 13, and a speaker 14. In this specification, the left arm robot 22L, the right arm robot 22R, the microphone 13, and the speaker 14 are collectively referred to simply as a robot 25.

[0103] In this embodiment, an example of robot control will be described in which a user orders a drink from the robot 25 and the robot 25 brews the ordered drink. For example, when the user orders a drink, the robot 25 takes a cup, brews coffee, adds sugar and milk as needed, and hands the finished coffee to the user. When the user orders a drink from the robot 25, the user may ask for recommended drinks based on the weather, etc.

[0104] In the system 200 of this embodiment, the robot system 20 transmits a prompt to the natural language processing system 50 inquiring about the next action of the robot 25, and causes the robot 25 to operate based on the response from the natural language processing system 50.

[0105] The control device 21 is a control device for controlling the robot 25 based on interaction with a user, and includes a processing unit 211, a memory unit 212, a voice input unit 114, a voice recognition unit 115, a voice synthesis unit 116, and a voice output unit 117. These are communicatively connected to each other, for example, via a bus. In this embodiment, the processing unit 211, the voice recognition unit 115, and the voice synthesis unit 116 are each configured separately, but the voice recognition unit 115 and the voice synthesis unit 116 may be part of the functional configuration of the processing unit 211. The control device 21 also includes a communication unit (not shown). The communication unit is realized, for example, by a circuit connected to a network. The communication unit communicates with external devices such as the natural language processing system 50 and the search server 60 via the network.

[0106] The processing unit 211 is configured with, for example, a CPU (Central Processing Unit) and a RAM (Random Access Memory) used as a main storage device when the CPU executes processing. The CPU, for example, loads a program stored in the storage unit 212 into the RAM and executes the program, thereby realizing various functions corresponding to the program.

[0107] The processing unit 211, for example, sends a prompt to a natural language processing system 50 that uses a large-scale language model, receives a response from the natural language processing system 50, and causes the robot 15 to execute a functional module, described later, identified by the received response.

[0108] The storage unit 212 is a so-called auxiliary storage device that includes a nonvolatile storage circuit such as a hard disk drive (HDD) for storing various information and a solid state drive (SSD). Note that the storage unit 212 may also be a drive or the like that reads and writes various information from and to portable storage media such as a CD-ROM, a DVD, and a flash memory.

[0109] The storage unit 212, like the storage unit 112 according to the first embodiment, includes a function list storage unit 112a, an input buffer 112b, a history log storage unit 112c, and a prerequisite storage unit 112d. The function modules stored in the function list storage unit 112a and / or the prerequisites stored in the prerequisite storage unit 112d may be appropriately set to function modules and / or prerequisites suitable for robot control in this embodiment. In addition to these, the storage unit 212 according to this embodiment further includes a thread list storage unit 212e. The thread list storage unit 212e will be described later.

[0110] The left arm robot 22L includes a robot control unit 221L, a first axis L1, a second axis L2, a third axis L3, and a fourth axis L4.

[0111] The robot control unit 221L receives operation commands for the left arm robot 22L generated by the processing unit 211 executing the functional modules, and controls each part of the left arm robot 22L in accordance with the operation commands. Specifically, the robot control unit 221L is a computer equipped with hardware such as a calculation unit such as a CPU, a main storage unit such as a semiconductor memory, an auxiliary storage unit such as a hard disk, and a communication unit.

[0112] The first axis L1, the second axis L2, the third axis L3, and the fourth axis L4 are rotation axes for moving left arm joints (not shown) provided on the left arm robot 22L using motors (not shown). By independently controlling these multiple axes L1 to L4, the left arm robot 22L can be made to perform intended movements.

[0113] The right-arm robot 22R, like the left-arm robot 22L, includes a robot control unit 221R, a first axis R1, a second axis R2, a third axis R3, and a fourth axis R4.

[0114] The robot control unit 221R receives operation commands for the right arm robot 22R generated by the processing unit 211 executing the functional modules, and controls each part of the right arm robot 22R in accordance with the operation commands. Specifically, the robot control unit 221R is a computer equipped with hardware such as a calculation unit such as a CPU, a main storage unit such as a semiconductor memory, an auxiliary storage unit such as a hard disk, and a communication unit.

[0115] The first axis R1, the second axis R2, the third axis R3, and the fourth axis R4 are rotation axes for moving right arm joints (not shown) provided on the right arm robot 22R using motors (not shown). By independently controlling these multiple axes R1 to R4, the right arm robot 22R can be made to perform intended movements.

[0116] In this embodiment, the processing unit 211 controls the execution of functional modules by multiple threads assigned to each of the control objects including each part of the robot 25. For this reason, the thread list storage unit 212e stores a thread list that associates multiple threads assigned to each of the control objects including each part of the robot 25 with flags that indicate whether the operation of the control object is being executed. When the processing unit 211 causes each part of the robot 25 to execute a functional module, it checks whether the flag of the corresponding thread is set, and if the flag of the thread is not set, it executes the functional module and controls the flag of the thread to be set.

[0117] FIG. 10 is a diagram showing an example of a thread list. The thread list has data on "thread name" and "flag." The "thread name" is the name of a thread assigned to each of the control objects, including each part of the robot 25 (i.e., the left-arm robot 22L and the right-arm robot 22R) and the search server 60. The "flag" indicates whether the operation of the control object is currently being executed. As shown in FIG. 10, one thread for the left-arm robot 22L is assigned to the left-arm robot 22L, and one thread for the right-arm robot 22R is assigned to the right-arm robot 22R. Multiple threads (search 01, search 02) are assigned to the search server 60. By assigning multiple threads to the search server 60, multiple searches can be performed simultaneously. In the example of FIG. 10, the flag for the left-arm robot 22L is set high, indicating that the left-arm robot 22L is in operation.

[0118] Next, a specific operation of the processing unit 211 will be described with reference to Fig. 11. Fig. 11 is a flowchart for explaining the flow of the operation of the processing unit 211.

[0119] In step S1101, processing unit 211 reads the function list stored in function list storage unit 112a and the prerequisites stored in prerequisite storage unit 112d.

[0120] In step S1102, the processing unit 211 determines whether there is an input from the voice recognition unit 115, that is, whether there is an instruction from the user via the microphone 13. If there is an input from the voice recognition unit 115, the flow proceeds to step S1103. If there is no input from the voice recognition unit 115, the flow returns to step S1102 and waits for an input from the voice recognition unit 115.

[0121] In step S1103, the processing unit 211 overwrites the input from the voice recognition unit 115 in the input buffer 112b.

[0122] In step S1104, the processing unit 211 transmits the function list and prerequisites read in step S1101, as well as the data overwritten in the input buffer 112b and the history log accumulated in the history log storage unit 112c to the natural language processing system 50. If no history log has been accumulated yet, a history log of empty data will be transmitted.

[0123] In step S1105, the processing unit 211 adds to the history log the data in the input buffer 112b and the response from the natural language processing system 50. Note that the data added to the history log in step S1105 is a set of data including an input to the natural language processing system 50 and an output corresponding to the input.

[0124] In step S1106, the processing unit 211 branches the process depending on the response from the natural language processing system 50. If the response from the natural language processing system 50 is to have the speaker 14 output (speak, in this case) a voice (that is, to execute the functional module Speaker in FIG. 2), the flow proceeds to step S1107. If the response from the natural language processing system 50 is something else, for example, to have the left arm robot L22 perform an action (here, move or stand still) (that is, to execute the functional module Move or Sleep in FIG. 2), the flow proceeds to step S1109. If the response from the natural language processing system 50 is to have the search server 60 perform a search (that is, to execute the functional module Search in FIG. 2), the flow also proceeds to step S1109. If the response from the natural language processing system 50 is "completed," the process returns to step S1102 and waits for the next input from the speech recognition unit 115.

[0125] In step S1107, the processing unit 211 transmits an execution instruction for the functional module (here, the functional module Speaker) identified based on the response from the natural language processing system 50 to the speech synthesis unit 116. In this case, the speech synthesis unit 116 executes the functional module to generate an operation command. Note that in step 1107, the processing unit 211 may transmit the operation command generated by executing the functional module to the speech synthesis unit 116.

[0126] In step S1108, the processing unit 211 overwrites the input buffer 112b with the estimation result (here, "Finished to speak") of the function module that the processing unit 116 has been instructed to execute. The flow then returns to step S1104, and the processing unit 211 transmits the function list, the prerequisites, the data overwritten in the input buffer 112b, and the history log to the natural language processing system 50.

[0127] In step S1109, the processing unit 211 checks whether the flag of the thread for executing the functional module identified based on the response from the natural language processing system 50 is set. If the flag of the thread for executing the functional module identified based on the response from the natural language processing system 50 is set, the processing unit 211 waits until the flag of the thread is set. If the flag of the thread for executing the functional module identified based on the response from the natural language processing system 50 is not set, the flow proceeds to step S1110.

[0128] In step S1110, the processing unit 211 sets a flag for a thread for executing the function module identified based on the response from the natural language processing system 50. Then, the processing unit 211 starts execution of the function module using the thread for which the flag has been set.

[0129] In step S1111, the processing unit 211 checks whether the estimation results of the functional module identified based on the response from the natural language processing system 50 (here, "Finished to move" and "Finished to sleep") exist in the function list storage unit 112a. If the estimation results exist in the function list storage unit 112a, the flow proceeds to step S1108, where the processing unit 211 overwrites the input buffer 112b with the estimation results of the functional module. Thereafter, the flow returns to step S1104, where the processing unit 211 transmits the function list, the prerequisites, the data overwritten in the input buffer 112b, and the history log to the natural language processing system 50. In this way, the processing unit 211 overwrites the input buffer 112b with the estimation results of the functional module independently of the execution state of the functional module.

[0130] If the estimation result of the functional module identified based on the response from the natural language processing system 50 does not exist in the function list storage unit 112a (that is, if the estimation result of the functional module is "Unable to expect"), the flow proceeds to step S1112. In step S1112, the processing unit 211 waits for a response from the thread (referred to as the "functional module execution thread") that executes the functional module whose flag was raised in step S1109. The functional module execution thread sends back the execution result obtained when the execution of the functional module is completed as a response, and the processing unit 211 overwrites the execution result in the input buffer 112b. Then, the processing unit 211 lowers the flag of the functional module execution thread. Thereafter, the flow returns to step S1104, and the processing unit 211 transmits the function list, the prerequisites, the data overwritten in the input buffer 112b, and the history log to the natural language processing system 50. As a result, steps S1104 to S1112 are looped until the natural language processing system 50 determines that the operation of the robot 25 is completed due to completion of the flow, and the robot 25 can be made to perform successive operations.

[0131] In addition, in Figure 11, an example has been described in which the processing unit 211 includes the data in the input buffer, the function list, the history log, and the prerequisites in a prompt and sends it to the natural language processing system 50, but the processing unit 211 may also include at least one of these in addition to the data in the input buffer 112b in a prompt and send it to the natural language processing system 50.

[0132] For example, if the prompt includes all of the input buffer data, function list, history log, and prerequisites, the robot 25 can be made to perform flexible and appropriate operations by utilizing the prerequisites that define the operating policy, the history log can be used to loop processing and make the robot 25 perform continuous operations that take past events into consideration, and the function list can be used to make the robot 25 perform intended operations. Furthermore, by utilizing the function list, the natural language processing system 50 can return an answer as to which function module should be executed. Since data identifying a predetermined function module is obtained as an answer, it can be accumulated as a history log in a format that can be used later. Therefore, the history log can be included in the prompt, and the next answer can be obtained appropriately.

[0133] The operations of the robot control units 221L and 221R according to this embodiment are similar to the operations of the robot control unit 121 according to the first embodiment. That is, the operations of the robot control units 221L and 221R according to this embodiment follow the flowchart of FIG.

[0134] As described above, in the control device 21 according to this embodiment, while the processing unit 211 is performing inference processing of a functional module using multithreading, it is possible to perform execution processing of a functional module using another thread. As a result, the control device 21 according to the second embodiment can perform inference processing of a functional module and execution processing of the functional module in parallel, thereby speeding up task processing of robot operation.

[0135] (Other embodiments) The above describes embodiments of the present disclosure, but the present disclosure should not be construed as being limited to the above embodiments, and can be applied to various embodiments and combinations within the scope that does not deviate from the gist of the present disclosure.

[0136] Furthermore, the processing flow described in the above embodiment is also an example, and unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged within the scope of the present invention.

[0137] In each of the above embodiments, instructions are given to the robots 15, 25 via the microphone 13. However, instructions may be given to the robots 15, 25 not only by voice via the microphone 13, but also by text information via a keyboard or touch panel. Furthermore, text information may be provided to the user using a display instead of the speaker 14.

[0138] In the first embodiment, the control device 11 has two queues: the mobile robot queue 118 and the search queue 119. However, a single queue may be provided and functional module execution instructions may be stored in this single queue. In this case, the functional module execution units 161a and 161b may determine whether or not to obtain a functional module execution instruction from the queue depending on the functional module name.

[0139] In the first embodiment, in step S604 of FIG. 6, the queue sending unit 111b checks whether the estimation result of the functional module matches the execution result of the functional module. On the other hand, in the second embodiment, this process is not performed. However, the processing unit 211 may be made to perform the process of step S604. That is, if the estimation result of the functional module matches the execution result of the functional module, the processing unit 211 may determine that the process is normal. On the other hand, if the estimation result and the execution result do not match, the processing unit 211 may determine that some abnormality has occurred in the process and perform error processing.

[0140] It may be provided in the following manner. (Aspect 1) A control device (11, 21) for controlling a robot (15, 25) based on an interaction with a user, an input unit (114) that receives instructions from the user for the robot; an output unit (116) that outputs a response to the instruction to the user; a function list storage unit (112a) in which each of a plurality of predetermined actions to be executed by the robot is defined as a function module and a function list of the function modules is stored; an input buffer (112b) for storing the instruction data input from the input unit; a processing unit (111, 211) that transmits a prompt including the function list and data read from the input buffer to a natural language processing system (50) that uses a large-scale language model, receives a response from the natural language processing system, and causes the robot to execute a function module identified by the response; Equipped with the function list storage unit stores an execution result returned when the function module is normally executed as an estimated result in association with the function module; The control device wherein, when the estimation result of the functional module specified by the response exists in the function list storage unit, the processing unit overwrites the estimation result in the input buffer.

[0141] (Aspect 2) 2. The control device according to claim 1, wherein, when the processing unit overwrites the input buffer with the estimation result, the processing unit transmits the prompt including the data read from the overwritten input buffer to the natural language processing system.

[0142] (Aspect 3) 3. The control device according to aspect 1 or 2, wherein the processing unit overwrites the input buffer with the estimation result independently of an execution state of the functional module.

[0143] (Aspect 4) 4. The control device according to any one of aspects 1 to 3, wherein, if the estimation result is not present in the function list storage unit, the processing unit overwrites the input buffer with the execution result obtained when the execution of the functional module is completed.

[0144] (Aspect 5) The control device according to any one of aspects 1 to 4, further comprising a queue (118, 119) that temporarily stores an execution instruction for the function module identified by the response.

[0145] (Aspect 6) The processing unit includes an inference unit (111a) and a queue transmission unit (111b), the inference unit transmits an execution instruction for the function module identified by the response to the queue transmission unit; 6. The control device according to aspect 5, wherein the queue transmission unit transmits an execution instruction for the functional module to the queue.

[0146] (Aspect 7) the queue transmission unit acquires the execution result of the functional module from the queue; 7. The control device according to aspect 6, wherein, when the estimation result of the functional module exists in the function list storage unit, the queue sending unit checks whether the estimation result and the execution result match.

[0147] (Aspect 8) 8. The control device according to any one of aspects 1 to 7, further comprising a processing device (16a, 16b) including a function module execution unit (161a, 161b) that executes the function module.

[0148] (Aspect 9) the processing unit controls execution of the functional modules by a plurality of threads assigned to respective control targets including each part of the robot; 5. The control device according to any one of aspects 1 to 4, further comprising: a thread list storage unit that stores a thread list in which the plurality of threads are associated with a flag indicating whether the operation of the control target is currently being executed.

[0149] (Aspect 10) The control device described in aspect 9, wherein the processing unit checks whether the flag of the corresponding thread is raised when executing the functional module in each part of the robot, and if the flag of the thread is not raised, executes the functional module and raises the flag of the thread, and lowers the flag of the thread when execution of the functional module is completed.

[0150] (Aspect 11) further comprising a history log storage unit (112c) that accumulates a history log including data read from the input buffer and responses from the natural language processing system obtained in response to the read data; 11. The control device according to any one of aspects 1 to 10, wherein the processing unit includes the history log in the prompt to be sent to the natural language processing system.

[0151] (Aspect 12) The robot further includes a precondition storage unit (112d) that divides the operation to be performed by the robot into a plurality of stages and stores preconditions that define an operation policy for each stage, 12. The control device according to any one of aspects 1 to 11, wherein the processing unit includes the precondition in the prompt that is sent to the natural language processing system.

[0152] (Aspect 13) The control device according to any one of aspects 1 to 12, wherein the processing unit receives parameter data to be given to the functional module together with data identifying the functional module from the natural language processing system, and gives the parameter data to the identified functional module.

[0153] (Aspect 14) A control device as described in aspect 13, wherein the parameter data includes at least data regarding coordinates required for the robot's operation, data regarding the time required for the robot's operation, or data regarding text to be output to the user.

[0154] (Aspect 15) A control device as described in aspect 12, wherein the prerequisite conditions include information about the environment in which the robot operates.

[0155] (Aspect 16) 13. The control device according to aspect 12, wherein the operational policy is described in UML.

[0156] (Aspect 17) A control device (11) according to any one of aspects 1 to 7; processing devices (16a, 16b) including functional module execution units (161a, 161b) that execute the functional modules; The robot (15) is controlled by an operation command generated by the function module execution unit executing the function module.

[0157] (Aspect 18) A robot system (10) according to aspect 17; The natural language processing system (50); A system (100) comprising:

[0158] (Aspect 19) A control device (11, 21) according to any one of aspects 1 to 16; A robot system (10, 20) comprising the robot (15, 25).

[0159] (Aspect 20) A control device (11, 21) according to any one of aspects 1 to 16; The robot (15, 25); A system (100, 200) comprising the natural language processing system (50).

[0160] (Aspect 21) 1. A control method for controlling a robot based on an interaction with a user, comprising: an input step of receiving an instruction from the user to the robot; an output step of outputting a response to the instruction to the user; a function list reading step of reading a function list from a function list storage unit storing a function list of the predetermined plurality of operations to be executed by the robot, each of the predetermined plurality of operations being defined as a function module; a storing step of storing the instruction data inputted in the input step in an input buffer; a processing step of sending a prompt including the function list and the data read from the input buffer to a natural language processing system utilizing a large-scale language model, receiving a response from the natural language processing system, and causing the robot to execute a function module identified by the response; Equipped with the function list storage unit stores an execution result returned when the function module is normally executed as an estimated result in association with the function module; The processing step overwrites the input buffer with the estimation result if the estimation result for the functional module identified by the response exists in the function list storage unit.

[0161] (Aspect 22) A control program for controlling a robot based on an interaction with a user, On the computer, an input step of receiving an instruction from the user to the robot; an output step of outputting a response to the instruction to the user; a function list reading step of reading a function list from a function list storage unit storing a function list of the predetermined plurality of operations to be executed by the robot, each of the predetermined plurality of operations being defined as a function module; a storing step of storing the instruction data inputted in the input step in an input buffer; a processing step of sending a prompt including the function list and the data read from the input buffer to a natural language processing system utilizing a large-scale language model, receiving a response from the natural language processing system, and causing the robot to execute a function module identified by the response; Execute the function list storage unit stores an execution result returned when the function module is normally executed as an estimated result in association with the function module; The processing step is a control program for overwriting the input buffer with the estimation result when the estimation result of the functional module specified by the response exists in the function list storage unit. [Explanation of symbols]

[0162] 100, 200... systems, 10, 20... robot systems, 11, 21: Control device, 12: Mobile robot, 22L: Left arm robot, 22R: Right arm robot, 13: Microphone, 14: Speaker, 15, 25...Robots, 50...Natural language processing systems, 111, 211: Processing unit, 112: Storage unit, 112a: Function list storage unit, 112b: Input buffer, 112c: History log storage unit, 112d: a precondition storage unit; 114: a speech input unit; 116: a speech synthesis unit; 121, 221L, 221R...Robot control unit

Claims

1. A control device (11, 21) for controlling a robot (15, 25) based on an interaction with a user, comprising: an input unit (114) that receives instructions from the user for the robot; an output unit (116) that outputs a response to the instruction to the user; a function list storage unit (112a) in which each of a plurality of predetermined actions to be executed by the robot is defined as a function module and a function list of the function modules is stored; an input buffer (112b) for storing the instruction data input from the input unit; a processing unit (111, 211) that transmits a prompt including the function list and data read from the input buffer to a natural language processing system (50) that uses a large-scale language model, receives a response from the natural language processing system, and causes the robot to execute a function module identified by the response; Equipped with the function list storage unit stores an execution result returned when the function module is normally executed as an estimated result in association with the function module; The control device wherein, when the estimation result of the functional module specified by the response exists in the function list storage unit, the processing unit overwrites the estimation result in the input buffer.

2. The control device according to claim 1 , wherein, when the processing unit overwrites the input buffer with the estimation result, the processing unit transmits the prompt including the data read from the overwritten input buffer to the natural language processing system.

3. The control device according to claim 2 , wherein the processing unit overwrites the input buffer with the estimation result independently of an execution state of the functional module.

4. The control device according to claim 1 , wherein, when the estimation result is not present in the function list storage unit, the processing unit overwrites the input buffer with an execution result obtained when the execution of the function module is completed.

5. 2. The control device according to claim 1, further comprising a queue (118, 119) for temporarily storing an execution instruction for the function module specified by the response.

6. The processing unit includes an inference unit (111a) and a queue transmission unit (111b), the inference unit transmits an execution instruction for the function module identified by the response to the queue transmission unit; The control device according to claim 5 , wherein the queue transmission unit transmits an execution instruction for the functional module to the queue.

7. the queue transmission unit acquires the execution result of the functional module from the queue; The control device according to claim 6 , wherein, when the estimation result of the functional module exists in the function list storage unit, the queue transmission unit checks whether the estimation result and the execution result match.

8. The control device according to claim 7, further comprising a processing unit (16a, 16b) including a function module execution unit (161a, 161b) that executes the function module.

9. the processing unit controls execution of the functional modules by a plurality of threads assigned to respective control targets including each part of the robot; The control device according to claim 1 , further comprising a thread list storage unit that stores a thread list in which the plurality of threads are associated with a flag indicating whether the operation of the control target is currently being executed.

10. 10. The control device according to claim 9, wherein the processing unit checks whether a flag for a corresponding thread is raised when the processing unit causes each part of the robot to execute the functional module, and if the flag for the thread is not raised, executes the functional module and raises the flag for the thread, and lowers the flag for the thread for which execution of the functional module has been completed.

11. a history log storage unit (112c) that accumulates a history log including data read from the input buffer and responses from the natural language processing system obtained in response to the read data; The control device according to claim 1 , wherein the processing unit includes the history log in the prompt sent to the natural language processing system.

12. The robot further includes a precondition storage unit (112d) that divides the operation to be performed by the robot into a plurality of stages and stores preconditions that define an operation policy for each stage, The control device according to claim 1 , wherein the processing unit includes the precondition in the prompt sent to the natural language processing system.

13. 2. The control device according to claim 1, wherein the processing unit receives parameter data to be given to the functional module together with data identifying the functional module from the natural language processing system, and gives the parameter data to the identified functional module.

14. The control device according to claim 13 , wherein the parameter data includes at least data on coordinates required for the robot's operation, data on time required for the robot's operation, or data on a message to be output to a user.

15. The control device according to claim 12 , wherein the preconditions include information about an environment in which the robot operates.

16. The control device according to claim 12 , wherein the operation policy is described in UML.

17. A control device (11) according to claim 7; a processing device (16a, 16b) including a function module execution unit (161a, 161b) that executes the function module; and the robot (15) controlled by an operation command generated by the function module execution unit executing the function module.

18. A robotic system (10) according to claim 17, The natural language processing system (50); A system (100) comprising:

19. A control device (11, 21) according to any one of claims 1 to 16, A robot system (10, 20) comprising the robot (15, 25).

20. A control device (11, 21) according to any one of claims 1 to 16, The robot (15, 25); A system (100, 200) comprising the natural language processing system (50).

21. 1. A control method for controlling a robot based on an interaction with a user, comprising: an input step of receiving an instruction from the user to the robot; an output step of outputting a response to the instruction to the user; a function list reading step of reading a function list from a function list storage unit storing a function list of the predetermined plurality of operations to be executed by the robot, each of the predetermined plurality of operations being defined as a function module; a storing step of storing the instruction data inputted in the input step in an input buffer; a processing step of sending a prompt including the function list and the data read from the input buffer to a natural language processing system utilizing a large-scale language model, receiving a response from the natural language processing system, and causing the robot to execute a function module identified by the response; Equipped with the function list storage unit stores an execution result returned when the function module is normally executed as an estimated result in association with the function module; The processing step overwrites the input buffer with the estimation result if the estimation result for the functional module identified by the response exists in the function list storage unit.

22. A control program for controlling a robot based on an interaction with a user, On the computer, an input step of receiving an instruction from the user to the robot; an output step of outputting a response to the instruction to the user; a function list reading step of reading a function list from a function list storage unit storing a function list of the predetermined plurality of operations to be executed by the robot, each of the predetermined plurality of operations being defined as a function module; a storing step of storing the instruction data inputted in the input step in an input buffer; a processing step of sending a prompt including the function list and the data read from the input buffer to a natural language processing system utilizing a large-scale language model, receiving a response from the natural language processing system, and causing the robot to execute a function module identified by the response; Execute the function list storage unit stores an execution result returned when the function module is normally executed as an estimated result in association with the function module; The processing step is a control program for overwriting the input buffer with the estimation result when the estimation result of the functional module specified by the response exists in the function list storage unit.

Images