Control device, control method, and control program

Abstract

To avoid execution of a robot operation that cannot be executed when a situation of the real world is taken into consideration, and to prevent the whole system from becoming inoperable.SOLUTION: A control device (11) for controlling a robot (10) on the basis of an interaction with a user includes: an input section (114) that receives an instruction from the user to the robot; an output section (117) that outputs a response to the instruction of the user; a function list storage section (112a) that stores a function list of functional modules defining predetermined operations to be executed by the robot; an input buffer (112b); and a processing section (111) that transmits, to a natural language processing system, a prompt including the function list and data from the input buffer, and executes the functional module identified by an answer. The processing section stores, in the input buffer, information about the current state of a target operated by the robot when the functional module cannot be executed normally.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to a technology for controlling a robot based on interaction with a user. [Background technology]

[0002] In recent years, robots that can move flexibly based on user interaction via instructions from the user in natural language have been attracting attention.

[0003] Patent Document 1 describes a robot control device that allows a robot to efficiently perform a target action simply by issuing a command for the target action in a form similar to natural language. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Publication No. 08-286708 Summary of the Invention [Problem to be solved by the invention]

[0005] However, in the robot control device described in Patent Document 1, even if it is impossible to execute a target action when the situation in the real world is taken into consideration, the robot may execute the action, causing the entire system to become inoperable.

[0006] In view of the above problems, the present invention aims to avoid the execution of robot actions that are impossible to execute when the real-world situation is taken into account, thereby preventing the entire system from becoming inoperable. [Means for solving the problem]

[0007] The present invention employs the following technical solutions to solve the above problems. The reference symbols in parentheses in the claims and this section are merely examples showing the correspondence with the specific solutions described in the embodiments below as one aspect, and do not limit the technical scope of the present invention.

[0008] A control device (11) according to one aspect of the present invention is a control device for controlling a robot (10) based on interaction with a user, and includes an input unit (114) that receives instructions from the user for the robot, an output unit (117) that outputs a response to the instructions to the user, a function list storage unit (112a) in which each of a plurality of predetermined actions to be performed by the robot is defined as a functional module and a function list of the functional modules is stored, an input buffer (112b) that saves data of the instructions input from the input unit, and a processing unit (111, 211) that sends a prompt including the function list and data read from the input buffer to a natural language processing system using a large-scale language model, receives a response from the natural language processing system as to which functional module in the function list should be executed, and executes the functional module identified by the response, and if the processing unit is unable to execute the functional module normally, saves information about the current state of an object operated by the robot in the input buffer.

[0009] With the above configuration, if the processing unit cannot normally execute the functional module identified by the response from the natural language processing system, the processing unit can include the current state of the target operated by the robot in a prompt and send it to the natural language processing system, causing the natural language processing system to reconsider the robot's operation. By causing the natural language processing system to reconsider, the natural language processing system can generate a more accurate response. Although the response from the natural language processing system may be ambiguous, the configuration of the present invention can prevent the robot from executing an erroneous operation even if the response is impossible to execute in light of the actual situation, thereby preventing the entire system from becoming inoperable.

[0010] A control method as one aspect of the present invention is a control method for controlling a robot based on interaction with a user, wherein each of a plurality of predetermined operations to be performed by the robot is defined as a functional module, and the control method includes: a reading step of reading the function list from a function list storage unit that stores a function list of the functional modules; an input step of accepting instructions from the user for the robot; a saving step of saving data of the instructions input by the input step in an input buffer; a processing step of sending a prompt including the function list and the data read from the input buffer to a natural language processing system that uses a large-scale language model, receiving from the natural language processing system a response as to which functional module in the function list should be executed, and executing the functional module identified by the response; and an output step of outputting a response to the instruction to the user, wherein if the functional module cannot be executed normally in the processing step, information regarding the current state of an object operated by the robot is saved in the input buffer.

[0011] A control program as one aspect of the present invention is a control program for controlling a robot based on interaction with a user, the control program causing a computer to execute the following steps: a reading step of reading a function list from a function list storage unit that stores a function list of the functional modules, each of which is defined as a functional module for a plurality of predetermined operations to be performed by the robot; an input step of accepting instructions from the user for the robot; a saving step of saving data of the instructions input by the input step in an input buffer; a processing step of sending a prompt including the function list and the data read from the input buffer to a natural language processing system that uses a large-scale language model, receiving from the natural language processing system an answer as to which functional module in the function list should be executed, and executing the functional module identified by the answer; and an output step of outputting a response to the instruction to the user; and if the functional module cannot be executed normally in the processing step, information regarding the current state of an object operated by the robot is saved in the input buffer. [Effects of the Invention]

[0012] According to the present invention, it is possible to provide a control device etc. that can avoid the execution of robot operations that are impossible to execute when taking into account real-world conditions, and can prevent the entire system from becoming inoperable. [Brief explanation of the drawings]

[0013] [Figure 1] FIG. 1 illustrates an example of a configuration of a robot system according to a first embodiment. [Figure 2] FIG. 10 is a diagram showing an example of a status list. [Figure 3A] FIG. 10 is a diagram showing an example of a function list. [Figure 3B] FIG. 10 is a diagram showing an example of a function list. [Figure 4A] FIG. 10 is a diagram illustrating an example of a precondition. [Figure 4B] FIG. 10 is a diagram illustrating an example of a precondition. [Figure 5] FIG. 10 is a diagram showing an example of a history log when a robot executes a series of operations as expected. [Figure 6] FIG. 10 is a diagram showing an example of a history log in a case where a robot does not execute a series of operations as expected. [Figure 7] 4 is a flowchart illustrating a processing flow of a processing unit according to the first embodiment. [Figure 8A] 10 is a flowchart illustrating details of processing by a processing unit in step S706 when a functional module Pick_cup_from is executed. [Figure 8B] 10 is a flowchart illustrating details of processing by a processing unit in step S706 when a functional module Place_cup_to is executed. [Figure 8C] 10 is a flowchart illustrating details of processing by a processing unit in step S706 when a functional module Pour_hot_water is executed. [Figure 8D] 10 is a flowchart illustrating details of processing by a processing unit in step S706 when a functional module Pour_cold_water is executed. [Figure 8E] 10 is a flowchart for explaining details of the processing of the processing unit in step S706 when the functional module Pour_alcohol is executed. [Figure 8F] 10 is a flowchart illustrating details of the processing of the processing unit in step S706 when the functional module Pour_ice is executed. [Figure 8G] 10 is a flowchart for explaining details of the processing of the processing unit in step S706 when the functional module Stir is executed. [Figure 8H] 10 is a flowchart illustrating details of processing by the processing unit in step S706 when the function module Get_cup_location is executed. [Figure 8I]10 is a flowchart illustrating details of the processing of the processing unit in step S706 when the functional module Move_to is executed. [Figure 8J] 10 is a flowchart illustrating details of the processing performed by the processing unit in step S706 when the function module Search_for is executed. [Figure 8K] 10 is a flowchart illustrating details of the processing of the processing unit in step S706 when the functional module Search_weather is executed. [Figure 8L] 10 is a flowchart for explaining details of the processing of the processing unit in step S706 when the functional module Speak is executed. [Figure 9] 10 is a flowchart showing the processing of a robot control unit. [Figure 10] 10 is a flowchart illustrating the flow of processing by a processing unit according to the second embodiment. [Figure 11] FIG. 10 illustrates an example of a configuration of a robot system according to a third embodiment. [Figure 12] 13 is a flowchart illustrating details of processing by a processing unit in step S706 when a functional module Pick_cup_from is executed, as an example, according to the third embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0014] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. Note that the embodiment described below shows an example of how the present invention can be implemented, and the present invention is not limited to the specific configuration described below. When implementing the present invention, a specific configuration corresponding to the embodiment may be appropriately adopted.

[0015] (First embodiment) Fig. 1 is a diagram showing an example of the configuration of a robot system 100 according to a first embodiment. The robot system 100 shown in Fig. 1 includes a robot 10, a natural language processing system 50, and a search server 60. The robot 10 and the natural language processing system 50 are connected to each other so as to be able to send and receive data via a network. The robot 10 and the search server 60 are connected to each other so as to be able to send and receive data via the network.

[0016] The robot 10 includes a control device 11, an arm robot 12, a microphone 13, and a speaker 14.

[0017] In this embodiment, an example of robot control will be described in which a user orders a drink from the robot 10 and the robot 10 prepares the ordered drink. When the user orders a drink from the robot 10, the user may ask the robot 10 for recommended drinks depending on the weather, etc.

[0018] The natural language processing system 50 is a natural language processing system that uses a large-scale language model (LLM). A large-scale language model is a deep learning model that generates a language model, which models natural language spoken by humans based on its occurrence probability, through pre-training based on a huge amount of data. As the natural language processing system 50 that uses a large-scale language model, for example, GPT-3, GPT-3.5, or GPT-4 can be used. When the natural language processing system 50 receives a request, it uses the large-scale language model to send to the request source a sentence that combines one or more probabilistically estimated words that follow the sentence included in the received request.

[0019] The search server 60 performs a search process based on a search request (an example of an operation command) sent from the robot 10, and provides information corresponding to the search request by sending search results (an example of an execution result) indicating the results of the search process to the robot 10.

[0020] In the robot system 100 of this embodiment, the robot 10 transmits a prompt to the natural language processing system 50 to inquire about the operation of the robot 10, and the robot 10 is operated based on the response from the natural language processing system 50.

[0021] The control device 11 is a control device for controlling the robot 10 based on interaction with a user, and includes a processing unit 111, a memory unit 112, a voice input unit 114, a voice recognition unit 115, a voice synthesis unit 116, and a voice output unit 117. These are communicatively connected to each other, for example, via a bus. In this embodiment, the processing unit 111, the voice recognition unit 115, and the voice synthesis unit 116 are each configured separately, but the voice recognition unit 115 and the voice synthesis unit 116 may be part of the functional configuration of the processing unit 111. The control device 11 also includes a communication unit (not shown). The communication unit is realized, for example, by a circuit connected to a network. The communication unit communicates with external devices such as the natural language processing system 50, the search server 60, and the arm robot 12 via the network.

[0022] The processing unit 111 is configured with, for example, a CPU (Central Processing Unit) and a RAM (Random Access Memory) that is used as a main storage device when the CPU executes processing. The CPU, for example, loads a program stored in the storage unit 112 into the RAM and executes the program, thereby realizing various functions corresponding to the program.

[0023] The processing unit 111 transmits a prompt to the natural language processing system 50 that uses a large-scale language model, receives a response from the natural language processing system 50, and executes a function module (described later) identified based on the received response. The processing unit 111 includes a state management unit 111a as a functional configuration.

[0024] The state management unit 111a manages the current state of an object operated by the robot 10. In this embodiment, the object operated by the robot 10 is, for example, a "cup" for holding a drink. That is, in this embodiment, the state management unit 111a manages the current state of the cup, specifically, information about where the cup is currently located.

[0025] FIG. 2 is a state list that defines multiple states that an object manipulated by the robot 10 can take. In this embodiment, the object manipulated by the robot 10 is a "cup," and so FIG. 2 shows a state list for the "location of the cup." The state list shown in FIG. 2 includes data for "variable," "parameter," and "Location_ID" in addition to the "location of the cup." The "variable" indicates the "location of the cup" and is assigned the value of "Location_ID." The "parameter" indicates a value that the "location of the cup" can take. Specifically, the parameter Table indicates that the cup is on a table, the parameter Pot indicates that the cup is in a hot water pot, and the parameter Hand indicates that the cup is in the hand of the arm robot 12 (i.e., the arm robot 12 is holding the cup in its hand). The "Location_ID" is an ID assigned to each of the "parameters." In this embodiment, Location_ID=0 is assigned to the parameter Table, Location_ID=1 is assigned to the parameter Pot, and Location_ID=2 is assigned to the parameter Hand.

[0026] The state management unit 111a manages the current state of the object operated by the robot 10 (i.e., the current location of the cup) by substituting Location_ID for the variable Cup_location. For example, if the current location of the cup is on a table, the state management unit 111a manages the current state of the cup by holding data Cup_location=0. Note that the state management unit 111a may manage the current state of the object operated by the robot 10 by substituting a parameter value for the variable Cup_location (e.g., Cup_location=Table) without using Location_ID.

[0027] If the function module described below can be executed normally, the state management unit 111a updates the current state of the object operated by the robot 10 (that is, the current location of the cup) based on the execution result of the function module.

[0028] The storage unit 112 is a so-called auxiliary storage device that includes a nonvolatile storage circuit such as an HDD (hard disk drive) and an SSD (solid state drive) that store various information. Note that the storage unit 112 may also be a drive or the like that reads and writes various information from and to portable storage media such as CD-RW, DVD-RAM, and flash memory.

[0029] The voice input unit 114 is connected by wire or wirelessly to the microphone 13 that collects voice and outputs a voice signal, and receives the voice signal from the microphone 13. In other words, the voice input unit 114 receives instructions from the user to the robot 10.

[0030] The speech recognition unit 115 performs speech recognition processing on the speech signal from the speech input unit 114 and outputs text information represented by the speech to the processing unit 111. For example, the speech recognition unit 115 can use a speech recognition technology that converts speech signals into text information using deep learning technology.

[0031] The voice synthesis unit 116 synthesizes voice from the operation command for the robot 10 generated by the execution of the function module identified by the processing unit 111, and outputs the voice to the voice output unit 117. The voice synthesis unit 116 can use a general voice synthesis technology.

[0032] The audio output unit 117 is connected by wire or wirelessly to the speaker 14, which converts an audio signal into audio and outputs it to the outside, and outputs the audio signal from the audio synthesis unit 116 to the speaker 14. In other words, the audio output unit 117 outputs a response to an instruction from the user to the user.

[0033] The arm robot 12 includes a robot control unit 121 and a plurality of axes including a first axis L1, a second axis L2, a third axis L3, and a fourth axis L4.

[0034] The robot control unit 121 receives operation commands for the arm robot 12 generated by the processing unit 111 executing the functional modules, and controls each part of the arm robot 12 according to the operation commands. Specifically, the robot control unit 121 is a computer equipped with hardware such as a calculation unit such as a CPU, a main storage unit such as a semiconductor memory, an auxiliary storage unit such as a hard disk, and a communication unit.

[0035] The multiple axes including the first axis L1, the second axis L2, the third axis L3, and the fourth axis L4 are rotation axes for moving arm joints (not shown) provided on the arm robot 12 using motors (not shown). By independently controlling these multiple axes, the arm robot 12 can be made to perform intended movements.

[0036] The following describes the storage areas provided in storage unit 112: function list storage unit 112a, input buffer 112b, history log storage unit 112c, and prerequisite storage unit 112d.

[0037] (Feature List) The storage unit 112 includes a function list storage unit 112a that stores a function list of multiple function modules. Here, a functional module is a package of a series of general-purpose operations of the robot 10. The function list is a collection of usage methods for each functional module, which is provided to the natural language processing system 50 as prior information when the natural language processing system 50 generates functional modules and parameters suitable for task processing. In other words, the function list is a list of multiple functional modules, each of which defines a plurality of predetermined operations to be performed by the robot 10 as a functional module.

[0038] Currently, it is difficult to generate a correct program (e.g., low code) for the robot 10 from scratch by sending a request in natural language to the natural language processing system 50. For this reason, simply sending a request in natural language to the natural language processing system 50 will not allow the robot 10 to perform the intended action.

[0039] Therefore, in this embodiment, processing unit 111 transmits to natural language processing system 50 a prompt containing a function list stored in function list storage unit 112a in addition to a natural language instruction from the user.

[0040] This allows the natural language processing system 50 to determine (or identify) from the function list the functional module to be executed by the robot 10 based on the user's instruction. As described above, the functional module defines a predetermined operation to be executed by the robot 10, so the processing unit 111 can identify the functional module based on the response from the natural language processing system 50 regarding which functional module in the function list should be executed, and can control the robot 10 by executing the identified functional module. Therefore, the robot 10 can appropriately perform the intended operation.

[0041] Furthermore, the functional modules receive as input parameters related to coordinates and / or time required for the operation of the robot 10, or parameters related to the text to be output to the user. In this case, the processing unit 111 transmits the parameters to be used by each functional module, including them in a function list, to the natural language processing system 50. The processing unit 111 then receives, from the natural language processing system 50, data identifying a functional module as well as parameter data to be given to the functional module, and gives the identified functional module the parameter data to be given to the functional module. Even in this case, the natural language processing system 50 transmits, as a response, the data identifying a functional module and the parameter data to be given to the functional module, so that the operation of the robot 10 is limited to the operation defined in the functional module, and the robot 10 can be operated more appropriately using parameter data that matches the function module.

[0042] 3A and 3B show an example of a function list stored in the function list storage unit 112a. The function list includes data on a "function module name," "function," "parameter," "return value," and "required condition." A "function module name" is the name of a function module. A "function" is a description of the function of the function module. By including data on the description of the function module in the function list, the natural language processing system 50 can learn how to use the function module and determine an appropriate function module for an instruction. A "parameter" is a parameter given to the function module when executing the function module. A "return value" is a value returned as an execution result when the function module is executed. In this embodiment, if the function module is executed normally, the function module returns a normal execution result, and if the function module is not executed normally, the function module returns an abnormal execution result. A "necessary condition" is a condition for the normal execution of the function module. In other words, if the function module is executed while satisfying all necessary conditions, the function module returns a normal execution result, and if the function module is executed without satisfying even one necessary condition, the function module returns an abnormal execution result. In this embodiment, function list storage unit 112a stores necessary conditions for the normal execution of a function module in association with the function module.

[0043] The function module Pick_cup_from is a function module for picking a cup from a specified location, which is a parameter. In this embodiment, two locations, a table and a pot, are specified as locations from which a cup can be picked. The necessary conditions for the function module Pick_cup_from to be executed successfully are that "the cup is not being held in the hand" and that "the cup is placed in the specified location." This is because if the cup is already being held in the hand, it cannot be picked up, and if the cup is not placed in the specified location, it cannot be picked up from the specified location. If the function module Pick_cup_from is executed successfully, the return value is a success message saying "The cup has been picked up." If the function module Pick_cup_from is executed unsuccessfully, the return value is "There is no cup at the specified location. It is currently in {Cup_location}." The current state of the cup managed by the state management unit 111a is substituted for {Cup_location} in the return value.

[0044] The Place_cup_to function module is a function module for placing a cup at a specified location, which is a parameter. In this embodiment, two locations are defined for placing a cup: a table and a pot. The necessary conditions for the Place_cup_to function module to be executed successfully are that the cup is being held in the hand and that the cup is not being placed in the specified location. This is because the cup cannot be placed if the cup is not being held in the hand, and the cup cannot be placed in the specified location if the cup is already placed in the specified location. If the Place_cup_to function module is executed successfully, the return value is a success message saying "The cup has been placed." If the Place_cup_to function module is executed unsuccessfully, the return value is "You do not have the cup. It is currently located in {Cup_location}." This return value is returned when the necessary condition "You are holding the cup" is not met. The current state of the cup managed by the state management unit 111a is substituted for {Cup_location} in the return value. If the necessary condition "The cup is not being placed in the specified location" is not met, an error message indicating that the specified location is invalid may be returned.

[0045] The functional module Pour_hot_water is a functional module for pouring a specified amount of hot water into a cup, which is a parameter. In this embodiment, three amounts are defined as the amount to be poured into a cup: less, normal, and more. A necessary condition for the functional module Pour_hot_water to be executed successfully is that a cup is placed in the pot. This is because if the cup is not placed in the pot, hot water will spill when poured from the pot. If the functional module Pour_hot_water is executed successfully, the return value is a success message saying, "The specified amount has been poured." If the functional module Pour_hot_water is not executed successfully, the return value is, "The cup is not placed in the pot. It is currently located in {Cup_location}." The current state of the cup managed by the state management unit 111a is substituted for {Cup_location} in the return value.

[0046] The functional module Pour_cold_water is a functional module for pouring a specified amount of water, which is a parameter, into a cup. In this embodiment, three amounts are defined as the amount to be poured into a cup: less, normal, and more. A necessary condition for the functional module Pour_cold_water to be executed successfully is that the user is holding a cup. This is because water cannot be poured from the water server unless the user is holding a cup. If the functional module Pour_cold_water is executed successfully, the return value is a success message saying, "Specified amount poured." If the functional module Pour_cold_water is not executed successfully, the return value is, "Cup is not being held. It is currently located at {Cup_location}." The current state of the cup managed by the state management unit 111a is substituted for {Cup_location} in the return value.

[0047] The functional module Pour_alcohol is a functional module for pouring a specified amount of alcohol, which is a parameter, into a cup. In this embodiment, three amounts are defined for pouring into a cup: less, normal, and more. A necessary condition for the functional module Pour_alcohol to be executed successfully is that the cup is being held in the hand. This is because alcohol cannot be poured from the alcohol server unless the cup is being held in the hand. If the functional module Pour_alcohol is executed successfully, the return value is a success message saying, "The specified amount has been poured." If the functional module Pour_alcohol is not executed successfully, the return value is, "The cup is not being held in the hand. It is currently located at {Cup_location}." The current state of the cup managed by the state management unit 111a is substituted for {Cup_location} in the return value.

[0048] The functional module Pour_ice is a functional module for putting a specified amount of ice, which is a parameter, into a cup. In this embodiment, three amounts are defined for the amount to be put into the cup: less, normal, and more. A necessary condition for the functional module Pour_ice to execute successfully is that the user is holding a cup. This is because ice cannot be poured from the ice server unless the cup is held in the user's hand. If the functional module Pour_ice executes successfully, the return value is a success message saying, "The specified amount has been poured." If the functional module Pour_ice executes unsuccessfully, the return value is, "You are not holding a cup. It is currently located at {Cup_location}." The current state of the cup managed by the state management unit 111a is substituted for {Cup_location} in the return value.

[0049] The Stir function module is a function module for stirring a liquid in a cup for a specified time, which is a parameter. In this embodiment, a necessary condition for the Stir function module to be executed successfully is that the cup is held in the hand. This is because the liquid in the cup cannot be stirred unless the cup is held in the hand. If the Stir function module is executed successfully, the return value is a success message saying, "Stired for the specified number of seconds." If the Stir function module is not executed successfully, the return value is, "The cup is not held in the hand. It is currently located at {Cup_location}." The current state of the cup managed by the state management unit 111a is substituted for {Cup_location} in the return value.

[0050] The function module Get_cup_location is a function module for checking the current location of a cup. Because the function module Get_cup_location is for checking the current location of a cup, there are no prerequisites for this function module to be executed correctly. When the function module Get_cup_location is executed, the return value is "{Cup_location}". The current state of the cup managed by the state management unit 111a is substituted into {Cup_location}.

[0051] The function module Move_to is a function module for moving to a specified position, which is a parameter. In this embodiment, three positions are defined: initial (initial position), water_supply_button_pre (position of the water supply button), and ice_supply_button_pre (position of the ice supply button). A necessary condition for the function module Move_to to be executed successfully is that "a cup is not being held" if the specified position is "initial." If the specified position is "water_supply_button_pre" or "ice_supply_button_pre," no necessary condition for the function module to be executed successfully is true. If the function module Move_to is executed successfully, the return value is a success message saying "Moved to the specified position." If the function module Move_to is not executed successfully, the return value is "You cannot move while holding a cup." Note that this return value is returned when the specified location is "initial" and the necessary condition of "not holding a cup" is not met.

[0052] The function module Search_for is a function module for searching the Internet using a specified word, which is a parameter. The function module Search_for has no prerequisites for successful execution. When the function module Search_for is executed, the return value is "{search result}".

[0053] The function module Search_weather is a function module for checking the weather at a specified location and date, which are parameters. The function module Search_weather has no prerequisites for running successfully. When the function module Search_weather is run, the return value is "The weather at {location} on {date} is {weather information}, and the temperature is {temperature information}."

[0054] The function module Speak is a function module for speaking the contents of the specified text, which is a parameter. The function module Speak has no prerequisites for this function module to be executed correctly. When the function module Speak is executed, the return value is "Speech completed."

[0055] In the function lists shown in Figures 3A and 3B, the function name keys and JSON values ​​are written in a simple format because they are complicated, but the function modules can be implemented in JSON format.

[0056] (Prerequisite) The storage unit 112 further includes a prerequisite storage unit 112d that stores prerequisites for the operation of the robot 10. The prerequisites include the work procedures of the robot 10. Specifically, the work procedures of the robot 10 define the operation policies for each stage by dividing the operation to be performed by the robot 10 into multiple stages. These operation policies may be written in a natural language such as Japanese or English, or may be written in Unified Modeling Language (UML).

[0057] When the work procedures of the robot 10 are given to the natural language processing system 50 as preconditions, if the preconditions are written in source code, the natural language processing system 50 will follow the source code, resulting in a highly precise answer but not a flexible answer.

[0058] Therefore, in this embodiment, the processing unit 111 transmits to the natural language processing system 50 a prompt containing preconditions stored in the precondition storage unit 112d in addition to instructions in natural language from the user.

[0059] The preconditions define the behavior of the robot 10 in multiple stages and define the behavioral policy for each stage, so the natural language processing system 50 can provide flexible yet appropriate answers in accordance with the behavioral policies. Therefore, based on these answers, the processing unit 111 can make the robot 10 perform appropriate behavior in accordance with the behavioral policy while ensuring flexibility in the behavior of the robot 10.

[0060] 4A and 4B are diagrams showing examples of preconditions stored in the precondition storage unit 112d. Area 4a defines "Your Role" as a precondition. For example, area 4a contains statements such as "You are an excellent AI robot bartender," and "You speak in a friendly, feminine manner and use honorific language." This allows the natural language processing system 51 to generate responses from a bartender who is excellent, friendly, and speaks in a feminine manner. Area 4a also contains statements such as to primarily serve tea-based drinks and to provide freshly brewed tea to those who cannot drink alcohol. This defines how to respond to users, i.e., is an example of an operational policy for the robot 10.

[0061] In area 4b, a "workspace" is defined as a prerequisite. This refers to the workspace in which the robot 10 operates, and various information about the object manipulated by the arm robot 12 is described. In addition, in area 4d, an initial state ("The initial cup is located on the table") for the cup (i.e., the object manipulated by the robot 10) is described as a note. That is, by default, the state management unit 111a stores data of Cup_location=0. By defining the initial state, the natural language processing system 50 can generate a response appropriate for the initial state, assuming that the cup is initially located on the table. In addition, area 4b also states, "If you lose track of the cup, please try to find it." This description enables the natural language processing system 50 to return a response that identifies the function module Get_cup_location depending on the situation. Although not described in FIGS. 4A and 4B, for example, if the prerequisites contained a description such as "Always keep track of the location of the cup," the natural language processing system 50 might always identify the function module Get_cup_location before identifying the function module related to the robot operation.

[0062] In area 4c, a "recipe" is defined as a prerequisite. The natural language processing system 50 can identify functional modules for making tea-based drinks, hot green tea (non-alcoholic), or iced green tea (non-alcoholic) according to this recipe. This "recipe" is an example of a recipe that divides the actions to be performed by the robot 10 into multiple steps and defines the action policy for each step.

[0063] (History log) The storage unit 112 further includes an input buffer 112b for storing input data, and a history log storage unit 112c for accumulating a history log.

[0064] The processing unit 111 first stores in the input buffer 112b the data of the instruction from the user input via the voice input unit 114. Next, the processing unit 111 stores in the input buffer 112b the execution result of the function module identified by the response from the natural language processing system 50.

[0065] The history log is log data that includes data that has been read from the input buffer 112b and transmitted to the natural language processing system 50, and responses that have been obtained from the natural language processing system 50 in response to the transmitted data. In this way, a set consisting of an input to the natural language processing system 50 and an output in response to the input is added as a history log to the history log storage unit 112c.

[0066] The processing unit 111, which receives a response to one query from the natural language processing system 50, causes the robot 10 to execute one function module included in the response. In this case, a task that requires the execution of multiple function modules cannot be completed.

[0067] Therefore, in this embodiment, after transmitting an instruction in natural language from the user, the processing unit 111 includes the history log accumulated in the history log storage unit 112c in addition to the data read from the input buffer 112b in a prompt and transmits the prompt to the natural language processing system 50.

[0068] As a result, the natural language processing system 50 sends a response based on the data read from the input buffer 112b and the history log, and the processing unit 111 can cause the robot 10 to perform continuous actions based on past interactions based on the history log.

[0069] Consider an example in which an instruction is issued to the robot 10 for the first time to cause it to operate. In this case, the data initially stored in the input buffer 112b is an instruction from the user. At this stage, no data has been accumulated in the history log. Therefore, the natural language processing system 50 outputs a first response to the processing unit 111 based on the user's instruction read from the input buffer 112b. The data read from the input buffer 112b and the response obtained from the natural language processing system 50 corresponding to the read data are added to the history log as a set, so that the instruction from the user and the first response obtained in response to the instruction are added to the history log. The processing unit 111 identifies a functional module based on the first response, and saves and overwrites the execution result of the identified functional module in the input buffer 112b. Then, the processing unit 111 again transmits the data read from the input buffer 112b and the history log together in a prompt to the natural language processing system 50. Then, the natural language processing system 50 outputs the next answer to the processing unit 111, and the processing unit 111 identifies the next function module.

[0070] In this way, the processing unit 111 can use the natural language processing system 50 to cause the robot 10 to execute a functional module that takes into account past events. Once the user gives an instruction to the robot 10, the processing unit 111 continues the process of sending prompts to the natural language processing system 50 until the natural language processing system 50 determines that the robot 10's operation has been completed, receiving an answer as to the operation the robot 10 should perform, and then causing the robot 10 to operate, thereby making it possible for the robot 10 to perform continuous operations that take into account past interactions based on the history log.

[0071] 5 and 6 are diagrams showing examples of history logs stored in the history log storage unit 112c. Fig. 5 shows a history log showing a case where the robot 10 performs a series of actions as expected in response to a user's instruction to "make hot tea," including picking up a cup from the table, placing the cup in the pot, pouring "more" amount of hot water, picking up the cup from the pot, placing the cup on the table, and replying, "Hot tea has been made." Fig. 6 shows a history log showing a case where the robot 10 does not perform the above series of actions as expected (for example, trying to pour hot water without picking up the cup).

[0072] In FIG. 5, the first line is log data indicating that the user has instructed the robot 10 to "make me some hot tea."

[0073] The second line is log data indicating that the natural language processing system 50, in response to the instruction "Please make me some hot tea," has determined the functional module Pick_cup_from from the function list and determined "table" as the parameter data to be given to the functional module Pick_cup_from. The natural language processing system 50 identified the functional module Pick_cup_from("table") because it received a prompt including a precondition describing the initial state that the cup is on the table. In this way, the log data on the first and second lines are the input to the natural language processing system 50 and the output from the natural language processing system 50, respectively, and are added as a set to the history log storage unit 112c as a history log.

[0074] The third line is log data indicating that the functional module Pick_cup_from("table") was executed normally by the processing unit 111, a success message "The cup has been picked up" was returned as a return value (i.e., a normal execution result), and this success message was overwritten and saved in the input buffer 112b. Here, the reason why the functional module Pick_cup_from("table") was executed normally is because the processing unit 111 confirmed, through the execution of the functional module Pick_cup_from("table"), that the current state of the cup managed by the state management unit 111a (here, the default Cup_location=0) is the specified location of the cup (i.e., table). Furthermore, if the functional module can be executed normally, the state management unit 111a updates the current state of the target operated by the robot 10 based on the execution result of the functional module. Therefore, based on the execution result of the functional module Pick_cup_from("table") (i.e., the success message "The cup has been picked up"), the state management unit 111a determines that the cup has been moved from the table to the hand and updates the value of Cup_location from 0 to 2.

[0075] The fourth line is log data indicating that when a prompt containing the execution result of the third line read from the input buffer 112b was sent to the natural language processing system 50, the natural language processing system 50 determined the function module Place_cup_to from the function list and determined "pot" as the parameter to be given to the function module Place_cup_to.

[0076] The fifth line is log data indicating that the functional module Place_cup_to("pot") was executed normally by the processing unit 111, a success message "The cup has been placed" was returned as a return value (i.e., a normal execution result), and this success message was overwritten and saved in the input buffer 112b. Here, the reason why the functional module Place_cup_to("pot") was executed normally is because the processing unit 111 confirmed, through the execution of the functional module Place_cup_to("pot"), that the current state of the cup held by the state management unit 111a (here, Cup_location=2) is the hand of the arm robot 12. Furthermore, based on the execution result of the functional module Place_cup_to("pot") (i.e., the success message "The cup has been placed"), the state management unit 111a determines that the cup has been moved from the hand to the pot, and updates the value of Cup_location from 2 to 1.

[0077] The sixth line is log data indicating that when a prompt containing the execution result of the fifth line read from the input buffer 112b was sent to the natural language processing system 50, the natural language processing system 50 determined the function module Pour_hot_water from the function list and determined "more" as the parameter to be given to the function module Pour_hot_water.

[0078] Line 7 is log data indicating that the functional module Pour_hot_water("more") was executed normally by the processing unit 111, a success message "The specified amount has been poured" was returned as a return value (i.e., a normal execution result), and this success message was overwritten and saved in the input buffer 112b. Here, the functional module Pour_hot_water("more") was executed normally because the processing unit 111 confirmed, through the execution of the functional module Pour_hot_water("more"), that the current state of the cup held by the state management unit 111a (here, Cup_location=1) is a pot. Note that even after executing the functional module Pour_hot_water("more"), the state management unit 111a does not update the value of Cup_location because the cup remains in the pot.

[0079] The eighth line is log data indicating that when a prompt containing the execution result of the seventh line read from the input buffer 112b was sent to the natural language processing system 50, the natural language processing system 50 determined the function module Pick_cup_from from the function list and determined "pot" as the parameter to be given to the function module Pick_cup_from.

[0080] Line 9 is log data indicating that the functional module Pick_cup_from("pot") was executed normally by the processing unit 111, a success message "The cup has been picked up" was returned as a return value (i.e., a normal execution result), and this success message was overwritten and saved in the input buffer 112b. Here, the reason why the functional module Pick_cup_from("pot") was executed normally is because the processing unit 111 confirmed, through the execution of the functional module Pick_cup_from("pot"), that the current state of the cup held by the state management unit 111a (here, Cup_location=1) is a pot. Furthermore, based on the execution result of the functional module Pick_cup_from("pot") (i.e., the success message "The cup has been picked up"), the state management unit 111a updates the value of Cup_location from 1 to 2, assuming that the cup has been moved from the pot to the hand.

[0081] Line 10 is log data indicating that when a prompt containing the execution result of line 9 read from input buffer 112b was sent to natural language processing system 50, natural language processing system 50 determined the function module Place_cup_to from the function list and determined "table" as the parameter to be given to function module Place_cup_to.

[0082] Line 11 is log data indicating that the functional module Place_cup_to("table") was executed normally by the processing unit 111, a success message "The cup has been placed" was returned as a return value (i.e., a normal execution result), and this success message was overwritten and saved in the input buffer 112b. Here, the reason why the functional module Place_cup_to("table") was executed normally is because the processing unit 111 confirmed, through the execution of the functional module Place_cup_to("table"), that the current state of the cup held by the state management unit 111a (here, Cup_location=2) is the hand of the arm robot 12. Furthermore, based on the execution result of the functional module Place_cup_to("table") (i.e., the success message "The cup has been placed"), the state management unit 111a determines that the cup has been moved from the hand to the table, and updates the value of Cup_location from 2 to 0.

[0083] Line 12 is log data indicating that when a prompt containing the execution result of line 11 read from input buffer 112b was sent to the natural language processing system 50, the natural language processing system 50 determined the function module Speak from the function list and determined "I have made hot tea" as the parameter to be given to the function module Speak.

[0084] The thirteenth line is log data indicating that the functional module Speak is executed by the processing unit 111, the message "Speaking has been done" is returned as a return value, and this message is overwritten and saved in the input buffer 112b.

[0085] Line 14 is log data indicating that when a prompt containing the execution result of line 13 read from input buffer 112b was sent to natural language processing system 50, a response was received from natural language processing system 50 that did not include a specification of the function module to be called (i.e., no function call).

[0086] Next, a description will be given of the history log in Fig. 6. In Fig. 6, the first line is log data indicating that the user has instructed the robot 10 to "make me some hot tea."

[0087] The second line is log data indicating that the natural language processing system 50, in response to the instruction "Please make some hot tea," deviated from the above series of operations, selected the function module Pour_hot_water from the function list, and determined "more" as the parameter data to be given to the function module Pour_hot_water. In this way, even if the natural language processing system 50 is given a prompt including a precondition describing the initial state that a cup is on the table, it may not necessarily return an appropriate answer.

[0088] The third line is log data indicating that the functional module Pour_hot_water("more") was executed by the processing unit 111 but was not executed normally, resulting in a return value (i.e., an abnormal execution result) of the message "The cup is not in the 'pot'. It is currently in the 'table'." This message was overwritten and saved in the input buffer 112b. Here, the functional module Pour_hot_water("more") was not executed normally because the processing unit 111 confirmed, through execution of the functional module Pour_hot_water("more"), that the current state of the cup held by the state management unit 111a (here, the default Cup_location = 0) is not a pot. In this case, the processing unit 111 obtains the current state of the cup (Cup_location = 0), assigns the current state of the cup to {Cup_location} in the return value of the abnormal execution result, and overwrites and saves information about the current state of the cup (i.e., the abnormal execution result indicating that the cup is currently in the table) in the input buffer 112b.

[0089] The fourth line is log data indicating that when a prompt containing the execution result of the third line read from the input buffer 112b was sent to the natural language processing system 50, the natural language processing system 50 determined the function module Pick_cup_from from the function list and determined "table" as the parameter to be given to the function module Pick_cup_from.

[0090] As described above, in this embodiment, if the function module Pour_hot_water("more") identified by the response from the natural language processing system 50 cannot be executed normally (in this case, an attempt was made to pour hot water into the pot even though a cup was not placed in it), the current state of the cup is included in a prompt and sent to the natural language processing system 50, causing the natural language processing system 50 to reconsider the operation of the robot 10. By causing the natural language processing system 50 to reconsider, the natural language processing system 50 can generate a more accurate response. Although the response from the natural language processing system 50 may contain ambiguity, the configuration of this embodiment makes it possible to avoid the robot 10 from executing an erroneous operation even when an response is impossible to execute in light of the actual situation, thereby preventing the entire system from becoming inoperable.

[0091] The fifth line is log data indicating that the functional module Pick_cup_from("table") was executed normally by the processing unit 111, a success message "The cup has been picked up" was returned as a return value (i.e., a normal execution result), and this success message was overwritten and saved in the input buffer 112b. Here, the reason why the functional module Pick_cup_from("table") was executed normally is because the processing unit 111 confirmed, through the execution of the functional module Pick_cup_from("table"), that the current state of the cup held by the state management unit 111a (here, the default Cup_location=0) is the specified cup location (table). Furthermore, based on the execution result of the functional module Pick_cup_from("table") (i.e., the success message "The cup has been picked up"), the state management unit 111a updates the value of Cup_location from 0 to 2, assuming that the cup has been moved from the table to the hand.

[0092] The sixth and subsequent lines are the same as the fourth and subsequent lines in FIG. 5, and therefore the explanation will be omitted.

[0093] The history log stores a history from the first instruction to the robot 10 as a set of inputs to the natural language processing system 50 and outputs from the natural language processing system 50, allowing the natural language processing system 50 to understand what has been done and to what extent. By sending this data to the natural language processing system 50, the natural language processing system 50 can obtain an appropriate answer about the next action to be taken, taking into account past events.

[0094] Next, specific processing by the processing unit 111 will be described with reference to Fig. 7. Fig. 7 is a flowchart for explaining the flow of processing by the processing unit 111.

[0095] In step S701, processing unit 111 reads the function list stored in function list storage unit 112a and the prerequisites stored in prerequisite storage unit 112d.

[0096] In step S702, the processing unit 111 determines whether there is an input from the voice recognition unit 115, that is, whether there is an instruction from the user via the microphone 13. If there is an input from the voice recognition unit 115, the flow proceeds to step S703. If there is no input from the voice recognition unit 115, the flow returns to step S702 and waits for an input from the voice recognition unit 115.

[0097] In step S703, the processing unit 111 overwrites the input from the voice recognition unit 115 in the input buffer 112b.

[0098] In step S704, the processing unit 111 transmits the function list and prerequisites read in step S701, as well as the data overwritten in the input buffer 112b and the history log accumulated in the history log storage unit 112c, to the natural language processing system 50. If no history log has been accumulated yet, a history log of empty data will be transmitted.

[0099] In step S705, the processing unit 111 adds to the history log the data in the input buffer 112b and the response from the natural language processing system 50. Note that the data added to the history log in step S705 is a set of data including an input to the natural language processing system 50 and an output corresponding to the input.

[0100] In step S706, the processing unit 111 executes a function module identified by the response from the natural language processing system 50. Since the processing unit 111 performs different processes depending on the function module identified by the response from the natural language processing system 50, each function module will be described later with reference to FIGS. 8A to 8L.

[0101] In step S707, the processing unit 111 overwrites the input buffer 112b with the execution result of the functional module.

[0102] In step S708, the processing unit 111 records the execution result of the function module in the history log, and the flow then returns to step S704.

[0103] In addition, in Figure 7, an example has been described in which the processing unit 111 includes the data in the input buffer 112b, the function list, the history log, and the prerequisites in the prompt and transmits it to the natural language processing system 50. However, the processing unit 111 may also include at least one of these in addition to the data in the input buffer 112b in the prompt and transmit it to the natural language processing system 50.

[0104] For example, if the prompt includes all of the input buffer data, function list, history log, and preconditions, the robot 10 can be made to perform flexible and appropriate operations by utilizing the preconditions that define the operating policy, the history log can be used to loop processing and make the robot 10 perform continuous operations that take past events into consideration, and the function list can be used to make the robot 10 perform intended operations. Furthermore, by utilizing the function list, the natural language processing system 50 can return an answer as to which function module should be executed. Since data identifying a predetermined function module is obtained as the answer, it can be accumulated as a history log in a format that can be used later. Therefore, the history log can be included in the prompt, and the next answer can be obtained appropriately.

[0105] Next, the processing of the processing unit 111 in step S706 for each functional module will be described with reference to FIGS. 8A to 8L.

[0106] FIG. 8A is a flowchart for explaining the details of the processing performed by the processing unit 111 in step S706 when the functional module Pick_cup_from is executed.

[0107] In step S801A, processing unit 111 checks whether the current state of the cup is the location specified by the response from natural language processing system 50. If the current state of the cup is not the specified location, the flow proceeds to step S802A. If the current state of the cup is the specified location, the flow proceeds to step S803A.

[0108] In step S802A, processing unit 111 acquires information about the current state of the cup as an abnormal execution result by including the current state of the cup acquired from state management unit 111a in the return value that is returned when the functional module is not executed normally. Then, the flow proceeds to step S707, where input buffer 112b is overwritten with the acquired abnormal execution result.

[0109] In step S803A, if the current state of the cup is at the location specified by the response from the natural language processing system 50, the processing unit 111 causes the arm robot 12 to perform the action of picking up the cup based on the action command generated by executing the functional module.

[0110] In step S804A, the processing unit 111 acquires a normal execution result as a return value that is returned when the functional module is executed normally.

[0111] In step S805A, state management unit 111a in processing unit 111 updates the current state of the cup based on the acquired normal execution result. Then, the flow proceeds to step S707, where the normal execution result acquired in step S804A is overwritten in input buffer 112b.

[0112] FIG. 8B is a flowchart for explaining the details of the processing performed by the processing unit 111 in step S706 when the functional module Place_cup_to is executed.

[0113] In step S801B, the processing unit 111 checks whether the current state of the cup is in the hand of the arm robot 12 (that is, whether the cup is being held by the hand of the arm robot 12). If the current state of the cup is not in the hand of the arm robot 12, the flow proceeds to step S802B. If the current state of the cup is in the hand of the arm robot 12, the flow proceeds to step S803B.

[0114] In step S802B, processing unit 111 includes the current state of the cup acquired from state management unit 111a in the return value returned when the functional module is not executed normally, and acquires information about the current state of the cup as an abnormal execution result. Then, the flow proceeds to step S707, where input buffer 112b is overwritten with the acquired abnormal execution result.

[0115] In step S803B, if the current state of the cup is the hand of the arm robot 12, the processing unit 111 checks whether or not the location specified by the response from the natural language processing system 50 is valid. If the specified location is not valid, the flow proceeds to step S804B, and if the specified location is valid, the flow proceeds to step S805B.

[0116] In step S804B, the processing unit 111 receives an error message indicating that the specified location is invalid (i.e., a cup is already placed in the specified location). Then, the flow proceeds to step S707, where the processing unit 111 overwrites the input buffer 112b with the received error message.

[0117] In step S805B, the processing unit 111 causes the arm robot 12 to execute an operation to place the cup based on the operation command generated by the execution of the functional module.

[0118] In step S806B, the processing unit 111 acquires a normal execution result as a return value that is returned when the functional module is executed normally.

[0119] In step S807B, state management unit 111a in processing unit 111 updates the current state of the cup based on the acquired normal execution result. Then, the flow proceeds to step S707, where the normal execution result acquired in step S806B is overwritten in input buffer 112b.

[0120] FIG. 8C is a flowchart for explaining the details of the processing performed by the processing unit 111 in step S706 when the functional module Pour_hot_water is executed.

[0121] In step S801C, the processing unit 111 checks whether the current state of the cup is a pot. If the current state of the cup is not a pot, the flow proceeds to step S802C. If the current state of the cup is a pot, the flow proceeds to step S803C.

[0122] In step S802C, processing unit 111 acquires information about the current state of the cup as an abnormal execution result by including the current state of the cup acquired from state management unit 111a in the return value that is returned when the functional module is not executed normally. Then, the flow proceeds to step S707, where input buffer 112b is overwritten with the acquired abnormal execution result.

[0123] In step S803C, if the current state of the cup is a pot, the processing unit 111 causes the arm robot 12 to perform an operation of pouring hot water into the cup based on the operation command generated by the execution of the function module.

[0124] In step S804C, processing unit 111 acquires a normal execution result as a return value that is returned when the functional module is executed normally. Then, the flow proceeds to step S707, where the normal execution result acquired in step S804C is overwritten in input buffer 112b. Note that, because the position of the cup does not change before and after the action of pouring hot water, in this process, state management unit 111a does not update the current state of the cup.

[0125] FIG. 8D is a flowchart for explaining the details of the processing performed by the processing unit 111 in step S706 when the functional module Pour_cold_water is executed.

[0126] In step S801D, the processing unit 111 checks whether the current state of the cup is the hand of the arm robot 12. If the current state of the cup is not the hand, the flow proceeds to step S802D. If the current state of the cup is the hand, the flow proceeds to step S803D.

[0127] In step S802D, processing unit 111 acquires information about the current state of the cup as an abnormal execution result by including the current state of the cup acquired from state management unit 111a in the return value that is returned when the functional module is not executed normally. Then, the flow proceeds to step S707, where input buffer 112b is overwritten with the acquired abnormal execution result.

[0128] In step S803D, if the current state of the cup is the hand, the processing unit 111 causes the arm robot 12 to execute an operation of pouring water into the cup based on the operation command generated by the execution of the function module.

[0129] In step S804D, processing unit 111 acquires a normal execution result as a return value that is returned when the functional module is executed normally. Then, the flow proceeds to step S707, where the normal execution result acquired in step S804D is overwritten in input buffer 112b. Note that because the position of the cup does not change before and after the action of pouring water, in this process, state management unit 111a does not update the current state of the cup.

[0130] FIG. 8E is a flowchart for explaining the details of the processing performed by the processing unit 111 in step S706 when the functional module Pour_alcohol is executed.

[0131] In step S801E, the processing unit 111 checks whether the current state of the cup is the hand of the arm robot 12. If the current state of the cup is not the hand, the flow proceeds to step S802E. If the current state of the cup is the hand, the flow proceeds to step S803E.

[0132] In step S802E, processing unit 111 acquires information about the current state of the cup as an abnormal execution result by including the current state of the cup acquired from state management unit 111a in the return value that is returned when the functional module is not executed normally. Then, the flow proceeds to step S707, where input buffer 112b is overwritten with the acquired abnormal execution result.

[0133] In step S803E, if the current state of the cup is the hand, the processing unit 111 causes the arm robot 12 to execute an operation of pouring alcohol based on the operation command generated by the execution of the functional module.

[0134] In step S804E, processing unit 111 acquires a normal execution result as a return value that is returned when the functional module is executed normally. Then, the flow proceeds to step S707, where the normal execution result acquired in step S804E is overwritten in input buffer 112b. Note that, because the position of the cup does not change before and after the act of pouring alcohol, in this process, state management unit 111a does not update the current state of the cup.

[0135] FIG. 8F is a flowchart for explaining the details of the processing performed by the processing unit 111 in step S706 when the functional module Pour_ice is executed.

[0136] In step S801F, the processing unit 111 checks whether the current state of the cup is the hand of the arm robot 12. If the current state of the cup is not the hand, the flow proceeds to step S802F. If the current state of the cup is the hand, the flow proceeds to step S803F.

[0137] In step S802F, processing unit 111 acquires information about the current state of the cup as an abnormal execution result by including the current state of the cup acquired from state management unit 111a in the return value that is returned when the functional module is not executed normally. Then, the flow proceeds to step S707, where input buffer 112b is overwritten with the acquired abnormal execution result.

[0138] In step S803F, if the current state of the cup is the hand, the processing unit 111 causes the arm robot 12 to execute an operation of putting ice into the cup based on the operation command generated by the execution of the function module.

[0139] In step S804F, processing unit 111 acquires a normal execution result as a return value that is returned when the functional module is executed normally. Then, the flow proceeds to step S707, where the normal execution result acquired in step S804F is overwritten in input buffer 112b. Note that, because the position of the cup does not change before and after the action of adding ice, in this process, state management unit 111a does not update the current state of the cup.

[0140] FIG. 8G is a flowchart for explaining the details of the processing by the processing unit 111 in step S706 when the functional module Stir is executed.

[0141] In step S801G, the processing unit 111 checks whether the current state of the cup is the hand of the arm robot 12. If the current state of the cup is not the hand, the flow proceeds to step S802G. If the current state of the cup is the hand, the flow proceeds to step S803G.

[0142] In step S802G, processing unit 111 acquires information about the current state of the cup as an abnormal execution result by including the current state of the cup acquired from state management unit 111a in the return value that is returned when the functional module is not executed normally. Then, the flow proceeds to step S707, where input buffer 112b is overwritten with the acquired abnormal execution result.

[0143] In step S803G, if the current state of the cup is the hand, the processing unit 111 causes the arm robot 12 to perform a steering operation based on the operation command generated by the execution of the function module.

[0144] In step S804G, processing unit 111 obtains a normal execution result as a return value that is returned when the functional module is executed normally. Then, the flow proceeds to step S707, where the normal execution result obtained in step S804G is overwritten in input buffer 112b. Note that because the position of the cup does not change before and after the steering operation, in this process, state management unit 111a does not update the current state of the cup.

[0145] FIG. 8H is a flowchart for explaining the details of the processing performed by the processing unit 111 in step S706 when the function module Get_cup_location is executed.

[0146] In step S801H, processing unit 111 executes the function module to acquire the current state of the cup acquired from state management unit 111a as information relating to the current state of the cup. Then, the flow proceeds to step S707, where input buffer 112b is overwritten with the current state of the cup, which is the acquired execution result.

[0147] FIG. 8I is a flowchart for explaining the details of the operation of the processing unit 111 in step S706 when the functional module Move_to is executed.

[0148] In step S801I, the processing unit 111 confirms the location specified by the response from the natural language processing system 50. If the specified location is the water supply button or the ice supply button, the flow proceeds to step S801I2. If the specified location is the initial position, the flow proceeds to step S804I.

[0149] In step S802I, the processing unit 111 causes the arm robot 12 to perform an operation to move to a designated location based on an operation command generated by the execution of the functional module.

[0150] In step S803I, the processing unit 111 acquires a normal execution result as a return value to be returned when the functional module is executed normally. Then, the flow proceeds to step S707, where the normal execution result acquired in step S803I is overwritten in the input buffer 112b.

[0151] In step S804I, the processing unit 111 checks whether the current state of the cup is the hand of the arm robot 12. If the current state of the cup is the hand, the flow proceeds to step S805I. If the current state of the cup is not the hand, the flow proceeds to step S806I.

[0152] In step S805I, the processing unit 111 acquires an error message saying, "You cannot move to the specified position while holding the cup." After that, the flow proceeds to step S707, where the processing unit 111 overwrites the input buffer 112b with the error message acquired as the execution result in step S805I.

[0153] In step S806I, the processing unit 111 causes the arm robot 12 to perform an operation to move to a designated location based on the operation command generated by the execution of the functional module.

[0154] In step S807I, the processing unit 111 acquires a normal execution result as a return value to be returned when the functional module is executed normally. Then, the flow proceeds to step S707, where the normal execution result acquired in step S807I is overwritten in the input buffer 112b.

[0155] FIG. 8J is a flowchart for explaining the details of the processing performed by the processing unit 111 in step S706 when the function module Search_for is executed.

[0156] In step S801J, the processing unit 111 executes a function module to perform a search.

[0157] In step S802J, the processing unit 111 acquires the search results as the execution results, and then the flow proceeds to step S707, where the acquired search results are overwritten in the input buffer 112b.

[0158] FIG. 8K is a flowchart for explaining the details of the processing by the processing unit 111 in step S706 when the functional module Search_weather is executed.

[0159] In step S801K, the processing unit 111 executes a function module to search for the weather.

[0160] In step S802K, processing unit 111 acquires the search results as the execution results, and then the flow proceeds to step S707, where input buffer 112b is overwritten with the acquired search results.

[0161] FIG. 8L is a flowchart for explaining the details of the processing by the processing unit 111 in step S706 when the functional module Speak is executed.

[0162] In step S801L, the processing unit 111 causes the robot 10 to speak by executing the function module.

[0163] In step S802L, the processing unit 111 acquires the execution result, and then the flow proceeds to step S707, where the acquired search result is overwritten in the input buffer 112b.

[0164] Next, specific processing by the robot control unit 121 will be described with reference to Fig. 9. Fig. 9 is a flowchart showing processing by the robot control unit 121.

[0165] In step S901, the robot control unit 121 determines whether or not there is an input from the processing unit 111. If there is an input from the processing unit 111, that is, an operation command for the arm robot 12, the flow proceeds to step S902. If there is no input from the processing unit 111, the flow returns to step S901.

[0166] In step S902, the robot control unit 121 executes the operation instructed in the operation command.

[0167] In step S903, the robot control unit 121 notifies the processing unit 111 of the end of the operation.

[0168] As described above, when the control device 11 according to the present embodiment cannot normally execute a functional module identified by a response from the natural language processing system 50 (for example, when trying to pour hot water into a pot when no cup is placed therein, as shown in FIG. 6 ), the control device 11 transmits the current state of the cup to the natural language processing system 50 as a prompt, thereby causing the natural language processing system 50 to reconsider the operation of the robot 10. By causing the natural language processing system 50 to reconsider, the natural language processing system 50 can generate a more accurate response. Although the response from the natural language processing system 50 may contain ambiguity, the configuration of the present embodiment makes it possible to prevent the robot 10 from executing an erroneous operation even when an answer is impossible to execute in light of the actual situation, thereby preventing the entire system from becoming inoperable.

[0169] (Second embodiment) Next, a robot system 200 according to a second embodiment will be described. In the robot system 100 according to the first embodiment, the natural language processing system 50 is made to identify a functional module that specifies the operation of the robot 10 without providing the natural language processing system 50 with information about the current state of the object (i.e., the cup) that the robot 10 is to operate. For this reason, as described in FIG. 6, even if the natural language processing system 50 is given information including the prerequisites and the history log including the initial state of the cup, the natural language processing system 50 may not always return an appropriate answer.

[0170] Therefore, in the robot system 200 according to the second embodiment, before the natural language processing system 50 is made to respond with a functional module that specifies the behavior of the robot 10, the current state of the cup that is the object to be operated by the robot 10 is given to the natural language processing system 50. This increases the likelihood that the natural language processing system 50 will generate a response that takes into account the current state of the cup, and therefore makes it possible to generate a response that is suitable for causing the robot 10 to perform a series of actions expected of the robot 10 without deviating from the expected series of actions.

[0171] The configuration of the robot system 200 according to this embodiment is the same as the configuration of the robot system 100 according to the first embodiment, and therefore a description thereof will be omitted.

[0172] Specific processing by the processing unit 111 according to this embodiment will be described with reference to Fig. 10. Fig. 10 is a flowchart for explaining the flow of processing by the processing unit 111. The processing in steps S1004 and S1005 differs from the processing by the processing unit 111 according to the first embodiment. The other processing is the same as the processing by the processing unit 111 according to the first embodiment.

[0173] In step S1001, processing unit 111 reads the function list stored in function list storage unit 112a and the prerequisites stored in prerequisite storage unit 112d. This step S1001 is the same as step S701 in FIG.

[0174] In step S1002, processing unit 111 determines whether there is input from voice recognition unit 115, that is, whether there is an instruction from the user via microphone 13. If there is input from voice recognition unit 115, the flow proceeds to step S1003. If there is no input from voice recognition unit 115, the flow returns to step S1002 and waits for input from voice recognition unit 115. This step S1002 is the same as step S702 in FIG. 7.

[0175] In step S1003, the processing unit 111 overwrites the input buffer 112b with the input from the speech recognition unit 115. This step S1003 is the same as step S703 in FIG.

[0176] In step S1004, the processing unit 111 acquires the current state of the cup from the state management unit 111a. The processing unit 111 may acquire the current state of the cup (i.e., the current state of the object manipulated by the robot 10) from the state management unit 111a by executing a function module Get_cup_location. In this sense, the function module Get_cup_location can be said to be a state acquisition module that acquires the current state of the object manipulated by the robot 10.

[0177] In step S1005, the processing unit 111 transmits to the natural language processing system 50 the function list and prerequisites read in step S1001, as well as the data overwritten in the input buffer 112b, the history log stored in the history log storage unit 112c, and the current state of the cup acquired in step S1004, all included in a prompt. If no history log has been stored yet, an empty history log will be transmitted.

[0178] In step S1006, the processing unit 111 adds the data in the input buffer 112b and the response from the natural language processing system 50 to the history log. This step S1006 is the same as step S705 in FIG.

[0179] In step S1007, the processing unit 111 executes the function module identified by the response from the natural language processing system 50. This step S1007 is the same as step S706 in FIG.

[0180] In step S1008, the processing unit 111 overwrites the execution result of the functional module in the input buffer 112b. This step S1008 is the same as step S707 in FIG.

[0181] In step S1009, the processing unit 111 records the execution result of the function module in the history log. Then, the flow returns to step S1004. This step S1009 is the same as step S708 in FIG.

[0182] The processing of the robot control unit 121 according to this embodiment is similar to the processing of the robot control unit 121 according to the first embodiment.

[0183] As described above, in this embodiment, by including the current state of the cup in the prompt and sending it to the natural language processing system 50, the natural language processing system 50 can generate an answer that takes into account the current state of the cup. This allows the natural language processing system 50 to identify a function module that can be executed normally. Therefore, the processing unit 111 is more likely to obtain a normal execution result by executing the function module identified by the answer from the natural language processing system 50, and can cause the robot 10 to smoothly perform a series of actions as expected.

[0184] In this embodiment, even if the current state of the cup is included in the prompt and sent to the natural language processing system 50, the natural language processing system 50 does not necessarily return the expected answer, so the processing unit 111 performs processing to check the current state of the cup, such as step 801A in Figure 8A and step 801B in Figure 8B.

[0185] (Third embodiment) Next, a robot system 300 according to a third embodiment will be described. In the robot system 100 according to the first embodiment and the robot system 200 according to the second embodiment, the processing unit 111 confirms the current state of the object operated by the robot 10 (i.e., the current state of the cup) using the current state of the cup managed by the state management unit 111a.

[0186] On the other hand, in the robot system 30 according to the third embodiment, the current state of the cup is confirmed using a sensor 122 such as a camera that is provided in the arm robot 12 and that acquires the current state of the cup.

[0187] 11 is a diagram showing an example of the configuration of a robot system 300 according to the third embodiment. Note that the same members as those in the robot system 100 according to the first embodiment are given the same reference numerals, and their description may be omitted.

[0188] The robot system 300 of this embodiment is similar to the robot system 100 of the first embodiment, except that the arm robot 22 has a sensor 122 and the processing unit 211 does not have a state management unit 111a.

[0189] In the arm robot 22, the sensor 122 is a wide-angle camera that can view the surroundings of the robot 10, and is provided, for example, on the arm. The sensor 122 may be provided anywhere as long as it can view the surroundings of the robot 10.

[0190] Next, specific processing by processing unit 211 according to this embodiment will be described, which is the same as the processing by processing unit 111 according to the first embodiment except for the processing by processing unit 211 in step S706 in Fig. 7 when the functional module is executed. Therefore, only the processing by processing unit 211 in step S706 in Fig. 7 when the functional module is executed will be described here.

[0191] FIG. 12 is a flowchart for explaining the details of the processing of the processing unit 211 in step S706 when the functional module Pick_cup_from is executed, as an example.

[0192] In step S1201A, processing unit 111 executes a functional module to acquire the current state of the cup using sensor 122, and checks whether the current state of the cup is at the location specified by the response from natural language processing system 50. In this respect, this embodiment differs from the first and second embodiments in that functional modules other than functional modules Search_for, Search_weather, and Speak (i.e., functional modules including a process for acquiring the current state of the cup) include an instruction to acquire the current state of the target using sensor 122. Therefore, processing unit 211 can acquire the current state of the cup using sensor 122 by executing the functional module. The other processes are similar to steps S802A to S805A in FIG. 8A, and therefore will not be described again.

[0193] In this embodiment, with regard to the processing of the processing unit 211 in step S706 when executing another functional module that includes processing for acquiring the current state of the cup, the content of the step can be reinterpreted as follows.

[0194] The content of step S801B in FIG. 8B should be read as "check with a sensor whether the cup is currently in a hand."

[0195] The content of step S801C in FIG. 8C should be read as "check whether the current state of the cup is a pot using a sensor."

[0196] The content of step S801D in FIG. 8D should be read as "check with a sensor whether the cup is currently in a hand state."

[0197] The content of step S801E in FIG. 8E should be read as "check with a sensor whether the cup is currently in a hand or not."

[0198] The content of step S801F in FIG. 8F should be read as "check with a sensor whether the current state of the cup is a hand or not."

[0199] The content of step S801G in FIG. 8G should be read as "check with a sensor whether the current state of the cup is a hand or not."

[0200] The content of step S801H in FIG. 8H should be read as "the current state of the cup is acquired by the sensor."

[0201] The content of step S804I in FIG. 8I should be read as "check with a sensor whether the cup is currently in a hand or not."

[0202] As described above, in this embodiment, the current state of the cup can be acquired by the sensor 122, and therefore there is no need to have the state management unit 111a according to the first and second embodiments manage the current state of the cup. The state management unit 111a updates the current state of the cup based on the normal execution result of the functional module, but there is no guarantee that the updated state is correct. In other words, it is not known whether the cup is actually placed in the updated location. In this embodiment, the current state of the cup is acquired using the sensor 122, so the information about where the cup is placed is correct (information that corresponds to the situation in the real world). Therefore, correct information about the current state of the cup can be provided to the natural language processing system 50.

[0203] The processing of the robot control unit 121 according to this embodiment is similar to the processing of the robot control unit 121 according to the first embodiment.

[0204] (Other embodiments) The above describes embodiments of the present disclosure, but the present disclosure should not be construed as being limited to the above embodiments, and can be applied to various embodiments and combinations within the scope that does not deviate from the gist of the present disclosure.

[0205] Furthermore, the processing flow described in the above embodiment is also an example, and unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged within the scope of the present invention.

[0206] In each of the above embodiments, instructions are given to the robot 10 via the microphone 13. However, instructions may be given to the robot 15, 25 not only by voice via the microphone 13, but also by text information via a keyboard or touch panel. Furthermore, text information may be provided to the user using a display instead of the speaker 14.

[0207] In the second embodiment, in step S1004 of FIG. 10, the current state of the cup may be acquired using the sensor 122 according to the third embodiment.

[0208] It may be provided in the following manner. (Aspect 1) A control device (11) for controlling a robot (10) based on an interaction with a user, an input unit (114) that receives instructions from the user for the robot; an output unit (117) that outputs a response to the instruction to the user; a function list storage unit (112a) in which each of a plurality of predetermined actions to be executed by the robot is defined as a function module and a function list of the function modules is stored; an input buffer (112b) for storing the instruction data input from the input unit; a processing unit (111, 211) that transmits a prompt including the function list and data read from the input buffer to a natural language processing system using a large-scale language model, receives a response from the natural language processing system indicating which function module in the function list should be executed, and executes the function module identified by the response; Equipped with When the processing unit cannot execute the functional module normally, the control device stores information about the current state of the object being operated by the robot in the input buffer.

[0209] (Aspect 2) 2. The control device according to aspect 1, wherein the processing unit includes a state management unit (111a) that manages a current state of an object operated by the robot.

[0210] (Aspect 3) 3. The control device according to aspect 2, wherein the state management unit updates the current state of the target based on a result of execution of the functional module when the functional module can be executed normally.

[0211] (Aspect 4) 4. The control device according to any one of aspects 1 to 3, wherein the function list storage unit stores necessary conditions for the function modules to be executed normally in association with the function modules.

[0212] (Aspect 5) a precondition storage unit (112d) that stores preconditions that define an action policy for the action to be performed by the robot; a history log storage unit (112c) that accumulates a history log including data read from the input buffer and responses from the natural language processing system obtained in response to the read data, The prerequisite defines an initial state of the object to be operated by the robot, 5. The control device according to any one of aspects 1 to 4, wherein the processing unit transmits the prompt including the prerequisites and the history log to the natural language processing system.

[0213] (Aspect 6) A control device according to any one of aspects 1 to 5, wherein the processing unit includes a current state of the subject in the prompt.

[0214] (Aspect 7) The function list defines a state acquisition module that acquires a current state of the object operated by the robot, 7. The control device according to claim 6, wherein the processing unit acquires a current state of the object by executing the state acquisition module.

[0215] (Aspect 8) 8. The control device according to any one of aspects 1 to 7, wherein the processing unit acquires a current state of the object operated by the robot using a sensor (122) that acquires the state of the object.

[0216] (Aspect 9) the functional module includes instructions for obtaining a current state of the object using the sensor; 9. The control device according to claim 8, wherein the processing unit acquires a current state of the object by executing the functional module.

[0217] (Aspect 10) 1. A control method for controlling a robot based on an interaction with a user, comprising: a reading step of reading a function list from a function list storage unit storing a function list of the predetermined plurality of operations to be executed by the robot, each of the predetermined plurality of operations being defined as a function module; an input step of receiving an instruction from the user to the robot; a storing step of storing the instruction data inputted in the input step in an input buffer; a processing step of transmitting a prompt including the function list and the data read from the input buffer to a natural language processing system utilizing a large-scale language model, receiving a response from the natural language processing system indicating which function module in the function list should be executed, and executing the function module identified by the response; an output step of outputting a response to the instruction to the user; Equipped with In the processing step, if the functional module cannot be executed normally, information about the current state of the object being operated by the robot is stored in the input buffer.

[0218] (Aspect 11) A control program for controlling a robot based on an interaction with a user, On the computer, a reading step of reading a function list from a function list storage unit storing a function list of the predetermined plurality of operations to be executed by the robot, each of the predetermined plurality of operations being defined as a function module; an input step of receiving an instruction from the user to the robot; a storing step of storing the instruction data inputted in the input step in an input buffer; a processing step of transmitting a prompt including the function list and the data read from the input buffer to a natural language processing system utilizing a large-scale language model, receiving a response from the natural language processing system indicating which function module in the function list should be executed, and executing the function module identified by the response; an output step of outputting a response to the instruction to the user; Execute If the functional module cannot be executed normally in the processing step, the control program stores information about the current state of the object being operated by the robot in the input buffer. [Explanation of symbols]

[0219] 100, 200, 300... robot system, 10... robot, 11: Control device; 12, 22: Arm robot; 121: Robot control unit; 122...sensor, 13...microphone, 14...speaker, 50 Natural language processing system, 60 Search server, 111, 211: Processing unit, 112: Storage unit, 112a: Function list storage unit, 112b: input buffer; 112c: history log storage unit; 112d: a precondition storage unit; 114: a speech input unit; 116: a speech synthesis unit; 117 Audio output section

Claims

1. A control device (11) for controlling a robot (10) based on an interaction with a user, comprising: an input unit (114) that receives instructions from the user for the robot; an output unit (117) that outputs a response to the instruction to the user; a function list storage unit (112a) in which each of a plurality of predetermined actions to be executed by the robot is defined as a function module and a function list of the function modules is stored; an input buffer (112b) for storing the instruction data input from the input unit; a processing unit (111, 211) that transmits a prompt including the function list and data read from the input buffer to a natural language processing system using a large-scale language model, receives a response from the natural language processing system indicating which function module in the function list should be executed, and executes the function module identified by the response; Equipped with When the processing unit cannot execute the functional module normally, the control device stores information about the current state of the object being operated by the robot in the input buffer.

2. The control device according to claim 1 , wherein the processing unit includes a state management unit (111 a) that manages a current state of an object operated by the robot.

3. The control device according to claim 2 , wherein the state management unit updates the current state of the target based on a result of execution of the functional module when the functional module can be executed normally.

4. 2. The control device according to claim 1, wherein the function list storage section stores necessary conditions for the normal execution of the function modules in association with the function modules.

5. a precondition storage unit (112d) that stores preconditions that define an action policy for the action to be performed by the robot; a history log storage unit (112c) that accumulates a history log including data read from the input buffer and responses from the natural language processing system obtained in response to the read data, The prerequisite defines an initial state of the object to be operated by the robot, The control device according to claim 1 , wherein the processing unit transmits the prompt including the prerequisites and the history log to the natural language processing system.

6. The control device of claim 1 , wherein the processor includes a current state of the subject in the prompt.

7. The function list defines a state acquisition module that acquires a current state of the object operated by the robot, The control device according to claim 6 , wherein the processing unit acquires the current state of the object by executing the state acquisition module.

8. The control device according to claim 1 , wherein the processing unit acquires a current state of the object operated by the robot using a sensor (122) that acquires the state of the object.

9. the functional module includes instructions for obtaining a current state of the object using the sensor; The control device according to claim 8 , wherein the processing unit acquires a current state of the object by executing the function module.

10. 1. A control method for controlling a robot based on an interaction with a user, comprising: a reading step of reading a function list from a function list storage unit storing a function list of the predetermined plurality of operations to be executed by the robot, each of the predetermined plurality of operations being defined as a function module; an input step of receiving an instruction from the user to the robot; a storing step of storing the instruction data inputted in the input step in an input buffer; a processing step of transmitting a prompt including the function list and the data read from the input buffer to a natural language processing system utilizing a large-scale language model, receiving a response from the natural language processing system indicating which function module in the function list should be executed, and executing the function module identified by the response; an output step of outputting a response to the instruction to the user; Equipped with In the processing step, if the functional module cannot be executed normally, information about the current state of the object being operated by the robot is stored in the input buffer.

11. A control program for controlling a robot based on an interaction with a user, On the computer, a reading step of reading a function list from a function list storage unit storing a function list of the predetermined plurality of operations to be executed by the robot, each of the predetermined plurality of operations being defined as a function module; an input step of receiving an instruction from the user to the robot; a storing step of storing the instruction data inputted in the input step in an input buffer; a processing step of transmitting a prompt including the function list and the data read from the input buffer to a natural language processing system utilizing a large-scale language model, receiving a response from the natural language processing system indicating which function module in the function list should be executed, and executing the function module identified by the response; an output step of outputting a response to the instruction to the user; Execute If the functional module cannot be executed normally in the processing step, the control program stores information about the current state of the object being operated by the robot in the input buffer.

Images