Robot control device

Abstract

A robot control device according to one embodiment comprises: a plurality of command generation units that generate a command for executing a task; a prediction unit that predicts the generation time of a command in the plurality of command generation units; and a schedule unit that selects a command generation unit to generate a command from the plurality of command generation units on the basis of the predicted generation time and a constraint imposed on the task, and adjusts the content of an instruction to the selected command generation unit. Each of the schedule unit, the plurality of command generation units, and the prediction unit is constituted by an LLM.

Description

Robot control device

[0001] The embodiment relates to a robot control device.

[0002] Robot control technologies that utilize foundation models have been proposed. Known examples of foundation models include large language models (LLMs) and vision language models (VLMs).

[0003] For example, Non-Patent Document 1 discloses generating a manipulator's motion trajectory without prior learning (zero shot) by generating a code for generating a three-dimensional affordance map using LLM.

[0004] Furthermore, Non-Patent Document 2 discloses that by using VLM to express output movements in text format, human instructions and video information are used as inputs to generate end-to-end movements on the robot side.

[0005] Huang, Wenlong, Chen Wang, Ruohan Zhang, Yunzhu Li, Jiajun Wu, and Li Fei-Fei. 2023. “VoxPoser: Composable 3D Value Maps for Robotic Manipulation with Language Models.”, 7th Conference on Robot Learning (CoRL 2023)Anthony Brohan, et.al., “RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic “Response Length Perception and Sequence Scheduling: An LLM-Empowered LLM Inference Pipeline”, 37th Conference on Nueral Information Processing Systems (NeurIPS 2023)

[0006] In the case of a robot that works in cooperation with humans, the actions to be executed may be subject to time constraints. In this case, in order to operate the robot at the appropriate timing, it is necessary to generate the robot's actions while satisfying the time constraints. However, in the above example, it is difficult to generate the robot's actions while taking the time constraints into consideration.

[0007] The present invention has been made in view of the above circumstances, and an object of the present invention is to provide a robot control device that generates robot movements taking into account time constraints.

[0008] A robot control device according to one aspect includes a plurality of command generating units that generate commands for executing a task, a prediction unit that predicts a generation time of the commands in the plurality of command generating units, and a scheduling unit that selects a command generating unit from the plurality of command generating units to generate a command based on the predicted generation time and constraints imposed on the task, and adjusts instructions to the selected command generating unit. Each of the scheduling unit, the plurality of command generating units, and the prediction unit is configured using a large language model (LLM).

[0009] According to the embodiment, it is possible to provide a robot control device that generates robot operations taking time constraints into consideration.

[0010] FIG. 1 is a block diagram showing an example of the configuration of a robot system according to a first embodiment. FIG. 2 is a block diagram showing an example of the hardware configuration of a robot control device according to the first embodiment. FIG. 3 is a block diagram showing an example of the functional configuration of the robot control device according to the first embodiment. FIG. 4 is a flowchart showing an example of a series of processes in a base model provided in the robot control device according to the first embodiment. FIG. 5 is a block diagram showing an example of the functional configuration of a robot control device according to a second embodiment. FIG. 6 is a flowchart showing an example of a series of processes in a base model provided in the robot control device according to the second embodiment. FIG. 7 is a block diagram showing an example of the functional configuration of a robot control device according to a modified example.

[0011] Hereinafter, embodiments will be described with reference to the drawings. In the following description, components having the same functions and configurations will be given the same reference numerals.

[0012] 1. First Embodiment 1.1 Configuration 1.1.1 Robot System FIG. 1 is a block diagram showing an example of the configuration of a robot system according to the first embodiment. The robot system 1 is a collaborative robot that works in cooperation with humans. More specifically, the robot system 1 includes a service robot that requires control taking into account the timing of its actions, such as handing water to an approaching person. The robot system 1 may also be configured to operate autonomously without receiving instructions from a human. As shown in FIG. 1 , the robot system 1 includes a manipulator 2, a sensor 3, a microphone 4, a speaker 5, and a robot control device 6.

[0013] The manipulator 2 is a driving mechanism of the robot system 1. The manipulator 2 constitutes, for example, a robot arm or the like, and physically executes a task given by a person (or set autonomously). The manipulator 2 also outputs state quantities such as the positions and velocities of representative points such as joints that constitute the manipulator 2 as operation information.

[0014] The sensor 3 is a sensor used for spatial recognition, such as a video sensor or a point cloud sensor. The video sensor includes, for example, an RGB-D camera and is configured to output color information and depth information of surrounding objects. The point cloud sensor includes, for example, a LiDAR (light detection and ranging) and is configured to output point cloud information in which each point has distance information. The sensor 3 outputs spatial information including a set of color information and depth information, or point cloud information.

[0015] The microphone 4 is configured to acquire information such as tasks given by a person, and outputs input voice as voice information.

[0016] The speaker 5 outputs communication occurring between the user and the person when the task is being performed as human-recognizable voice.

[0017] The robot control device 6 corresponds to the brain of the robot system 1. The robot control device 6 controls the output from the manipulator 2 and the speaker 5 based on various information obtained from the manipulator 2, the sensor 3, and the microphone 4.

[0018] 1.1.2 Robot Control Device FIG. 2 is a block diagram showing an example of the hardware configuration of the robot control device according to the first embodiment.

[0019] As shown in FIG. 2 , the robot control device 6 includes, for example, a control circuit 11 , a storage 12 , a communication module 13 , a drive 14 , and a storage medium 15 .

[0020] The control circuit 11 is a circuit that controls the overall components of the robot control device 6. The control circuit 11 includes a CPU (central processing unit), RAM (random access memory), ROM (read only memory), etc. The CPU of the control circuit 11 controls the entire robot control device 6 in accordance with the programs stored in the ROM of the control circuit 11. The RAM of the control circuit 11 has a working area for the CPU of the control circuit 11. The ROM of the control circuit 11 stores programs and the like used by the robot control device 6.

[0021] The storage 12 includes, for example, a hard disk drive (HDD) or a solid state drive (SSD). The storage 12 stores information used in various processes in the robot control device 6.

[0022] The communication module 13 is a circuit used for transmitting and receiving data between the manipulator 2, the sensor 3, the microphone 4, and the speaker 5 and the robot control device 6. The communication module 13 may be configured to connect the robot control device 6 to a network (not shown).

[0023] The drive 14 is a device for reading software stored in the storage medium 15. The drive 14 includes, for example, a CD (Compact Disk) drive or a DVD (Digital Versatile Disk) drive.

[0024] The storage medium 15 is a medium that stores software electrically, magnetically, optically, mechanically, or chemically. The storage medium 15 may store a program used by the robot control device 6.

[0025] FIG. 3 is a block diagram illustrating an example of the functional configuration of the robot control device according to the first embodiment.

[0026] 3 , the robot control device 6 functions as a computer including a motion information acquisition unit 21, a spatial information acquisition unit 22, a voice information acquisition unit 23, a base model 24, a command execution unit 25, and a voice synthesis unit 26. The base model 24 corresponds to the LLM portion of the robot control device 6. The motion information acquisition unit 21, the spatial information acquisition unit 22, the voice information acquisition unit 23, the command execution unit 25, and the voice synthesis unit 26 correspond to the non-LLM portion of the robot control device 6.

[0027] The motion information acquisition unit 21 acquires, as motion information, state quantities such as the position and velocity of the representative point output from the manipulator 2. The motion information acquisition unit 21 outputs the acquired motion information to the base model 24.

[0028] The spatial information acquisition unit 22 acquires the spatial information output from the sensor 3. The spatial information acquisition unit 22 outputs the acquired spatial information to the base model 24.

[0029] The voice information acquisition unit 23 acquires voice information output from the microphone 4. The voice information acquisition unit 23 outputs the acquired voice information to the base model 24.

[0030] The base model 24 includes an information integration unit 31, a schedule unit 32, a prediction unit 33, multiple command generation units 35, and a request statement generation unit 36. The base model 24 also stores a task execution record DB 34. In the example of Fig. 3, the multiple command generation units 35 include N command generation units 35-1, ..., and 35-N (N is an integer).

[0031] The information integration unit 31 aggregates information to be taken into consideration when generating a prompt. Specifically, the information integration unit 31 generates integrated information by integrating the motion information input from the motion information acquisition unit 21, the spatial information input from the spatial information acquisition unit 22, and the audio information input from the audio information acquisition unit 23. The information integration unit 31 outputs the integrated information to the schedule unit 32.

[0032] The scheduler 32 is an LLM that generates prompts based on the integrated information. Specifically, the scheduler 32 generates prompts PAs based mainly on the audio information in the integrated information. The prompts PAs are, for example, text data describing tasks to be executed. When generating the prompts PAs, the scheduler 32 does not need to consider time constraints on the tasks to be executed. The scheduler 32 outputs the generated prompts PAs to the prediction unit 33.

[0033] Furthermore, the scheduling unit 32 selects a command generating unit 35 that instructs the selected command generating unit 35 to generate a command from among the multiple command generating units 35-1 to 35-N based on the integrated information and the command generation time prediction results for each of the multiple command generating units 35-1 to 35-N input from the prediction unit 33. The scheduling unit 32 then generates a prompt PB for instructing the selected command generating unit 35 to generate a command. The prompt PB is, for example, text data in which the content of the prompt PA and the level of detail of the conditions have been changed. When generating the prompt PB, the scheduling unit 32 adjusts the content and level of detail of the prompt PA by taking into account the time constraints for the task to be executed, the command generation time, the operating speed of the manipulator 2, and the like. Furthermore, in conjunction with the generation of the prompt PB, the scheduling unit 32 further generates a prompt PC if communication between the robot system 1 and a person collaborating with the robot system 1 is required. The prompt PC is, for example, text data instructing the generation of voice data to be communicated to a person during collaborative operation. The scheduling unit 32 outputs the generated prompt PB to a command generating unit 35 selected from the plurality of command generating units 35-1 to 35-N. The scheduling unit 32 also outputs the generated prompt PC to the request sentence generating unit 36.

[0034] The prediction unit 33 is an LLM that predicts the generation time of a command in each of the multiple command generation units 35-1 to 35-N. When the prediction unit 33 receives a prompt PA from the schedule unit 32, it references the task execution record DB 34 and searches for a task similar to the prompt PA. Then, the prediction unit 33 predicts the generation time of a command to be generated based on the prompt PA for each of the command generation units 35-1 to 35-N, based on the execution time of the command generated for the task similar to the prompt PA.

[0035] The task execution record DB 34 is, for example, a database that aggregates information about tasks that have been executed in the past using the base model 24. The task execution record DB 34 stores tasks that have been executed in the past and the execution time for each task.

[0036] The prediction unit 33 may include a predictor capable of making predictions within a fixed time, such as a "Transformer" that predicts generation times using sentences that set similar tasks in the past as input, or a regression model that predicts calculation times using feature quantities such as the similarity of the sentences that set tasks and sentence length as explanatory variables. This allows the prediction unit 33 to predict command generation times within a negligibly short time. The prediction unit 33 outputs the predicted command generation times for each of the generated command generation units 35-1 to 35-N to the schedule unit 32.

[0037] The multiple command generation units 35-1 to 35-N are LLMs that generate commands for the manipulator 2 to execute tasks based on prompts PB input from the schedule unit 32. Of the multiple command generation units 35-1 to 35-N, those that receive prompts PB from the schedule unit 32 generate commands based on the prompts PB. Specifically, for example, based on the prompt PB generated in accordance with the task of "putting a cup on a desk," the command execution unit 25 generates a series of commands such as "Pick Cup," "Move Cup to Desk," and "Place Cup on Desk." On the other hand, among the multiple command generation units 35-1 to 35-N, those that do not receive prompts PB from the schedule unit 32 wait without generating commands. The commands generated by the command generation units 35 that receive prompts PB are output to the command execution unit 25.

[0038] The multiple command generators 35-1 to 35-N may have different characteristics. Specifically, for example, one of the multiple command generators 35-1 to 35-N may be the "VoxPoser" disclosed in Non-Patent Document 1. Furthermore, for example, another of the multiple command generators 35-1 to 35-N may be the "RT-2" disclosed in Non-Patent Document 2. By having different characteristics, the multiple command generators 35-1 to 35-N may generate commands of different lengths (i.e., levels of detail) for the same task. Furthermore, the multiple command generators 35-1 to 35-N may generate commands for the same task at different times.

[0039] The request sentence generation unit 36 ​​is an LLM for generating a sentence (request sentence) indicating information to be conveyed to a person by voice, based on the prompt PC input from the schedule unit 32. The request sentence generation unit 36 ​​outputs the generated request sentence to the voice synthesis unit 26.

[0040] As described above, the scheduler 32, the predictor 33, the command generators 35-1 to 35-N, and the request generator 36 constitute a hierarchical LLM system.

[0041] When the command execution unit 25 receives a command from any one of the plurality of command generation units 35-1 to 35-N, it executes the command. That is, the command execution unit 25 controls the operation of the manipulator 2 in accordance with the command.

[0042] When the voice synthesis unit 26 receives the request sentence from the request sentence generation unit 36, it converts the request sentence into voice. The voice synthesis unit 26 outputs the converted voice from the speaker 5.

[0043] 1.2 Operation Next, the operation of the robot control device according to the first embodiment will be described.

[0044] FIG. 4 is a flowchart showing an example of a series of processes in the base model provided in the robot control device according to the first embodiment.

[0045] 4, when the manipulator 2, the sensor 3, and the microphone 4 respectively output motion information, spatial information, and audio information (start), the information integration unit 31 integrates the motion information, spatial information, and audio information (S11). The integrated information includes information such as the motion speed of the manipulator 2, the task to be executed, and constraints on executing the task. The constraints include constraints on the time required to execute the task. The information integration unit 31 outputs the integrated information to the schedule unit 32.

[0046] The scheduling unit 32 generates a prompt PA corresponding to the task based on the integrated information generated in the process of S11 (S12). The scheduling unit 32 outputs the generated prompt PA to the prediction unit 33.

[0047] The prediction unit 33 predicts the generation time of a command in each of the plurality of command generation units 35-1 to 35-N based on the prompt PA generated in the process of S12 (S13). Specifically, when predicting the generation time of a command, the prediction unit 33 refers to past task execution records stored in the task execution record DB 34 and searches for similar tasks. Based on the execution times of similar tasks, the prediction unit 33 calculates the generation time of a command that each of the plurality of command generation units 35-1 to 35-N will generate using the prompt PA as input. The prediction unit 33 outputs the predicted command generation time to the schedule unit 32.

[0048] The scheduling unit 32 selects one command generation unit 35 from the plurality of command generation units 35-1 to 35-N based on the predicted generation time calculated in the process of S13, the movement speed of the manipulator 2 included in the integrated information generated in the process of S11, and the time constraint (S14). For example, the scheduling unit 32 selects a command generation unit 35 that can complete command generation and command execution (i.e., control of the manipulator 2) within the time constraint. Specifically, for example, in the task of "placing a cup on a desk," if a person is quickly approaching the desk on which the cup is placed, the scheduling unit 32 may select a command generation unit 35 configured to reuse a previously prepared action set so that the manipulator 2 can move quickly. Alternatively, if a person is slowly approaching the desk on which the cup is placed, the scheduling unit 32 may select a command generation unit 35 configured to take into account the trajectory of the manipulator 2 so that the manipulator 2 can move safely. If there are multiple command generators 35 that satisfy the above conditions, the scheduler 32 may select the command generator 35 that generates the command with the highest performance from among the command generators 35 that satisfy the conditions.

[0049] The scheduling unit 32 adjusts the prompt PA generated in the process of S12 based on the predicted generation time calculated in the process of S13, the movement speed of the manipulator 2 included in the integrated information generated in the process of S11, and the time constraints (S15). For example, in the situation described above where a person is slowly approaching a desk on which a cup is placed, the scheduling unit 32 adds an additional instruction to the prompt PA, such as "generate a slow trajectory taking safety into consideration." The scheduling unit 32 outputs the prompt PB obtained by the adjustment in the process of S15 to the command generating unit 35 selected in the process of S14.

[0050] Upon receiving the prompt PB, the command generating unit 35 generates a command based on the prompt PB (S16).

[0051] Then, the command generating unit 35 outputs the command generated in the process of S16 to the command executing unit 25 (S17). The command executing unit 25 controls the manipulator 2 in accordance with the command.

[0052] When the processing of S17 is completed, the series of processing in the base model 24 is completed (end).

[0053] 1.3 Effects of the First Embodiment According to the first embodiment, each of the multiple command generators 35-1 to 35-N generates a command for executing a task. The prediction unit 33 predicts the command generation time of the multiple command generators 35-1 to 35-N. The scheduler 32 selects a command generator 35 from the multiple command generators 35-1 to 35-N that will generate a command based on the predicted generation time and the constraints imposed on the task. Each of the scheduler 32, the multiple command generators 35-1 to 35-N, and the prediction unit 33 is configured using LLM. This allows the robot controller 6 to consider the time required for command generation when executing a task. Therefore, if a task has a time constraint, the robot controller 6 can select a command generator 35 that can generate commands quickly within a range that satisfies the constraint. Furthermore, if a task does not have a time constraint, the robot controller 6 can select a command generator 35 that can generate commands that enable more cooperative actions.

[0054] Furthermore, the prediction unit 33 references the task execution record DB 34 to predict the command generation times of the multiple command generation units 35-1 to 35-N based on the execution times of past tasks. This improves the accuracy of command generation time predictions when similar tasks have been executed in the past. This allows the schedule unit 32 to select a more appropriate command generation unit 35.

[0055] 2. Second Embodiment Next, a second embodiment will be described. The second embodiment differs from the first embodiment in that after instructing a plurality of command generating units to generate a command, a command generating unit that is to output the command is selected. The following mainly describes configurations and operations that differ from the first embodiment. Descriptions of configurations and operations that are equivalent to those of the first embodiment will be omitted as appropriate.

[0056] 2.1 Configuration Fig. 5 is a block diagram showing an example of the functional configuration of a robot control device according to the second embodiment. Fig. 5 corresponds to Fig. 3 in the first embodiment.

[0057] As shown in FIG. 5, the robot control device 6 includes a motion information acquisition unit 21, a spatial information acquisition unit 22, a voice information acquisition unit 23, a base model 24, a command execution unit 25, and a voice synthesis unit 26.

[0058] The configurations of the motion information acquisition unit 21, the spatial information acquisition unit 22, the voice information acquisition unit 23, the command execution unit 25, and the voice synthesis unit 26 are the same as those in the first embodiment.

[0059] The base model 24 includes an information integration unit 41, a schedule unit 42, a prediction unit 43, a plurality of command generation units 45, and a request statement generation unit 46. In the example of Fig. 5, the plurality of command generation units 45 include N command generation units 45-1, ..., and 45-N (N is an integer).

[0060] The information integration unit 41 aggregates information to be taken into consideration when generating a prompt. Specifically, the information integration unit 41 generates integrated information by integrating the motion information input from the motion information acquisition unit 21, the spatial information input from the spatial information acquisition unit 22, and the audio information input from the audio information acquisition unit 23. The information integration unit 41 outputs the integrated information to the schedule unit 42.

[0061] The scheduler 42 is an LLM that generates prompts based on the integration information. Specifically, the scheduler 42 generates prompts PA based on the integration information. The scheduler 42 outputs the generated prompts PA to each of the multiple command generators 45-1 to 45-N.

[0062] Furthermore, the scheduling unit 42 receives from the prediction unit 43 a prediction result of the generation time required for each of the plurality of command generating units 45-1 to 45-N to generate a command based on the prompt PA. Based on the prediction result, the scheduling unit 42 selects a command generating unit 45 that is to output a command from among the plurality of command generating units 45-1 to 45-N that are currently generating a command. The scheduling unit 42 sends a command output instruction to the selected command generating unit 45. The scheduling unit 42 may also send a command output stop instruction to the command generating units 45 that were not selected.

[0063] In addition, in conjunction with the generation of the prompt PA, the scheduler 42 further generates a prompt PC when communication is required between the robot system 1 and a person collaborating with the robot system 1. The scheduler 42 outputs the generated prompt PC to the request sentence generator 46.

[0064] The prediction unit 43 is an LLM that predicts the generation time of a command in each of the multiple command generation units 45-1 to 45-N. The prediction unit 43 receives the predicted number of characters in the command based on the prompt PA from each of the multiple command generation units 45-1 to 45-N. The prediction unit 43 predicts the generation time of the command generated by each of the multiple command generation units 45-1 to 45-N based on the predicted number of characters in the command. The prediction unit 43 outputs the predicted command generation time to the schedule unit 42. Note that the predictor applied to the prediction unit 43 may be a regression model that uses, as explanatory variables, the average generation speed of each LLM in the past, the clock speed of the GPU being used, etc.

[0065] The multiple command generation units 45-1 to 45-N are LLMs that generate commands for the manipulator 2 to execute tasks based on the prompt PA input from the schedule unit 42. Each of the multiple command generation units 45-1 to 45-N starts generating a command based on the prompt PA input from the schedule unit 32. Of the multiple command generation units 45-1 to 45-N, a command generation unit 45 that receives a command output instruction from the schedule unit 42 outputs the generated command to the command execution unit 25. On the other hand, of the multiple command generation units 45-1 to 45-N, a command generation unit 45 that does not receive a command output instruction from the schedule unit 42 waits without outputting the generated command to the command execution unit 25. Note that when the schedule unit 42 receives a command output stop instruction, the command generation unit 45 that received the command output stop instruction may stop generating the command midway.

[0066] The request sentence generation unit 46 is an LLM for generating a sentence (request sentence) indicating information to be conveyed to a person by voice, based on the prompt PC input from the schedule unit 42. The request sentence generation unit 46 outputs the generated request sentence to the voice synthesis unit 26.

[0067] As described above, the scheduler 42, the predictor 43, the command generators 45-1 to 45-N, and the request generator 46 constitute a hierarchical LLM system.

[0068] 2.2 Operation Next, the operation of the robot control device according to the second embodiment will be described.

[0069] Fig. 6 is a flowchart showing an example of a series of processes in a base model provided in a robot control device according to the second embodiment. Fig. 6 corresponds to Fig. 4 in the second embodiment.

[0070] 6, when the manipulator 2, the sensor 3, and the microphone 4 respectively output motion information, spatial information, and audio information (start), the information integration unit 41 integrates the motion information, spatial information, and audio information (S21). The integrated information includes information such as the motion speed of the manipulator 2, the task to be executed, and constraints on executing the task. The constraints include constraints on the time required to execute the task. The information integration unit 41 outputs the integrated information obtained by the integration to the schedule unit 42.

[0071] The scheduling unit 42 generates a prompt PA corresponding to the task based on the integrated information generated in the process of S21 (S22). The scheduling unit 42 outputs the generated prompt PA to the plurality of command generating units 45-1 to 45-N.

[0072] Each of the plurality of command generating units 45-1 to 45-N starts generating a command based on the prompt PA generated in the process of S22 (S23).

[0073] After the process of S23, each of the plurality of command generation units 45-1 to 45-N predicts the number of characters in the command being generated (S24). Each of the plurality of command generation units 45-1 to 45-N outputs the predicted result of the number of characters in the command being generated to the prediction unit 43.

[0074] The prediction unit 43 predicts the generation time of the command in each of the plurality of command generation units 45-1 to 45-N based on the number of characters of the command predicted in the process of S24 (S25). The prediction unit 43 outputs the predicted command generation time to the schedule unit 42 as a prediction result.

[0075] The scheduling unit 42 selects one command generating unit 45 from the plurality of command generating units 45-1 to 45-N based on the predicted generation time calculated in the process of S25, the movement speed of the manipulator 2 included in the integrated information generated in the process of S21, and the time constraints (S26). The scheduling unit 42 sends a command output instruction to the selected command generating unit 45.

[0076] The command generation unit 45 that has received the command output instruction completes the generation of the command that was started in the process of S23 (S27). Note that the command generation unit 45 that has not received the command output instruction may or may not complete the generation of the command that was started in the process of S23.

[0077] Then, the command generating unit 45 that has received the command output instruction outputs the command generated in the process of S27 to the command executing unit 25 (S28). The command executing unit 25 controls the manipulator 2 in accordance with the command.

[0078] When the process of S28 is completed, the series of processes in the base model 24 is completed (end).

[0079] 2.3 Effects of the Second Embodiment According to the second embodiment, each of the multiple command generation units 35-1 to 35-N predicts the number of characters in a command for executing a task. The prediction unit 33 predicts the generation time of the command in the multiple command generation units 35-1 to 35-N based on the predicted number of characters. This improves the accuracy of prediction of the command generation time even when a similar task has not been executed in the past. This allows the schedule unit 32 to select a more appropriate command generation unit 35.

[0080] 3. Modifications, etc. Various modifications can be applied to the first and second embodiments described above.

[0081] In the first and second embodiments described above, the base model 24 outputs a series of commands for a task to the command execution unit 25 in a lump, but this is not limited to this. For example, the base model 24 may separately include an LLM that generates a command related to an initial operation among the series of commands for a task, and multiple LLMs that generate commands related to subsequent operations. The base model 24 may then generate a command related to the initial operation and output it to the command execution unit 25 while selecting one of the multiple LLMs that generate commands related to subsequent operations.

[0082] 7 is a block diagram showing an example of the functional configuration of a robot control device according to a modified example, in which the modified example is applied to the first embodiment.

[0083] As shown in FIG. 7, the plurality of command generators 35 may include an initial command generator 35-0 and N command generators 35-1, . . . , and 35-N.

[0084] The schedule unit 32 generates a prompt PA and also generates a prompt PD. The prompt PD is text data for instructing the execution of a basic initial operation that is executed at the beginning of any operation. The prompt PD may be a part of the prompt PA. The schedule unit 32 outputs the prompt PD to the initial command generation unit 35-0 before selecting the command generation unit 35 that will output the command.

[0085] Furthermore, the scheduling unit 32 selects a command generating unit 35 to instruct the generation of a command from among the plurality of command generating units 35-1 to 35-N, based on the integration information and the results of prediction of the command generation time in each of the plurality of command generating units 35-1 to 35-N input from the prediction unit 33. Then, the scheduling unit 32 generates a prompt PB for instructing the generation of a command, and outputs it to the selected command generating unit 35.

[0086] The initial command generator 35-0 is an LLM assigned to generate commands related to basic initial operations that are executed at the beginning of any operation. The initial command generator 35-0 generates commands related to initial operations based on the prompt PD. When the initial command generator 35-0 completes generating a command, it outputs the generated command to the command execution unit 25, regardless of whether the generation of subsequent commands has been completed.

[0087] The plurality of command generating units 35-1 to 35-N are LLMs assigned to generate commands related to subsequent operations following an initial operation. The command generating units 35 receive a prompt PB from the schedule unit 32 and generate commands based on the prompt PB. When the command generating units 35 complete command generation, they output the generated commands to the command executing unit 25.

[0088] With the above configuration, the initial operation can be executed while the LLM that generates the command for the subsequent operation is being selected, thereby improving the response speed of the manipulator 2.

[0089] Moreover, the first and second embodiments described above may be combined.

[0090] For example, in the second embodiment, when outputting a prompt PA to multiple command generation units 45-1 to 45-N, the scheduling unit 42 may select in advance candidate command generation units 45 that will output the prompt. When selecting candidate command generation units 45 that will output the prompt, the prediction unit 43 may predict command generation times based on past task execution times by referencing the task execution record DB 34. Then, the scheduling unit 42 may select candidate command generation units 45 that will output the prompt based on the prediction results.

[0091] With the above configuration, it is possible to select an appropriate LLM with high prediction accuracy both when executing a task similar to a past task and when executing a task that has never been seen before.

[0092] The present invention is not limited to the above-described embodiments, and various modifications can be made in the implementation stage without departing from the spirit of the invention. Furthermore, the embodiments may be implemented in appropriate combinations, in which case the combined effects can be obtained. Furthermore, the above-described embodiments include various inventions, and various inventions can be extracted by combining selected elements from the disclosed elements. For example, if the problem can be solved and the desired effect can be obtained even if some elements are deleted from all elements shown in the embodiments, the configuration from which these elements are deleted can be extracted as an invention.

[0093] REFERENCE SIGNS LIST 1...Robot system 2...Manipulator 3...Sensor 4...Microphone 5...Speaker 11...Control circuit 12...Storage 13...Communication module 14...Drive 15...Storage medium 21...Motion information acquisition unit 22...Spatial information acquisition unit 23...Speech information acquisition unit 24...Base model 25...Command execution unit 26...Speech synthesis unit 31, 41...Information integration unit 32, 42...Schedule unit 33, 43...Prediction unit 34...Task execution record DB 35-0...Initial command generation unit 35-1 to 35-N, 45-1 to 45-N...Command generation unit 36...Request sentence generation unit

Claims

1. A robot control device comprising: a plurality of command generation units that generate commands for executing a task; a prediction unit that predicts the generation time of the commands in the plurality of command generation units; and a scheduling unit that selects a command generation unit from the plurality of command generation units to generate a command based on the predicted generation time and constraints imposed on the task, and adjusts the instructions to the selected command generation unit, wherein each of the scheduling unit, the plurality of command generation units, and the prediction unit is configured using an LLM (large language model).

2. The robot control device according to claim 1, wherein the prediction unit predicts command generation times in the plurality of command generation units based on past task execution times.

3. A robot control device as described in claim 1, wherein each of the plurality of command generation units predicts the number of characters in the command for executing the task, and the prediction unit predicts the generation time of the command in the plurality of command generation units based on the predicted number of characters.

4. A robot control device as described in claim 1, wherein the plurality of command generation units include a first command generation unit that generates a first command for executing an initial operation of the task, and a plurality of second command generation units that generate second commands for executing an operation of the task subsequent to the initial operation, the prediction unit predicts a generation time of the second command in the plurality of second command generation units, and the schedule unit selects a second command generation unit that will generate the second command from the plurality of second command generation units based on the predicted generation time and a constraint imposed on the task, and instructs the first command generation unit to generate the first command prior to the selection.

Images