Control system, control method, and program
The control system uses generative models to generate conversational text between robots and users, addressing the lack of natural language confirmation in existing systems, allowing easy task management and improved user understanding of robot processes.
Patent Information
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- NAGOYA ELECTRICAL EDUCATIONAL FOUNDATION
- Filing Date
- 2025-01-10
- Publication Date
- 2026-07-23
AI Technical Summary
Existing systems for controlling multiple robots using generative models lack the ability to confirm the process and results in natural language, necessitating specialized knowledge for task assignment and decomposition.
A control system and method that utilizes first and second generative models to acquire robot information and control instructions, generate control plans, and produce conversational text between robots and users, enabling natural language confirmation of processes and results.
Enables users to easily confirm and modify task decomposition and allocation by displaying conversational text, enhancing efficiency and understanding of robot cooperation.
Smart Images
Figure 2026121195000001_ABST
Abstract
Description
Technical Field
[0004] ,
[0006] , , , , , ,
[0005] , , , , ,
[0001] The present disclosure relates to a control system, a control method, and a program for controlling a plurality of robots.
Background Art
[0002] There are known a plurality of robots that understand and execute task instructions using a generative model such as a large language model (LLM) generated by machine learning or the like (see, for example, Non-Patent Document 1).
Prior Art Documents
Non-Patent Documents
[0003]
Non-Patent Document 1
Summary of the Invention
Problems to be Solved by the Invention
[0004] As described above, when a plurality of robots cooperate to execute a task using a generative model such as a large language model generated by machine learning or the like, specific control is assigned to each robot, but there is a desire to confirm the process and results in natural language.
[0005] The present disclosure has been made to solve such problems, and when a plurality of robots cooperate to execute a task and specific control is assigned to each robot, the main object is to provide a control system, a control method, and a program that allow a user to confirm the process and results in natural language.
Means for Solving the Problems
[0006] One aspect of this disclosure for achieving the above objectives is: A first acquisition means for acquiring robot information about multiple robots, A second acquisition means for acquiring control instructions for the plurality of robots, A control plan generation means generates a control plan for each robot, based on the robot information of each robot acquired by the first acquisition means and the control instructions acquired by the second acquisition means, using a first generation model, for each robot to coordinately execute the control instructions. A conversational text generation means generates conversational text between each robot and conversational text between each robot and the user, using a second generation model, based on the control plan generated by the control plan generation means, An output means for outputting the conversation text generated by the conversation text generation means, Equipped with, control system That is the case. On this flight, The first and second generative models may consist of large-scale language models. On this flight, The robot information may include at least one of the following: robot performance, robot status, robot history, robot position, and robot function. One aspect of this disclosure for achieving the above objectives is: Steps to obtain robot information about multiple robots, The steps include: obtaining control instructions for the plurality of robots; Based on the robot information of each robot acquired and the control instructions acquired, a first generative model is used to generate a control plan for each robot in which each robot can cooperate to execute the control instructions. Based on the generated control plan, the step of generating conversational text between the robots and conversational text between the robots and the user, using a second generative model, for each robot to execute the control plan; The steps include outputting the generated conversation text, including, Control method That is the case. One aspect of this disclosure for achieving the above objectives is: A process for acquiring robot information about multiple robots, A process for obtaining control instructions for the aforementioned multiple robots, A process to generate a control plan for each robot, using a first generative model, based on the robot information of each robot acquired and the control instructions acquired, so that each robot can cooperate in executing the control instructions. Based on the generated control plan, a process is performed to generate conversational text between the robots and conversational text between the robots and the user, using a second generative model, for each robot to execute the control plan. The process of outputting the generated conversation text, Make the computer execute it. program That is the case. One aspect of this disclosure for achieving the above objectives is: It comprises multiple control units, each associated with a different robot, and each unit issuing control instructions to the associated robot. Each control unit interacts with a common generative model in response to a predetermined control instruction, and based on the result of that interaction, issues control instructions to the corresponding robots, thereby coordinating the execution of the predetermined control instruction. control system That is the case. [Effects of the Invention]
[0007] According to this disclosure, it is possible to provide a control system, control method, and program that allow a user to verify the process and results in natural language when multiple robots cooperate to perform a task and specific controls are assigned to each robot. [Brief explanation of the drawing]
[0008] [Figure 1]It is a block diagram showing a schematic hardware configuration of a control system according to the present embodiment. [Figure 2] It is a block diagram showing a schematic system configuration of a control system according to the present embodiment. [Figure 3] It is a flowchart showing an example of a flow of a control method according to the present embodiment. [Figure 4] It is a block diagram showing a schematic system configuration of a control system according to the present embodiment.
Mode for Carrying Out the Invention
[0009] Embodiment 1 Hereinafter, the present embodiment will be described with reference to the drawings. For example, when a plurality of robots cooperate to execute a task according to an instruction from a user, conventionally, the user himself / herself has to assign specific operations to each robot from the perspective of the entire task and needs to have specialized knowledge for that. Also, an autonomous distributed system in which robots that operate autonomously execute a task through mutual communication, mediation, agreement, etc. (for example, non-patent literature: https: / / project-roco.github.io / ) is conceivable. However, if the states of all robots can be grasped through communication, it is often more efficient to perform task allocation uniformly as a whole.
[0010] On the other hand, the control system according to the present embodiment can perform task decomposition and allocation according to the states and characteristics of each robot in response to instructions from a user or the like, and furthermore, can confirm the process and results in the form of a natural language conversation. And since the user can easily confirm the process and results in the form of a natural language conversation, for example, it is possible to easily modify or change task decomposition and allocation.
[0011] FIG. 1 is a block diagram showing a schematic hardware configuration of a control system according to the present embodiment.
[0012] The control system 1 has a typical computer hardware configuration, including, for example, a processor 11 such as a CPU (Central Processing Unit) or GPU (Graphics Processing Unit), internal memory 12 such as RAM (Random Access Memory) or ROM (Read Only Memory), storage devices 13 such as an HDD (Hard Disk Drive) or SSD (Solid State Drive), input / output I / F 14 for connecting peripheral devices such as a display, and communication I / F 15 for communicating with devices outside the device.
[0013] The control system 1 is configured, for example, as a server device. The control system 1 and each robot 16 may transmit and receive signals using wireless communication such as Bluetooth® or Wi-Fi®.
[0014] In this embodiment, the control system 1, in its first step, generates a control plan for each robot 16 to execute the control instructions in cooperation with each robot 16, based on the control instructions for each robot 16 and the robot information of each robot 16. Then, in its second step, the control system 1 uses the generated control plans for each robot 16 to generate conversation text between each robot 16 and conversation text between each robot 16 and the user, and displays these to the user.
[0015] In this way, by efficiently generating and displaying the above conversational text in two stages, when multiple robots 16 cooperate to perform a task and specific controls are assigned to each robot 16, the user can confirm the process and results in natural language.
[0016] Figure 2 is a block diagram illustrating the schematic system configuration of the control system according to this embodiment. The control system 1 according to this embodiment includes a first acquisition unit 2, a second acquisition unit 3, a control plan generation unit 4, a dialogue generation unit 5, and a display unit 6.
[0017] The first acquisition unit 2, the second acquisition unit 3, the control plan generation unit 4, the conversation text generation unit 5, and the display unit 6 may be provided on a specific robot 16, or they may be distributed among multiple robots 16.
[0018] The first acquisition unit 2 is a specific example of the first acquisition means. The first acquisition unit 2 acquires robot information relating to a plurality of robots 16. The robot information includes information such as the performance of the robot 16, the state of the robot 16, the history of the robot 16, the position of the robot 16, and the functions of the robot 16. The first acquisition unit 2 may also acquire robot information for each robot 16 from each robot 16.
[0019] Furthermore, robot 16 includes any robot that performs various types of control in response to control instructions, such as mobile robots, grasping robots, arm robots, home appliance robots (vacuum cleaners, refrigerators, washing machines, lighting fixtures, etc.), terminal robots, and device robots.
[0020] The second acquisition unit 3 is a specific example of the second acquisition means. The second acquisition unit 3 acquires control instructions for multiple robots 16.
[0021] The second acquisition unit 3 may, for example, acquire voice commands from the user as control commands using microphones provided on each robot 16. The control commands may be pre-set in a storage unit or the like. The storage unit may consist of the storage device 13 or the like. The second acquisition unit 3 can acquire control commands from the storage unit.
[0022] The second acquisition unit 3 may acquire data, for example, by generating control instructions in response to sensor values acquired by the robot 16's sensors, specific actions of the robot 16, or certain environmental conditions. The control instructions include, for example, instructions for the robot 16 to transport an object, lift an object, push an object, move, operate a home appliance, etc.
[0023] The control plan generation unit 4 is a specific example of a control plan generation means. Based on the robot information of each robot 16 acquired by the first acquisition unit 2 and the control instructions acquired by the second acquisition unit 3, the control plan generation unit 4 generates a control plan for each robot 16 to execute the control instructions in cooperation with each robot 16, using the first generation model 7.
[0024] The control plan for each robot 16 includes information such as a rational mediation plan that determines what each robot 16 should do, and a proposed coordinated action plan for each robot 16 in response to control instructions. The first generative model 7 is, for example, a trained model whose inputs are robot information and control instructions, and whose output is the control plan for each robot 16.
[0025] The first generation model 7 may be pre-stored in a storage unit such as the storage device 13. The first generation model 7 is configured, for example, as a Large Language Model (LLM).
[0026] Large-scale language models include language models created based on machine learning techniques such as neural networks. For example, large-scale language models include language models trained using supervised learning, semi-supervised learning, or unsupervised learning techniques. In this embodiment, machine learning is a concept that encompasses deep learning.
[0027] The language models included in large-scale language models are, for example, some language models such as GPT4o, BERT (Bidirectional Encoder Representations from Transformers), RoBERTa (Robustly-optimized BERT approach), DeBERTa (Decoding-enhanced BERT with disentangled attention), Longformer, ALBERT (A Lite BERT), and ELECTRA.
[0028] The conversation generation unit 5 is a specific example of a conversation generation means. Based on the control plans for each robot 16 generated by the control plan generation unit 4, the conversation generation unit 5 generates conversations between the robots 16 and conversations between each robot 16 and the user, using the second generation model 8, for each robot 16 to execute its control plan.
[0029] The second generative model 8 is a pre-trained model in which, for example, the input is the control plan for each robot 16, and the output is the conversation between each robot 16 and the conversation between each robot 16 and the user.
[0030] The second generative model 8 may be pre-stored in a storage unit such as the storage device 13. The second generative model 8 is configured, for example, as a Large Language Model (LLM). The first and second generative models 8 are configured as different language models, but they may be configured as different language models. Alternatively, the first and second generative models 8 may be configured integrally as a large language model.
[0031] The display unit 6 is one specific example of an output means. The display unit 6 outputs by displaying the dialogue generated by the dialogue generation unit 5. The display unit 6 is composed of, for example, a liquid crystal display device, an organic EL (electro-luminescence) display device, etc. The dialogue generated by the dialogue generation unit 5 may also be output by a speaker or the like.
[0032] The following describes a specific example of the conversation between each robot 16 and the conversation between each robot 16 and the user that is displayed on the display unit 6.
[0033] This scenario assumes that robots A and B are controlled according to user instructions. Furthermore, robot A is configured to be inoperable while robot B is operational.
[0034] User: "Robot A, go to the living room table and check if the wallet is there." Robot A: "Understood. Robot B, carry me to the table in the living room." Robot B: "Robot A is currently working, but will come over as soon as it's finished." Robot A: "Understood." Robot B: "I have now coupled with Robot A. I will now head to the living room." User: "Okay, I understand." Robot A: "Understood." Robot B: "We have arrived." Robot A: "I found the wallet." User: "Thank you."
[0035] As described above, in response to instructions from the user, tasks are broken down and assigned according to the state and characteristics of each robot A and B, and the user can check the natural language conversational text displayed on the display unit 6 that shows the process and results.
[0036] Next, the control method according to this embodiment will be described. Figure 3 is a flowchart showing an example of the flow of the control method according to this embodiment.
[0037] The first acquisition unit 2 acquires robot information relating to multiple robots 16 (step S101).
[0038] The second acquisition unit 3 acquires control instructions for the multiple robots 16 (step S102).
[0039] The control plan generation unit 4 generates a control plan for each robot 16 to execute the control instructions in cooperation with each robot 16, based on the robot information of each robot 16 acquired by the first acquisition unit 2, the control instructions acquired by the second acquisition unit 3, and the first generated model 7 (step S103).
[0040] The conversation generation unit 5 generates conversations between the robots 16 and between each robot 16 and the user, based on the control plans for each robot 16 generated by the control plan generation unit 4, using the second generation model 8 (step S104).
[0041] The display unit 6 displays the dialogue generated by the dialogue generation unit 5 (step S105).
[0042] As described above, the control system 1 according to this embodiment uses the robot information and control instructions of each robot 16 and the first generation model 7 to generate a control plan for each robot 16 to execute the control instructions in cooperation with each robot 16. Then, based on the control plan, the control system 1 uses the second generation model 8 to generate conversation text between each robot 16 and conversation text between each robot 16 and the user for each robot 16 to execute the control plan, and displays the conversation text.
[0043] Thus, the control system 1 according to this embodiment generates and displays to the user conversation text between each robot 16 and conversation text between each robot 16 and the user, enabling each robot 16 to efficiently execute the control plan in two stages. As a result, when multiple robots 16 cooperate to perform a task and specific control is assigned to each robot 16, the user can easily confirm the process and results in natural language.
[0044] Embodiment 2 Figure 4 is a block diagram showing a schematic system configuration of the control system according to this embodiment. When multiple robots 23 interact using a generation model 22 and coordinately execute an instructed task, requiring mediation between them as needed, there is a challenge in streamlining the execution of the process and reducing the processing load.
[0045] In contrast, the control system 20 according to this embodiment includes a plurality of control units 21 that are associated with a plurality of robots 23 and that issue control instructions to each associated robot 23.
[0046] Each control unit 21 interacts with a common generative model 22 in response to a predetermined control instruction, and based on the result of that interaction, issues control instructions to the corresponding robots 23, thereby coordinating the execution of the predetermined control instructions. The conversations between the control units 21 may include, for example, conversations to ask for the opinion of another robot 23, conversations to ask for a judgment from the user, and conversations to propose mediation plans for actions between the robots 23.
[0047] In this way, each control unit 21 can share information about each robot 23 by interacting using a common generative model 22, thereby improving the efficiency of mediation processing between robots 23 and enabling the generation of actions based on knowledge of the other robot's situation and capabilities. Consequently, the execution of this process can be made more efficient and the processing load reduced.
[0048] The predetermined control instructions mentioned above are, for example, voice commands from the user. The predetermined control instructions may also be generated, for example, based on sensor values acquired by the robot 23's sensors, specific actions of the robot 23, or certain environmental conditions. The predetermined control instructions include, for example, transporting an object, lifting an object, pushing an object, moving, operating home appliances, etc.
[0049] The generative model 22 described above is, for example, a large-scale language model. The dialogue described above is, for example, a dialogue concerning mediation to determine what each robot 23 will do, or the coordinated actions of each robot 23 in response to control instructions.
[0050] Furthermore, the above-mentioned dialogues shall include voice conversations, text-based conversations such as chat, conversations involving the display of images containing language, conversations using gestures such as sign language, and conversations that combine these methods as arbitrarily.
[0051] The control system 20 is configured, for example, as a server device. The control system 20 and each robot 23 may transmit and receive signals using wireless communication such as Bluetooth® or Wi-Fi®. Each control unit 21 may be provided on the corresponding robot 23.
[0052] As described above, the control system 20 according to this embodiment makes it possible to issue instructions to multiple robots 23 and mediate between them using a shared generation model 22, without having to prepare a dedicated generation model for each of the multiple robots 23. Furthermore, even in a complex control system that processes instructions from a user while mediating between multiple robots 23 as needed, it is possible to streamline processing and reduce the processing load.
[0053] While several embodiments of this disclosure have been described, these embodiments are presented as examples only and are not intended to limit the scope of the invention. These novel embodiments can be carried out in a variety of other forms, and various omissions, substitutions, and modifications can be made without departing from the spirit of the invention. These embodiments and their variations are included in the scope and spirit of the invention, as well as in the claims and their equivalents.
[0054] This disclosure can also be implemented, for example, by having a processor execute a computer program, as shown in Figure 3.
[0055] Programs can be stored and supplied to a computer using various types of non-transitory computer-readable media. Non-transitory computer-readable media include various types of tangible storage media. Examples of non-transitory computer-readable media include magnetic recording media (e.g., flexible disks, magnetic tapes, hard disk drives), magneto-optical recording media (e.g., magneto-optical disks), CD-ROMs (Read Only Memory), CD-Rs, CD-R / Ws, and semiconductor memory (e.g., mask ROMs, PROMs (Programmable ROMs), EPROMs (Erasable PROMs), flash ROMs, RAMs (random access memory)).
[0056] Programs may be supplied to a computer by various types of transient computer-readable medium. Examples of transient computer-readable medium include electrical signals, optical signals, and electromagnetic waves. Transitory computer-readable medium can be supplied to a computer via wired communication channels such as electric wires and optical fibers, or via wireless communication channels.
[0057] Each component of the control systems 1 and 20 according to the above-described embodiment can be implemented not only by program, but also, in whole or in part, by dedicated hardware such as an ASIC (Application Specific Integrated Circuit) or an FPGA (Field-Programmable Gate Array). [Explanation of Symbols]
[0058] 1. Control System 2 First acquisition part 3 Second acquisition part 4. Control Plan Generation Unit 5. Conversation text generation unit 6 Display section 7 Generative Models 8 Generative Models 11 processors 12 internal memory 13 Storage Devices 16 Robots 20 Control Systems 21 Control Unit 22 Generative Models 23 Robots
Claims
1. A first acquisition means for acquiring robot information about multiple robots, A second acquisition means for acquiring control instructions for the plurality of robots, A control plan generation means generates a control plan for each robot, based on the robot information of each robot acquired by the first acquisition means and the control instructions acquired by the second acquisition means, using a first generation model, for each robot to coordinately execute the control instructions. A conversational text generation means generates conversational text between each robot and conversational text between each robot and the user, using a second generation model, based on the control plan generated by the control plan generation means, An output means for outputting the conversation text generated by the conversation text generation means, Equipped with, Control system.
2. A control system according to claim 1, The first and second generative models described above are composed of large-scale language models. Control system.
3. A control system according to claim 1, The robot information includes at least one of the following: robot performance, robot status, robot history, robot position, and robot function. Control system.
4. Steps to obtain robot information about multiple robots, The steps include: obtaining control instructions for the plurality of robots; Based on the robot information of each robot acquired and the control instructions acquired, a first generative model is used to generate a control plan for each robot in which each robot can cooperate to execute the control instructions. Based on the generated control plan, the step of generating conversational text between the robots and conversational text between the robots and the user, using a second generative model, for each robot to execute the control plan; The steps include outputting the generated conversation text, including, Control method.
5. A process for acquiring robot information about multiple robots, A process for obtaining control instructions for the aforementioned multiple robots, Based on the robot information of each robot acquired and the control instructions acquired, a process is performed to generate a control plan for each robot, in which each robot will cooperate to execute the control instructions, using a first generative model. Based on the generated control plan, a process is performed to generate conversational text between the robots and conversational text between the robots and the user, using a second generative model, for each robot to execute the control plan. The process of outputting the generated conversation text, Make the computer execute it. program.