Robot control system and robot control method using same

Abstract

A robot control system according to an embodiment of the present invention includes: a voice reception unit for receiving a voice command of a user for controlling a robot; a command conversion unit for converting the voice command received by the voice reception unit into a text command; a prompt text generation unit for generating a prompt text for calling a large language model (LLM) on the basis of the text command; a code generation unit for processing a primary robot control code generated by the LLM on the basis of the prompt text for calling the LLM, so as to generate a secondary robot control code for controlling the robot; and a control unit for controlling the robot on the basis of the secondary robot control code.

Description

Robot control system and robot control method using the same

[0001] The present invention relates to a control system capable of controlling a robot such as an industrial robot, a collaborative robot, a humanoid robot, and a robot control method using the same.

[0002] Traditionally, programming languages ​​and codes were used to control robots. However, this method required a high level of specialized knowledge and technical skills, making it difficult for ordinary robotics users, not experts in the field, to modify the code to control the robot.

[0003] Therefore, in order to change the robot's movements from a preset state, we had to rely on specialized personnel.

[0004] In the case of robots or humanoid robots used in small businesses, there are robots that recognize the user's voice and respond or are controlled, but most of them are robots that only execute the algorithm that most closely matches the user's voice command among multiple pre-programmed algorithms.

[0005] These robots cannot respond to user requests that are outside of their pre-programmed specifications.

[0006] The problem to be solved by the present invention is to provide a robot control system and a robot control method that, when a user provides a voice command using natural language, controls the robot by writing a control code corresponding to the command.

[0007] The tasks of the present invention are not limited to the tasks mentioned above, and other tasks not mentioned will be clearly understood by those skilled in the art from the description below.

[0008] A robot control system according to an embodiment of the present invention for solving the above problem includes a voice receiving unit that receives a user's voice command for controlling a robot, a command conversion unit that converts the voice command received by the voice receiving unit into a text command, a prompt text generation unit that generates a prompt text for calling an LLM (Large Language Model) based on the text command, a code generation unit that processes a primary robot control code generated by an LLM based on the prompt text for calling the LLM to generate a secondary robot control code for controlling the robot, and a control unit that controls the robot based on the secondary robot control code.

[0009] In addition, a robot control method according to an embodiment of the present invention for solving the above problem includes a step of receiving a user's voice command for controlling a robot, a step of converting the voice command into a text command, a step of generating a prompt text for calling an LLM (Large Language Model) based on the text command, a step of processing a primary robot control code generated by an LLM based on the prompt text for calling the LLM to generate a secondary robot control code for controlling the robot, and a step of controlling the robot based on the secondary robot control code.

[0010] Other specific details of the present invention are included in the detailed description and drawings.

[0011] According to embodiments of the present invention, at least the following effects are achieved.

[0012] Even users without specialized knowledge of robot control can easily control the robot.

[0013] When a user provides a voice command using natural language, the programming of control code according to the voice command is performed by LLM, so that the user's various voice commands can be easily and accurately reflected.

[0014] The effects according to the present invention are not limited to those exemplified above, and more diverse effects are included in this specification.

[0015] FIG. 1 is a block diagram illustrating a robot control system according to one embodiment of the present invention.

[0016] FIG. 2 is a flowchart illustrating a robot control method using a robot control system according to one embodiment of the present invention.

[0017] Figure 3 is a flowchart illustrating detailed steps of step S30 of Figure 2.

[0018] Figure 4 is a flowchart illustrating detailed steps of step S50 of Figure 2.

[0019] FIG. 5 is a schematic diagram illustrating an application example of a robot control system according to one embodiment of the present invention.

[0020] The advantages and features of the present invention, and the methods for achieving them, will become clearer with reference to the embodiments described in detail below together with the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below and may be implemented in various different forms. These embodiments are provided solely to ensure that the disclosure of the present invention is complete and to fully inform those skilled in the art of the scope of the invention, and the present invention is defined solely by the scope of the claims.

[0021] Furthermore, the embodiments described herein will be described with reference to cross-sectional and / or schematic drawings, which are ideal illustrations of the present invention. Therefore, the form of the illustrations may be modified due to manufacturing techniques and / or tolerances. Furthermore, in each drawing illustrated in the present invention, each component may be depicted somewhat enlarged or reduced for convenience of explanation. Throughout the specification, the same reference numerals denote the same components.

[0022] Hereinafter, the present invention will be described with reference to drawings for explaining a robot control system according to an embodiment of the present invention and a robot control method using the same.

[0023] FIG. 1 is a block diagram illustrating a robot control system according to one embodiment of the present invention, and FIG. 2 is a flowchart illustrating a robot control method using the robot control system according to one embodiment of the present invention.

[0024] Referring to FIG. 1, a robot control system (1) according to one embodiment of the present invention may include a robot (10), a voice receiving unit (20), a command conversion unit (30), a prompt text generation unit (40), a prompt template text storage unit (50), an environmental data storage unit (60), a code generation unit (70), and a control unit (80).

[0025] Referring to FIG. 2, a robot control method according to one embodiment of the present invention may include a step of receiving a user's voice command (S10), a step of converting the voice command into a text command (S20), a step of generating a prompt text for calling an LLM (Large Language Model) based on the text command (S30), a step of generating a primary robot control code by calling the LLM (S40), a step of generating a secondary robot control code based on the primary robot control code (S50), and a step of controlling a robot based on the secondary robot control code (S60).

[0026] In Fig. 1, a collaborative robot is shown as an example of a robot (10), but various robots such as industrial robots and humanoid robots can be used as the robot (10).

[0027] In step S10 where a user's voice command is received, the voice receiving unit (20) receives the user's voice command. The user provides a voice command for an action to be performed by the robot (10). For example, the user may provide a voice command such as, "Grab the items from the preparation area and palletize them," or "Move two bottles from the conveyor belt to the box." The voice command may be provided in natural language at a level that a non-robotics expert would use to request a task from the robot. The voice receiving unit (20) may record the received voice command.

[0028] In the step (S20) where a voice command is converted into a text command, the command conversion unit (30) converts the voice command received by the voice receiving unit (20) into a text command. The command conversion unit (30) may include software that converts voice into text, such as an STT (Speech to Text) model. The command conversion unit (30) may convert a voice command recorded by the voice receiving unit (20) into a text command.

[0029] In the step (S30) where a prompt text for an LLM call is generated based on a text command, a prompt text generation unit (40) generates a prompt text for an LLM call based on the text command.

[0030] Figure 3 is a flowchart illustrating detailed steps of step S30 of Figure 2.

[0031] Referring to FIG. 3, step S30 may include a step of loading prompt template text (S31), a step of loading environment data (S32), and a step of creating prompt text for LLM calling (S33).

[0032] In the step (S31) where the prompt template text is loaded, the prompt text generation unit (40) can load the prompt template text from the prompt template text storage unit (50).

[0033] The prompt template text may include a description of the control function, example robot control code and rules for writing robot control code, information about environment data for executing the user's command, and a phrase that instructs the robot control code to be written based on the text command.

[0034] For example, the prompt template text can be written as shown in [Table 1] below.

[0035] 구간프롬프트 탬플릿 텍스트(A)IndyClient is a python class for controlling a 6 DOF robot arm, which exposes the following:IndyClient() -> Initializes the prog instance, which exposes the following function:robot.move_to(location="") # Moves the robot arm to the specified location.robot.is_moving() # Returns True if the robot is moving.robot.use_tool(action="") # Executes the action with the current tool.(B)Below is the example code for executing a task with the IndyClient class.(e.g.) command: "Indy, move 2 bottles from the conveyor to the box."```from sbt_helper import IndyClientimport datetime, asynciorobot = IndyClient()async def move_and_wait_robot(location):robot.move_to("conveyor")while(robot.is_moving()):await asyncio.sleep(1)async def main():move_and_wait_robot("idle")for i in range(2):move_and_wait_robot("conveyor")robot.use_tool("grab")move_and_wait_robot("box")robot.use_tool("release")move_and_wait_robot("idle")asyncio.run(main())(C)Rules:- You are allowed to create new functions using these, but you are not allowed to use any other hypothetical functions.- And you are allowed to create recursive functions to solve the movement plans.- Your response will be executed directly as a python file, so all descriptions must be written within the code as inline comments. (e.g. # This code will move 1 block from left to right)(D)The following locations and tools are currently available for you to use.Defined Locations:<LOCATION_LIST>Defined Tool actions:<TOOL_ACTION_LIST>Now, write a python code to execute a requested task based on the following plain-language command.<COMMAND_INPUT>.

[0036] Section (A) of the prompt template text may contain a description of the control function. The description of the control function may be necessary for LLM (90) to understand the example robot control code described in section (B). The description of the control function described in section (A) may be based on a high-level robot control API (Application Programming Interface). The high-level robot control API is a robot control program API that is simplified for LLM (90) to understand, and can simplify code that may be dozens of lines when using a low-level robot API into a single control function call.

[0037] High-level robot control APIs and low-level robot control APIs can be compared as shown in [Table 2] below.

[0038] Function type High-level robot API usage example Low-level robot API call example Movemove_to("left")MoveL(waypoints=[10.0, 1.0, 5.0, 0.0, 0.0, 0.0]) Called with the location name defined in the user preferences Called with the 3D coordinates of the location with the entered name among the locations defined in the user preferences Moved to the location by calculating the size of the objects at the corresponding location and adjusting the coordinates so as not to collide with them. Mechanism useuse_tool("close")use_tool("close", "gripper")MoveL(waypoints=[0.0, 0.0, 1.0, 0.0, 0.0, 0.0]))SetDO([(9, True)]) Called using the action name defined in the user preferences, and when multiple mechanisms are connected at the same time, called using the mechanism name together Called using the IO / movement command set defined in the user preferences according to the entered action name and mechanism name Pick and placepick_and_place(from="left", to="right", tool="gripper", pick_action=["close"], place_action=["open"]) / PickMoveL(waypoints=[10.0, 1.0, 5.0, 0.0, 0.0, 0.0])SetDO([(8, True)])MoveL(waypoints=[10.0, 1.0, 0.0, 0.0, 0.0, 0.0])SetDO([(9, True)])MoveL(waypoints=[10.0, 1.0, 5.0, 0.0, 0.0, 0.0]) / PlaceMoveL(waypoints=[20.0, 1.0, 5.0, 0.0, 0.0, 0.0])MoveL(waypoints=[20.0, 1.0, 0.0, 0.0, 0.0, 0.0])SetDO([(8, True)])MoveL(waypoints=[20.0, 1.0, 5.0, 0.0, 0.0, 0.0]) Call using the location name, tool name and action name defined in the user environment settings. Calling the movement and tool use described above in a complex manner. Automatically calculates the location and calls commands according to the input values ​​for the object approach and tool use process for pick and place work. Calling the robot vision detect(name="Vision_1", action="has_label") Detect("Vision_1", "has_label") Calling using the robot vision name and recognition function name defined in the user environment settings. Calling the low-level robot API function using the input values. Returning the list of defined locations, objects, tools and tool actions get_location_names() / get_object_names() / get_tool_names() / get_tool_action_names(tool="gripper") (Not using the low-level robot API) Returning the data defined in the user environment settings. Checking the current object location and number get_current_object_list() (Not using the low-level robot API) Returning the object location, number information, etc. recorded in the current user environment settings data. Reflecting the result of the object moving by the movement / pick and place described above. When determining the location using the return robot vision, the location results recognized by the vision are returned.

[0039] Referring back to [Table 1], section (B) of the prompt template text may contain example robot control code. The example robot control code is robot control code that causes the robot to perform a specific command. The example robot control code may be robot control code written using the control function described in section (A), and may be robot control code based on a high-level robot control API. The example robot control code may be necessary for LLM (90) to understand how to write robot control code. [Table 1] contains example robot control code that causes the robot to perform the command, “Move two bottles from the conveyor to the box.”

[0040] Section (C) of the prompt template text may contain rules for writing robot control code. These rules may include permissible and unpermissible matters, and requirements for writing robot control code.

[0041] The (D) section of the prompt template text may contain environmental requirements for executing the user's command. Environmental data may include location information, mechanism information, object information, object location information, object count information, vision information, and the like. The environmental data is loaded in step S32, described below, and its specific details will be described later.

[0042] In step S31, the prompt text generation unit (40) can select the type of environmental data for executing the text command based on the text command and record it as environmental requirement information in section (D) of the prompt template text. That is, if the text command is “Grab the items from the preparation area and palletize them,” the prompt text generation unit (40) can record location information and mechanism information as environmental requirement information for executing the command, as in section (D) of [Table 1].

[0043] Referring to [Table 1], in the prompt template text, environment requirement information for executing the user's command may be indicated with placeholders (e.g., "<", ">"). This is to easily substitute and input information corresponding to the environment requirement information (e.g., LOCATION_LIST, TOOL_ACTION_LIST) described in the prompt template text among the information included in the environment data described below. The specific details of this will be described below.

[0044] Also, referring to [Table 1], the (D) section of the prompt template text may include a phrase that instructs the creation of robot control code based on a text command. Referring to [Table 1], in the prompt template text, the part where the text command is to be inserted (COMMAND_INPUT) may be indicated with a placeholder (e.g., "<", ">").

[0045] Meanwhile, in the step (S32) where environmental data is loaded, the prompt text generation unit (40) can load environmental data from the environmental data storage unit (60).

[0046] Environmental data may include location information around the robot (10), information on equipment mounted on the robot (10), information on work target objects, vision information of the robot (10), etc.

[0047] The location information may be information about fixed locations (e.g., a preparation area, a pallet, a starting position, a table, a shelf, etc.) located around the robot (10). The location information may include a name for each location, a maximum number of objects per location, a limit on the type of objects per location, etc.

[0048] The device information may include the name of each device mounted on the robot (10), a list of possible actions for each device, etc.

[0049] Information about the object to be worked on may include the object name, the name of the object-specific interactive mechanism, and the object-specific interactive mechanism behavior.

[0050] Vision information may include a list of recognizable visions connected to the robot (e.g., object position recognition function, object count recognition function, etc.).

[0051] Environmental data may include object location information including information about the location of each object and object count information including information about the number of each object.

[0052] In the step (S33) where prompt text for LLM calling is created, the prompt text generation unit (40) can create prompt text for LLM calling based on text commands, prompt template text, and environment data.

[0053] In step S33, the prompt text generation unit (40) can insert environmental data corresponding to the environmental requirement information (e.g., LOCATION_LIST, TOOL_ACTION_LIST) of the prompt template text. Since the environmental requirement information is indicated with placeholders (e.g., "<", ">"), the environmental requirement information can be easily identified.

[0054] Additionally, in step S33, the prompt text generation unit (40) can insert a text command into a portion of the prompt template text where a text command is to be inserted (e.g., COMMAND_INPUT). The portion where the text command is to be inserted is indicated with a placeholder (e.g., "<", ">"), so that the portion can be easily identified.

[0055] The prompt text for LLM calls created by step S33 can be written as shown in [Table 3] below.

[0056] 구간LLM 호출용 프롬프트 텍스트(A)IndyClient is a python class for controlling a 6 DOF robot arm, which exposes the following:IndyClient() -> Initializes the prog instance, which exposes the following function:robot.move_to(location="") # Moves the robot arm to the specified location.robot.is_moving() # Returns True if the robot is moving.robot.use_tool(action="") # Executes the action with the current tool.(B)Below is the example code for executing a task with the IndyClient class.(e.g.) command: "Indy, move 2 bottles from the conveyor to the box."```from sbt_helper import IndyClientimport datetime, asynciorobot = IndyClient()async def move_and_wait_robot(location):robot.move_to("conveyor")while(robot.is_moving()):await asyncio.sleep(1)async def main():move_and_wait_robot("idle")for i in range(2):move_and_wait_robot("conveyor")robot.use_tool("grab")move_and_wait_robot("box")robot.use_tool("release")move_and_wait_robot("idle")asyncio.run(main())(C)Rules:- You are allowed to create new functions using these, but you are not allowed to use any other hypothetical functions.- And you are allowed to create recursive functions to solve the movement plans.- Your response will be executed directly as a python file, so all descriptions must be written within the code as inline comments. (eg # This code will move 1 block from left to right)(D)The following locations and tools are currently available for you to use.Defined Locations:"prepare_area", "pallet_area1"~"pallet_area10"Defined Tool actions:"grab", "release"Now, write a python code to execute a requested task based on the following plain-language command."Grabbing items from the preparation area and palletizing them. Please do it".

[0057] By comparing [Table 1] and [Table 3], you can see that the bolded part in section (D) has been modified.

[0058] In step S33, the prompt text generation unit (40) can generate a prompt text for calling LLM by replacing the environment requirement information (e.g., LOCATION_LIST, TOOL_ACTION_LIST) of the prompt template text in the environment data with the corresponding environment data ("prepare_area", "pallet_area1" to "pallet_area10") and inserting a text command ("grab the items from the preparation area and palletize them") into the prompt template text.

[0059] In the step (S40) where the primary robot control code is generated by calling LLM (90), the prompt text generation unit (40) calls LLM (90) using the prompt text for calling LLM, and LLM (90) generates the primary robot control code based on the prompt text for calling LLM.

[0060] In the step (S50) where the secondary robot control code is generated based on the primary robot control code, the code generation unit (70) processes the primary robot control code generated by the LLM (90) to generate the secondary robot control code for controlling the robot (10).

[0061] Figure 4 is a flowchart illustrating detailed steps of step S50 of Figure 2.

[0062] Referring to FIG. 4, the step (S50) in which the secondary robot control code is generated may include a step (S51) in which text causing an error in the primary robot control code is deleted, a step (S52) in which the existence of a load code and an execution code in the primary robot control code is confirmed, and a step (S53) in which the secondary robot control code is completed.

[0063] In the step (S51) where text causing an error is deleted from the primary robot control code, the code generation unit (70) can delete text causing an error in the code execution process by the control unit (80) from the primary robot control code created by the LLM (90). For example, text for descriptions or information other than the robot control code included in the result format (e.g., Json format) generated by the LLM (90) based on the prompt text for calling the LLM can be deleted. Alternatively, in step S51, the code generation unit (70) can be configured to copy only the robot control code from the result format generated by the LLM (90).

[0064] In addition, at step S51, the code generation unit (70) can check for the presence of a string for displaying code on the web, such as “```”, in the robot control code generated by LLM (90), and if the string exists, the code indicated as “```” can be deleted.

[0065] In the step (S52) where the presence of the import code and the execution code is confirmed in the first robot control code, the code generation unit (70) can confirm the presence of the import code and the execution code in the robot control code generated through step S51.

[0066] For example, the code generation unit (70) can check whether an import code such as “import asyncio” exists in the robot control code generated through step S51, and if not, insert the import code into the robot control code.

[0067] In addition, the code generation unit (70) can check whether an execution code such as “asyncio.run(main())” exists in the robot control code generated through step S51, and if it does not exist, can insert the execution code into the robot control code.

[0068] At the stage where the secondary robot control code is completed (S53), the code generation unit (70) can finally generate the secondary robot control code that can be executed without error by the control unit (80).

[0069] An example of a completed secondary robot control code based on the prompt text for LLM call in [Table 3] may be as in [Table 4].

[0070] # This script is designed to move items from the "prepare_area" to pallet locations# ("pallet_area1" to "pallet_area10") in response to the given task: "준비영역에서 물건들을 잡아서 팔레타이징 해줘"# The robot will pick items one by one from the "prepare_area" and place them in consecutive pallet locations.from sbt_helper import IndyClientimport asyncio# Initialize the robot clientrobot = IndyClient()# Helper function to move the robot to a specified location and wait until the movement is completeasync def move_and_wait_robot(location):robot.move_to(location) # Command the robot to move to the given locationwhile robot.is_moving(): # Wait until the robot finishes movingawait asyncio.sleep(1) # Pause to avoid busy waiting# Recursive function to perform the palletizing taskasync def palletize_items(current_pallet=1, max_pallets=10):# Base case: Stop if all pallet locations are usedif current_pallet > max_pallets:return# Move to "prepare_area" to pick an itemawait move_and_wait_robot("prepare_area")robot.use_tool("grab") # Grab the item# Move to the current pallet area to place the itempallet_location = f"pallet_area{current_pallet}" # Determine the current pallet locationawait move_and_wait_robot(pallet_location)robot.use_tool("release") # Release the item# Move back to "prepare_area" or idle stateawait move_and_wait_robot("prepare_area")# Recursive call for the next palletawait palletize_items(current_pallet + 1, max_pallets)# Main function to start the palletizing processasync def main():await move_and_wait_robot("prepare_area") # Start in the "prepare_area"await palletize_items() # Begin the palletizing task# Execute the main function using asyncioasyncio.run(main()).

[0071] At the stage where the secondary robot control code is completed (S53), the completed secondary robot control code can be stored as a program file in the control unit (80).

[0072] In the step (S60) where the robot is controlled based on the secondary robot control code, the control unit (80) can control the robot (10) by executing the secondary robot control code.

[0073] FIG. 5 is a schematic diagram illustrating an application example of a robot control system according to one embodiment of the present invention.

[0074] Referring to FIG. 5, a robot control system (2) according to one embodiment of the present invention may include a robot (10), a teach pendant (11), and a robot controller (12).

[0075] The voice receiving unit (20), command switching unit (30), prompt text generating unit (40), and code generating unit (70) may be included in the teach pendant (11). Alternatively, the functions of the voice receiving unit (20), command switching unit (30), prompt text generating unit (40), and code generating unit (70) may be performed by the teach pendant (11).

[0076] For example, the voice receiving unit (20) may be performed by a recording function included in the teach pendant (11). The command conversion unit (30) may be software included in the teach pendant (11) or an STT model server communicatively connected to the teach pendant (11). The prompt text generation unit (40) and the code generation unit (70) may be software included in the teach pendant (11).

[0077] The teach pendant (11) is connected to the LLM server so that it can communicate with the LLM server and can call the LLM or receive the primary robot control code generated by the LLM.

[0078] The robot controller (12) is connected to the robot (10) via wired / wireless connection and provides signals for controlling the robot (10) to the robot (10). A prompt template text storage unit (50), an environmental data storage unit (60), and a control unit (80) may be included in the robot controller (12).

[0079] Depending on the embodiment, unlike that illustrated in FIG. 5, the prompt template text storage unit (50) and / or the environment data storage unit (60) may be included in the teach pendant (11), or the prompt text generation unit (40) and / or the code generation unit (70) may be included in the robot controller (12). Alternatively, at least some of the voice receiving unit (20), the command switching unit (30), the prompt text generation unit (40), the prompt template text storage unit (50), the environment data storage unit (60), and the code generation unit (70) may be included in a separate operation processing unit other than the teach pendant (11) and the robot controller (12).

[0080] According to the robot control system and robot control method according to one embodiment of the present invention described above, a user can control the robot by giving a voice command close to natural language.

[0081] This allows even users without specialized knowledge of robot control to easily control the robot, and thus improves the convenience of robot use by the general public, as workers working with robots or users of small businesses using collaborative robots can easily control the robot.

[0082] In addition, when a user provides a voice command using natural language, the programming of control code according to the voice command is performed by LLM, so that various voice commands of the user can be easily and accurately reflected.

[0083] In addition, since the robot control code written by LLM is verified and the final robot control code is generated, cases where the robot is not actually controlled due to code errors, etc. can be minimized.

[0084] Those skilled in the art will appreciate that the present invention can be implemented in other specific forms without altering its technical spirit or essential characteristics. Therefore, the embodiments described above should be understood as illustrative in all respects and not restrictive. The scope of the present invention is indicated by the claims below rather than the detailed description above, and all changes or modifications derived from the meaning and scope of the claims and their equivalents should be construed as being included within the scope of the present invention.

[0085] <National Research and Development Project Supporting This Invention>

[0086] [Project ID] 1415184739

[0087] [Assignment Number] 20014398

[0088] Ministry of Trade, Industry and Energy

[0089] [Project Management (Specialist) Agency] Korea Industrial Technology Evaluation and Planning Institute

[0090] [Research Project Name] Industrial Technology Innovation Project (Robot Industry Technology Development Project)

[0091] [Research Project Title] Development of a 5kHz or higher robot controller capable of responding to arbitrary kinematic shapes, being convenient, safe, and capable of integrating artificial intelligence.

[0092] [Contribution rate] 1 / 1

[0093] [Name of the project performing organization] Neuromeca Co., Ltd.

[0094] [Research Period] April 1, 2021 - December 31, 2024

Claims

1. A voice receiving unit that receives a user's voice commands to control the robot; A command conversion unit that converts the voice command received by the voice receiving unit into a text command; A prompt text generation unit that generates a prompt text for calling an LLM (Large Language Model) based on the above text command; A code generation unit that processes the primary robot control code generated by LLM based on the prompt text for calling the LLM and generates a secondary robot control code for controlling the robot; and A robot control system, comprising a control unit that controls the robot based on the second robot control code.

2. In paragraph 1, The prompt text for the above LLM call is: A robot control system comprising at least one of a description of a control function used in the above primary robot control code, an example robot control code, and a robot control code writing rule, and the text command.

3. In paragraph 1, The prompt text for the above LLM call is: A robot control system including at least one of position information of the surroundings of the robot, information on a mechanism mounted on the robot, information on a work target object, and vision information of the robot.

4. In paragraph 3, A robot control system, wherein the above location information includes at least one of a location name, a maximum number of objects per location, and a type of objects per location.

5. In paragraph 3, A robot control system, wherein the above mechanism information includes at least one of a mechanism name and a list of possible actions for each mechanism.

6. In paragraph 3, A robot control system, wherein the above-described task target object information includes at least one of an object name, an object-specific interactable mechanism name, and an object-specific interactable mechanism action.

7. In paragraph 3, A robot control system, wherein the vision information includes a recognizable list of visions connected to the robot.

8. In paragraph 1, The above code generation unit, A robot control system that deletes text causing an error in the first robot control code and generates the second robot control code by confirming the existence of a loading code and an execution code.

9. A step of receiving a user's voice command to control the robot; A step in which the above voice command is converted into a text command; A step of generating a prompt text for calling an LLM (Large Language Model) based on the above text command; A step of processing the primary robot control code generated by LLM based on the prompt text for calling the LLM to generate a secondary robot control code for controlling the robot; and A robot control method, comprising: a step of controlling the robot based on the second robot control code.

10. In paragraph 9, The steps for generating the prompt text for the above LLM (Large Language Model) call are: A step of loading a prompt template text including at least one of a description of a control function used in the above first robot control code, an example robot control code, and a robot control code writing rule; A step of loading environmental data including at least one of location information of the surroundings of the robot, information on a device mounted on the robot, information on a work target object, and vision information of the robot, and A robot control method, comprising a step of generating a prompt text for the LLM call based on the text command, the prompt template text, and the environment data.

11. In paragraph 10, A robot control method, wherein the above location information includes at least one of a location name, a maximum number of objects per location, and a type of objects per location.

12. In paragraph 10, A robot control method, wherein the above mechanism information includes at least one of a mechanism name and a list of possible actions for each mechanism.

13. In paragraph 10, A robot control method, wherein the above-mentioned task target object information includes at least one of an object name, an object-specific interactable mechanism name, and an object-specific interactable mechanism action.

14. In paragraph 10, A robot control method, wherein the vision information includes a recognizable list of visions connected to the robot.

15. In paragraph 9, The step in which the above secondary robot control code is generated is: A step in which text causing an error in the above first robot control code is deleted, and A robot control method, comprising a step of confirming the presence of a loading code and an execution code in the above first robot control code.

Images