Construction method of mechanical arm intelligent interaction and control system based on cross-modal large language model
By building a robot arm intelligent interaction and control system based on a cross-modal large language model, the problem of limited application of robot arm in complex environments is solved, flexible response and natural language understanding for complex task scenarios are achieved, and operation efficiency is improved and user experience lay the foundation for a wider application field.
Patent Information
- Application Number
- CN202411274256.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-12
- Publication Date
- 2024-11-22
AI Technical Summary
The application of robotic arm systems in complex environments is limited by pre-programmed paths and manually set operational processes, and cannot flexibly deal with unpreset task scenarios, and the potential of large language models is not fully utilized.
Build a robot arm intelligent interaction and control system based on a cross-modal large language model, and realize the analysis of natural language instructions and the conversion of robot arm control signals by designing task-oriented large language model agents, system intermediate layers and user interfaces. Combining simulation models and path planning algorithms, flexible response to complex task scenarios is achieved.
It improves the flexibility and adaptability of the robotic arms, reduces the burden of manual planning, improves operational efficiency and user experience, and realizes effective work in a changing environment, laying the foundation for the development of the robotic arms in a wider range of applications.
Abstract
Description
Technical Field
[0001] The present invention relates to the field of automation technology, and in particular to a method for constructing a robotic arm intelligent interaction and control system based on a cross-modal large language model. Background Art
[0002] In the current field of automation technology, the operating accuracy and freedom of the robotic arm system have reached an advanced level, but its application is still limited to pre-programmed paths and manually set operating procedures. This dependence makes the robotic arm unable to flexibly respond to unpreset task scenarios, requiring a lot of manpower for detailed planning, limiting its application potential in complex environments.
[0003] At the same time, large language models (LLMs), such as ChatGPT, have demonstrated their superior capabilities in natural language understanding and task planning. They are able to understand complex human instructions, interact freely, propose action plans, and assist in the decision-making process. However, their potential is limited to the text interaction level and has not yet been fully utilized.
[0004] In view of this, the robot control system integrating LLM provides a solution that breaks through traditional boundaries. Through LLM's in-depth understanding of human language, the robot arm can receive and interpret direct language instructions, automatically execute operation processes, and achieve flexible response to complex task scenarios. This not only reduces the burden of manual planning, but also significantly improves operational efficiency and adaptability.
[0005] Therefore, the present invention proposes a method for constructing a robotic arm intelligent interaction and control system based on a cross-modal large language model. Summary of the invention
[0006] The purpose of the present invention is to provide a method for constructing a robotic arm intelligent interaction and control system based on a cross-modal large language model to solve the problems raised in the above background technology.
[0007] In order to solve the above technical problems, the present invention provides the following technical solutions: a method for constructing a robot arm intelligent interaction and control system based on a cross-modal large language model, wherein the method for constructing the system comprises the following steps:
[0008] S1. Calibration simulation model
[0009] Modify the parameters of the URDF model of the robot arm to correct the penetration problem and coordinate errors in the original model;
[0010] Add a bilateral gripper model to the URDF model and associate it with the original robotic arm model;
[0011] Configure gazebo simulation properties in the URDF model and convert the URDF model into a complete xacro model. The gazebo simulation properties include but are not limited to color, friction coefficient, and dynamic parameters.
[0012] Use the Move It Assistant tool to configure the relevant parameters of the Xacro model in the RVIZ simulation environment, wherein the relevant parameters include but are not limited to the collision matrix, joint chain, controller and simulation times;
[0013] S2. Configure the simulation environment
[0014] By using the Planning Scene object, a desktop for placing objects is added in the rv iz simulation environment, and physical parameters of the desktop are configured, including but not limited to height, size, and thickness;
[0015] Import the URDF model of the object to be grabbed, place it on the desktop, and wait for the grabbing command;
[0016] Import the configured Xacro model to place the robot arm, desktop, and object to be grasped in the same RVI Z simulation scene;
[0017] S3. Build a task-oriented large language model agent
[0018] According to the task system execution process, a serialized dialogue format and instruction list based on task implementation are designed, and a task-oriented large language model agent is constructed. The large language model agent is used to accurately understand and respond to natural language instructions in the dialogue, manage the conversation context memory, and generate system actions;
[0019] S4. Development system middle layer
[0020] Designing a system middle layer for communication between the robot control system and the large language model agent, the system middle layer is also used to manage the parsing and scheduling of natural language instructions, and convert the natural language instructions into control signals for the robot;
[0021] S5. Build the system user interface
[0022] The system user interface is used to interact with the large language model agent and the robotic arm control system, supporting remote access and operation functions;
[0023] S6. Build controller
[0024] Import and call Move It's built-in path planning algorithm and controller to plan the robot's motion path based on the parameters in the configuration file;
[0025] S7. Build tool modules that can achieve tasks such as moving or grasping
[0026] Build a Python program file that can realize tasks such as moving or grasping. The Python program file is used to complete one or a group of robotic arm motion tasks, that is, to parse the control signal of the robotic arm, issue the robotic arm motion command, and call the path planner and controller to perform operations such as moving or grasping.
[0027] According to the above technical solution, the method for constructing the large language model intelligent agent described in S3 includes: manually constructing a large language model conversation sample json l dataset suitable for robotic arm control guidance, and using the dataset to fine-tune the original neural network of the large language model.
[0028] According to the above technical solution, the method for constructing the large language model agent described in S3 includes: constructing an Agent based on a large language model, configuring a set of tools related to task execution, building a conversation context memory chain, writing a scheduler to manage complex conversation threads and reasonably using tool modules.
[0029] According to the above technical solution, the method for constructing the large language model agent described in S3 includes: using the GPT-4o model, writing a conversation management program and designing a system role for the conversation.
[0030] According to the above technical solution, S4 also includes designing two-sided interfaces that adapt to different operating systems and operating environments during the communication establishment process, and building intranet penetration with the network layer inside the virtual machine.
[0031] According to the above technical solution, the system middle layer described in S4 has exception handling and log recording functions;
[0032] According to the above technical solution, S6 also includes limiting the rotation angle and speed of each axis of the robot arm in the configuration file.
[0033] According to the above technical solution, after completing the task required by the user, the system user interface prompts the user that the task is completed and requests the user to confirm.
[0034] Compared with the prior art, the beneficial effects achieved by the present invention are as follows: the present invention provides a method for constructing a robot arm intelligent interaction and control system based on a cross-modal large language model. The control system constructed by this method realizes direct control and precise operation of the robot arm by integrating advanced large language models (LLMs), such as ChatGPT; it has high flexibility and adaptability, and can work effectively under changing environments and different task requirements. By interacting with the user's natural language, the system not only enhances the convenience of operation, but also improves the user experience, realizes the improvement of the robot arm control system in natural language understanding and complex task execution, and lays the foundation for the future development of the robot arm in a wider range of application fields. DETAILED DESCRIPTION
[0035] The technical solutions in the embodiments are described clearly and completely below. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0036] The present invention provides a technical solution: a method for constructing a robot arm intelligent interaction and control system based on a cross-modal large language model, and the system construction method includes the following steps:
[0037] S1. Build a simulation model
[0038] Modify the parameters of the URDF model provided by the robotic arm device to correct the penetration problem and coordinate errors in the original model; add a bilateral gripper model to the URDF model and associate it with the original robotic arm model; configure gazebo simulation properties such as color, friction coefficient, and dynamic parameters in the URDF model to convert the URDF model into a complete xacro model; use the Move It Assistant tool to configure the relevant parameters of the xacro model in the RV iz simulation environment, including the collision matrix, joint chain, controller, and simulation times;
[0039] S2. Configure the simulation environment
[0040] Through the Planning Scene object, add a desktop for placing objects in the RV IZ simulation environment, and configure the physical parameters of the desktop such as height, size and thickness; import the URDF model of the object to be grasped, and place it on the desktop, waiting for the grasping instruction; import the configured Xacro model, so that the robot arm, the desktop and the object to be grasped are in the same RV IZ simulation scene;
[0041] S3. Build a task-oriented large language model agent
[0042] According to the task system execution process, we designed a serialized dialogue format and instruction list based on task implementation, and built a task-oriented large language model agent. The large language model is used to accurately understand and respond to natural language instructions in the dialogue. We tested three different large language model construction methods on the development platform:
[0043] (1) Manually construct a large language model conversation sample JSON L dataset suitable for robot arm control guidance to ensure that the project model can accurately understand and respond to natural language instructions in the conversation. Use the manually constructed conversation sample dataset to fine-tune the original neural network of the large language model to more accurately meet the project requirements;
[0044] (2) Use the GPT-4o model with multimodal understanding capabilities, which can perform basic image recognition and judgment. By writing conversation management programs and designing conversation systems, the coherence and consistent relevance of long conversations during project tasks are ensured;
[0045] (3) Design an agent based on a large language model, configure a set of tools related to task execution, build a conversation context memory chain, write a scheduler to manage complex conversation threads, and use tool modules reasonably;
[0046] After multiple tests and thorough comparisons, the large language model Agent was found to be the most effective, and on this basis, a running framework for interactive control tasks was built;
[0047] S4. Development system middle layer
[0048] The system middle layer designed for communication between the robot control system and the GPT model, as a communication bridge between the robot control system and the large language model Agent, is also used to manage the parsing and scheduling of natural language instructions, and convert natural language instructions into control signals for the robot; in addition, the system middle layer also has exception handling and logging functions to ensure the stable operation of the system. In the process of establishing communication, a two-sided interface that adapts to different operating systems and operating environments is designed, and an intranet penetration is built inside the virtual machine and the network layer to achieve effective transmission of terminal instructions; ensure that the processed natural language instructions are effectively converted into control signals, thereby commanding the robot to perform precise actions;
[0049] S5. Build the system user interface
[0050] The system user interface is used to interact with the large language model and the robot control system, supports remote access and operation functions, makes the management of contextual conversation thread flow more flexible, and can also achieve effective interaction with users;
[0051] S6. Build controller
[0052] During simulation, the built-in path planning algorithm and controller of Move It are imported and called to realize the fixed-point movement and grasping of the robot arm, and the movement path of the robot arm is planned based on the parameters in the configuration file; if the target grasping position is within the working space of the robot arm, the robot arm can plan a reasonable path to move to the specified position, and the gripper controller controls it to close to grasp the target object; in addition, the rotation angle and speed of each axis of the robot arm are restricted in the configuration file to ensure that the robot arm will not move too fast and hit people or objects around it, which has a certain degree of safety;
[0053] S7. Write grab and place task scripts
[0054] After the environment configuration is completed, a Python program file that can realize tasks such as moving or grabbing is constructed. The Python program file is used to complete one or a group of robotic arm motion tasks, that is, to parse the control signal of the robotic arm, issue the robotic arm motion command, call the path planner and the controller to perform operations such as moving or grabbing; the target of the specific action to be performed is received as the input parameter, and the robotic arm uses the global planner and the local actuator module to plan and execute the path. After reaching the specified position, the gripper gripper performs the required operation and executes the control action of grabbing or releasing the object.
[0055] Finally, it should be noted that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art can still modify the technical solutions described in the aforementioned embodiments or replace some of the technical features therein by equivalents. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.
Claims
1. Construction method of intelligent interaction and control system of robotic arm based on cross-modal large language model, It is characterized in that The method for constructing the system comprises the following steps: S1. Calibration simulation model Modify the parameters of the URDF model of the robot arm to correct the penetration problem and coordinate errors in the original model; Add a bilateral gripper model to the URDF model and associate it with the original robotic arm model; Configure gazebo simulation properties in the URDF model and convert the URDF model into a complete xacro model. The gazebo simulation properties include but are not limited to color, friction coefficient, and dynamic parameters. Use the MoveIt Assistant tool to configure the relevant parameters of the xacro model in the rviz simulation environment, including but not limited to the collision matrix, joint chain, controller and simulation times; S2. Configure the simulation environment Through the Planning Scene object, add a desktop for placing objects in the rviz simulation environment and configure the physical parameters of the desktop, including but not limited to height, size and thickness; Import the URDF model of the object to be grabbed, place it on the desktop, and wait for the grabbing command; Import the configured xacro model so that the robot arm, desktop and object to be grasped are in the same rviz simulation scene; S3. Build a task-oriented large language model agent According to the task system execution process, a serialized dialogue format and instruction list based on task implementation are designed to build a task-oriented large language model agent. The large language model is used to accurately understand and respond to natural language instructions in the dialogue, complete the context reasoning process, form a memory chain, and have a flexibly manageable tool module set, and can call any one or more tools in the dialogue context; S4. Development system middle layer Designing a system middle layer for communication between the robot control system and the large language model, the system middle layer is also used to manage the parsing and scheduling of natural language instructions and convert the natural language instructions into control signals for the robot; S5. Build the system user interface The system user interface is used to interact with the large language model and the robotic arm control system, and supports remote access and operation functions; S6. Build controller Import and call MoveIt's built-in path planning algorithm and controller to plan the robot's motion path based on the parameters in the configuration file; S7. Build tool modules that can achieve tasks such as moving or grasping Build a Python program file that can realize tasks such as moving or grasping. The Python program file is used to complete one or a group of robotic arm motion tasks, that is, to parse the control signal of the robotic arm, issue the robotic arm motion command, and call the path planner and controller to perform operations such as moving or grasping.
2. According to the method for constructing a robot arm intelligent interaction and control system based on a cross-modal large language model according to claim 1, Features: The method for constructing the large language model intelligent agent described in S3 includes: manually constructing a conversation sample jsonl dataset suitable for robot arm control guidance, and using the dataset to fine-tune the original neural network of the large language model.
3. According to the method for constructing a robot arm intelligent interaction and control system based on a cross-modal large language model according to claim 1, Features: The method for constructing the large language model agent described in S3 includes: constructing an Agent based on the large language model, configuring a tool set related to task execution, constructing a conversation context memory chain, writing a scheduler to manage complex conversation threads and using tool modules reasonably.
4. According to the method for constructing a robot arm intelligent interaction and control system based on a cross-modal large language model according to claim 1, Features: The method for constructing the large language model intelligent agent described in S3 includes: using the GPT-4o model, writing a conversation management program, and designing a system role for the conversation.
5. The method for constructing a robot arm intelligent interaction and control system based on a cross-modal large language model according to claim 1, Features: S4 also includes designing two-sided interfaces that adapt to different operating systems and operating environments during the communication process, and building intranet penetration with the network layer inside the virtual machine.
6. The method for constructing a robot arm intelligent interaction and control system based on a cross-modal large language model according to claim 1, Features: The system middle layer described in S4 has exception handling and log recording functions.
7. The method for constructing a robot arm intelligent interaction and control system based on a cross-modal large language model according to claim 1, Features: S6 also includes limiting the rotation angle and speed of each axis of the robot arm in the configuration file.
8. The method for constructing a robot arm intelligent interaction and control system based on a cross-modal large language model according to claim 1, Features: After completing the task requested by the user, the system user interface prompts the user that the task is completed and requests the user to confirm.
Citation Information
Cited By
Robot action generation method and device, robot, medium and product
CN122463192A
Robot action generation method and device, robot, medium and product
CN122463192B
