Mechanical arm intelligent control method and device, computer device and readable storage medium

Abstract

The application relates to a mechanical arm intelligent control method and device, computer equipment and a readable storage medium. The method comprises the following steps: obtaining natural language information and historical conversation records of a target user; performing end-to-end autonomous reasoning analysis according to the natural language information and the historical conversation records to obtain the intention of the target user; initiating a tool discovery request to a tool registry through an MCP protocol, and determining a mechanical arm operation tool set and a visual perception tool set according to the tool discovery request; determining a tool calling sequence according to the mechanical arm operation tool set, the visual perception tool set and the intention of the target user; and generating a target control instruction according to the tool calling sequence to drive the mechanical arm to perform an operation task corresponding to the natural language information. The application can improve the intelligent level of mechanical arm control.

Description

Technical Field

[0001] This application relates to the field of robotic arm control technology, and in particular to a robotic arm intelligent control method, device, computer equipment, and readable storage medium. Background Technology

[0002] With the rapid iteration of artificial intelligence technology, large language models have demonstrated powerful application capabilities in natural language understanding, reasoning planning, and tool invocation, providing a new technical path for intelligent robot control. Meanwhile, industrial robots have been widely adopted in many operational scenarios such as manufacturing, warehousing, and logistics. However, traditional robotic arm control methods still rely on teach programming and offline programming to achieve operational control, resulting in complex operating procedures, poor generalization flexibility, and difficulty in adapting to dynamically changing unstructured task scenarios.

[0003] Currently, existing robot control systems generally use predefined instruction mapping to hard-code user input into fixed action sequences, which makes it difficult to handle ambiguous or open intentions. Furthermore, there is a lack of standardized tool discovery and invocation protocols between large language models and robot control, resulting in high device coupling, complex system integration, and poor scalability. At the same time, large language models are only used as dialogue generation components, lacking end-to-end autonomous reasoning and tool orchestration capabilities, unable to flexibly respond to open instructions, and requiring additional coding for each new task type, making expansion difficult.

[0004] Therefore, how to improve the intelligence level of robotic arm control has become an urgent problem to be solved. Summary of the Invention

[0005] Therefore, it is necessary to provide a method, device, computer equipment, and readable storage medium for intelligent control of robotic arms, which can improve the level of intelligence in robotic arm control.

[0006] In a first aspect, this application provides a robotic arm intelligent control method, applied to a target intelligent agent in a robotic arm intelligent control system. The system also includes a robotic arm, and the target intelligent agent is a large language model intelligent agent. The method includes: Obtain the target user's natural language information and historical conversation records; End-to-end autonomous reasoning analysis is performed based on natural language information and historical conversation records to obtain the target user's intent; A tool discovery request is initiated to the tool registry via the MCP protocol, and the robotic arm operation toolset and the visual perception toolset are determined based on the tool discovery request. Determine the tool invocation sequence based on the robotic arm operation toolset, the visual perception toolset, and the target user's intent; The system generates target control instructions based on the tool call sequence to drive the robotic arm to perform operation tasks corresponding to natural language information.

[0007] Secondly, this application provides a robotic arm intelligent control device, applied to a target intelligent agent in a robotic arm intelligent control system. The system also includes a robotic arm, and the target intelligent agent is a large language model intelligent agent. The device includes: The acquisition module is used to obtain the target user's natural language information and historical conversation records; The analysis module is used to perform end-to-end autonomous reasoning analysis based on natural language information and historical conversation records to obtain the target user's intent; The first determination module is used to initiate a tool discovery request to the tool registry through the MCP protocol, and determine the robotic arm operation tool set and the visual perception tool set according to the tool discovery request. The second determining module is used to determine the tool invocation sequence based on the robotic arm operation tool set, the visual perception tool set, and the target user's intent; The generation module is used to generate target control instructions based on the tool call sequence to drive the robotic arm to perform operation tasks corresponding to natural language information.

[0008] Thirdly, this application provides a computer device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the steps in the method described above.

[0009] Fourthly, this application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps in the above-described method.

[0010] Fifthly, this application provides a computer program product comprising a computer program that, when executed by a processor, implements the steps of the method described above.

[0011] The aforementioned intelligent control method, device, computer equipment, and readable storage medium for robotic arms utilize a large language model intelligence agent as the control core. It combines a visual perception toolset with a robotic arm operation toolset to achieve MCP tool call orchestration, and relies on historical conversation records to complete contextual understanding and accurately interpret the user's natural language intent. Compared to existing technologies that use hard-coded instruction mapping and fixed action sequences, this intelligent control method for robotic arms autonomously completes intent recognition and tool call sequence planning through a large language model intelligence agent. Combined with the MCP protocol, it achieves unified invocation of vision and robotic arm equipment, thereby solving the technical problems of weak intent recognition capability, poor scalability, and low intelligence level in traditional robotic arm control methods, and improving the intelligence level of robotic arm control. Attached Figure Description

[0012] Figure 1A A system architecture diagram of an intelligent control system for a robotic arm provided in this application embodiment; Figure 1B An application environment diagram of an intelligent control method for a robotic arm provided in this application embodiment; Figure 2 A flowchart illustrating an intelligent control method for a robotic arm provided in an embodiment of this application; Figure 3 A flowchart illustrating the process of determining a tool invocation sequence is provided in an embodiment of this application. Figure 4 A schematic diagram illustrating the execution of inference by a target intelligent agent, provided as an embodiment of this application; Figure 5 A structural block diagram of a robotic arm intelligent control device provided in an embodiment of this application; Figure 6 An internal structural diagram of a computer device provided in an embodiment of this application; Figure 7 An internal structural diagram of another computer device provided in an embodiment of this application; Figure 8 This is an internal structural diagram of a computer-readable storage medium provided in an embodiment of this application. Detailed Implementation

[0013] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0014] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0015] Please see Figure 1A This application provides a system architecture diagram for a robotic arm intelligent control system. The robotic arm intelligent control system includes, but is not limited to: a presentation layer, a business management layer, an MCP service layer, and a device control layer. The business management layer deploys a target intelligent agent, which is a large language model intelligent agent, used to perform reasoning, decision-making, and task orchestration. The device control layer deploys a robotic arm, including a single-arm robotic arm and / or a dual-arm robotic arm, used to perform physical operation tasks.

[0016] The presentation layer features a web-based visual interactive environment, essentially a human-computer interface (HCI), responsible for HCI interaction, information rendering, and status visualization. This HCI provides users with a unified visual operation entry point, supporting both text and voice input to adapt to different usage scenarios. Simultaneously, the presentation layer integrates SSE streaming technology to receive intermediate content and task feedback generated by the target agent's inference, rendering the page segment by segment to achieve streaming output of dialogue content and improve interaction fluency. Furthermore, relying on the WebSocket bidirectional communication mechanism, it continuously monitors underlying hardware and task operation data, pushing real-time information such as the robotic arm's motion status, gripper status, agent task execution events, and tool call records, dynamically refreshing and visually displaying the system's operating status to ensure the real-time nature and transparency of HCI interaction.

[0017] The business management layer employs an Agent-first architecture as its core scheduling logic, integrating speech recognition, session management, a knowledge base module (RAG knowledge base module), a target agent, and a robot client module. It is responsible for parsing user interaction information, identifying intent, decomposing tasks, and generating control commands. Upon entering the system, user interaction information undergoes type detection. For speech input, speech-to-text processing is performed, converting the speech signal into standardized text commands. All text-formatted user commands are routed to the target agent. The target agent retrieves historical session records stored in the session management module to construct the session context, and simultaneously calls the RAG knowledge base to retrieve relevant professional auxiliary knowledge. Combining the session context and auxiliary knowledge information, it accurately identifies the user's intent. Furthermore, the agent utilizes the MCP protocol to discover cross-layer tools, autonomously planning tool call sequences to achieve multi-tool collaborative orchestration. The robot client module includes single-arm and dual-arm robot clients, which connect to the underlying robotic arm control service via REST APIs to issue robotic arm task commands and execute tasks. The agent's reasoning process and task planning results are pushed to the presentation layer via SSE for real-time feedback throughout the process.

[0018] Among them, the MCP service layer is an intermediate protocol adaptation layer that connects the business management layer and the device control layer, and is used to achieve unified abstraction, standardized encapsulation, and life cycle management of the system's hardware tool capabilities; through the MCP bridging service, the capabilities are encapsulated, and various tools are registered, discovered, and scheduled and managed relying on the tool registry (i.e., the MCP tool registry). The MCP service layer abstracts the manipulator operation capabilities into standardized MCP tools, including functions such as gripper opening and closing, manipulator linear motion, circular arc trajectory motion, force control detection, and device status query; at the same time, it encapsulates the vision device capabilities into a standard MCP tool interface, supporting capabilities such as real-time image acquisition of the camera, depth information detection, and vision data acquisition; through the collaborative orchestration and invocation of the manipulator tool and the vision perception tool, the vision perception and manipulator operation are linked, supporting complex operation scenarios such as target positioning, target recognition, and intelligent grasping.

[0019] Among them, the device control layer is used to connect to physical hardware devices, and relies on the device manufacturer's native SDK to complete hardware communication, device control, and raw data acquisition. The device control layer includes a robot controller module and a camera service module. Among them, the robot controller module is responsible for parsing the control instructions issued by the upper layer, implementing actions such as manipulator linear motion, circular arc trajectory motion, gripper opening and closing, and force control detection, and at the same time, it collects hardware feedback information such as manipulator pose, operating status, and force data in real time; the camera service module is responsible for driving the vision camera device to complete real-time image acquisition, environmental depth detection, and vision raw data acquisition. The device control layer uniformly encapsulates the different hardware interfaces, provides a standardized call interface to the upper layer MCP service layer, shields the underlying hardware communication differences, and provides reliable hardware driving capabilities and raw perception data support for upper layer intelligent decision-making, tool invocation, and operation execution.

[0020] Please refer to Figure 1B , the embodiment of the present application provides an application environment diagram of a manipulator intelligent control method. Among them, the terminal 102 communicates with the server 104 through a communication network. The data storage system can store the data that the server 104 needs to process. The data storage system can be integrated on the server 104, or placed in the cloud or other network servers. The server 104 and / or the terminal 102 are also communicatively connected to a manipulator, a camera, a microphone, and a speaker. Among them, the manipulator is an automated execution device with multi-degree-of-freedom motion capabilities, which can receive control instructions and execute corresponding physical motions, and at the same time can collect its own motion state data (such as joint angles, end poses, etc.) through built-in sensors.

[0021] Terminal 102 can be, but is not limited to, various personal computers, laptops, smartphones, tablets, drones, low-altitude aircraft, IoT devices, and portable wearable devices. IoT devices can include smart speakers, smart TVs, smart air conditioners, smart in-vehicle devices, and projection equipment. Portable wearable devices can include smartwatches, smart bracelets, and head-mounted displays. Head-mounted displays can be virtual reality (VR) devices, augmented reality (AR) devices, and smart glasses. Server 104 can be a standalone server, a server cluster or distributed system composed of multiple servers, or a cloud server providing cloud computing services.

[0022] like Figure 2 As shown in the figure, this application provides a flowchart of an intelligent control method for a robotic arm, which is applied to... Figure 1A The target intelligent agent in Figure 1B The method will be illustrated using terminal 102 or server 104 as examples. It is understood that the computer device may include at least one of a terminal and a server. The method includes the following steps: S101. Obtain the target user's natural language information and historical conversation records.

[0023] In some embodiments, the intelligent control system for the robotic arm further includes a human-machine interface and a session management module, S101, obtaining the natural language information and historical session records of the target user, including: A1. Receive interactive information input by the target user through a human-computer interaction interface; the interactive information includes text information or voice information. A2. Determine natural language information based on interactive information; A3. Retrieve the target user's historical session records from the session management module.

[0024] The human-computer interaction interface (HCI) is a web front-end interactive page deployed on the presentation layer. Users can input information through text input boxes or voice recording buttons provided by the HCI. When the user selects text input, the front-end captures the text information entered by the user in real time, performs format cleaning and illegal character filtering on the text information, and generates a standardized text data stream. When the user selects voice input, the front-end calls the browser's audio acquisition interface to collect the user's simulated voice signal in real time, and encodes the simulated voice signal into a digital audio stream for temporary storage. Furthermore, the HCI adds a unique session identifier, user timestamp, and device number to the collected text or voice information, generating an original interactive data packet (i.e., interactive information) with traceability information, completing the collection and encapsulation of interactive information, thereby achieving compatible reception of interactive information of different input types.

[0025] If the interaction information is text, the cleaned and standardized text is directly used as natural language information; if the interaction information is voice, the encapsulated audio data stream is transmitted to the voice recognition module of the business management layer. The voice recognition module performs noise reduction, voice feature extraction and text decoding on the audio signal, converting the unstructured voice signal into structured text content, thereby obtaining the corresponding natural language information.

[0026] The session management module includes a built-in session database for persistently storing user commands, agent inference content, tool call records, and robotic arm execution results for each interaction. Data is categorized and stored using a unique session identifier as an index. After acquiring natural language information, the system extracts the unique session identifier from the current interaction data packet and performs a precise search within the session management module based on this identifier, filtering out historical session records under the same task session. These historical session records include contextual data such as historical user natural language commands, historical intent recognition results, historical tool call sequences, and device execution feedback.

[0027] It is evident that by using text / voice multimodal interactive input, unified speech transcription processing, and conversation context retrieval, multi-source input and continuous conversation understanding of user natural language intent are achieved, providing complete and standardized interactive data support for subsequent agent intent parsing.

[0028] S102. Perform end-to-end autonomous reasoning analysis based on natural language information and historical conversation records to obtain the target user's intent.

[0029] In some embodiments, the intelligent control system for the robotic arm further includes a knowledge base module, S102, which performs end-to-end autonomous reasoning analysis based on natural language information and historical conversation records to obtain the target user's intent, including: B1. Construct a session context based on historical session records; B2. Retrieve auxiliary knowledge information corresponding to the natural language information from the knowledge base module; B3. Perform semantic parsing on natural language information to obtain semantic parsing results; B4. Determine the target user's intent based on the conversation context, auxiliary knowledge information, and semantic parsing results.

[0030] This process involves filtering and redundancy cleansing historical session records, removing invalid empty data, duplicate commands, and abnormal error logs, while retaining logically related interaction data. The historical session records are then sorted chronologically according to timestamps to reconstruct the logical sequence of user interactions. Finally, the cleaned and sorted historical user commands, historical agent inference results, tool call records, and robotic arm execution feedback information are structurally assembled to construct a continuous, complete, and logically sequenced session context.

[0031] The process involves converting natural language information into vector retrieval features, specifically generating high-dimensional semantic vectors through semantic vector encoding. These high-dimensional semantic vectors serve as retrieval indexes for similarity matching within the knowledge base module. The knowledge base module pre-processes the vectorization of professional knowledge, such as robotic arm operation terminology, object grasping rules, visual recognition parameters, and motion constraints, into a database. A pre-set similarity threshold is used to filter out related knowledge fragments that exceed this threshold. These filtered related knowledge fragments are then integrated, deduplicated, and structurally encapsulated to generate auxiliary knowledge information, providing professional knowledge constraints for subsequent semantic parsing and intent determination.

[0032] First, the text content of the natural language information is processed through word segmentation, part-of-speech tagging, and entity extraction to accurately identify key semantic elements contained in the text content. These key semantic elements include entity information such as the operation object, action instructions, work location, movement requirements, and constraints. Simultaneously, the target agent combines syntactic structure analysis to determine the logical relationships and modification / qualification relationships of statements, distinguishing between core instructions and irrelevant redundant statements. Furthermore, semantic completion and ambiguity resolution are performed for vague, colloquial, and abbreviated user instructions, correcting deviations in spoken expression. Finally, a semantic parsing result containing entity information, action logic, semantic weights, and ambiguity annotations is generated, completing the conversion of unstructured natural language into structured semantic data.

[0033] The target agent integrates three types of information for comprehensive reasoning and judgment. It uses the sequential conversational context as the semantic logic basis to determine whether user commands involve related needs such as task continuation, command modification, or action termination. It also uses auxiliary knowledge retrieved from the knowledge base as a professional constraint basis, combining the physical movement limitations of the robotic arm and tool usage specifications to correct unreasonable semantic judgment results. Finally, it combines refined semantic parsing results to pinpoint the core operation commands. Through multi-dimensional information fusion reasoning, the target agent hierarchically classifies user intentions, distinguishing between ordinary dialogue interaction intentions, simple single-step operation intentions, and complex multi-tool collaborative operation intentions, ultimately outputting standardized and structured target user intentions. These target user intentions clearly record the task type, operation object, execution requirements, and task constraints, providing accurate decision-making basis for subsequent toolset matching and tool call sequence planning.

[0034] It is evident that by combining conversational context, auxiliary knowledge information, and semantic parsing results to determine user intent from multiple dimensions, the accuracy and scenario adaptability of natural language instruction understanding are significantly improved, effectively reducing ambiguity and misjudgment, and providing a reliable intent basis for subsequent task planning.

[0035] S103. Initiate a tool discovery request to the tool registry via the MCP protocol, and determine the robotic arm operation tool set and the visual perception tool set according to the tool discovery request.

[0036] In some embodiments, S103, determining the robotic arm operation tool set and the visual perception tool set according to the tool discovery request, includes: C1. Based on the tool discovery request, determine the list of callable tools corresponding to the tool registry; C2. From the list of available tools, retrieve at least one robotic arm operation tool and at least one visual perception tool to form a robotic arm operation tool set and a visual perception tool set, respectively.

[0037] The robotic arm intelligent control system can build a standardized tool management system based on the MCP protocol. This tool registry, deployed within the MCP service layer, serves as a structured tool index database. The tool registry stores the unique tool identifier, function description, input parameters, compatible hardware type, calling permissions, and interface mapping address for all tools, with all tools undergoing unified semantic encapsulation. First, the business management layer sends a tool discovery request to the MCP service layer, carrying the target agent's identity and permission identifier. The MCP service layer performs permission verification based on this identifier to match the list of callable tools associated with the current target agent. Different types of agents can be configured with differentiated tool permissions, achieving isolated control of tool resources.

[0038] The target agent reads the list of callable tools based on the identity and permission identifier, and combines this with the tool metadata stored in the list to complete tool filtering and classification. The robotic arm operation tools include gripper opening and closing control tools, linear motion control tools, circular trajectory motion control tools, force control detection tools, and equipment status query tools, used to complete the physical operations of the robotic arm. The visual perception tools include real-time camera image acquisition tools, depth information detection tools, and visual target data acquisition tools, used to complete environmental perception, target recognition, and position detection. The target agent retrieves at least one robotic arm operation tool from the list of callable tools, integrating them to form a robotic arm operation tool set; it also retrieves at least one visual perception tool, integrating it to form a visual perception tool set.

[0039] It is evident that by matching permissions to locate the list of callable tools and aggregating robotic arm operation and visual perception toolsets as needed, the standardization and scalable management of tool capabilities are achieved, providing a compliant and efficient tool resource pool for subsequent dynamic tool orchestration of intelligent agents.

[0040] S104. Determine the tool call sequence based on the robotic arm operation toolset, the visual perception toolset, and the target user's intent.

[0041] like Figure 3 As shown, this application embodiment provides a flowchart for determining a tool invocation sequence. S104, determining the tool invocation sequence based on the robotic arm operation tool set, the visual perception tool set, and the target user intent includes: D1. Break down the target user's intent into multiple sub-tasks; D2. Based on the execution logic of each subtask in the multiple subtasks, match the corresponding target tools from the robotic arm operation tool set and the visual perception tool set respectively to obtain the target tool set; D3. Based on the execution order and dependencies of each subtask in the multiple subtasks, sort the target tools in the target tool set to obtain the tool call sequence.

[0042] The target agent reads the structured user intent, analyzes the task objective, target object, execution constraints, and environmental conditions of the robotic arm operation, and decomposes the overall task hierarchically according to the execution logic, resulting in several logically independent, single-granularity subtasks. During this decomposition, the target agent combines the physical constraints of the robotic arm operation with industrial execution logic to break down the complex continuous operation into perception-based subtasks, motion-based subtasks, execution-based subtasks, and state verification subtasks; it also clarifies the task objective, input conditions, execution requirements, and output results of each subtask. For example, the grasping task is decomposed into multiple subtasks such as environmental perception, target localization, robotic arm approach, gripper closure, and posture verification. Furthermore, all decomposed subtasks are structured and numbered, preserving the pre-dependencies between them.

[0043] The target agent iterates through each subtask, matching tools based on the execution logic, functional attributes, and execution requirements of each subtask. For perception-related subtasks, the target agent selects visual tools such as image acquisition and depth detection from the visual perception tool set; for motion, gripping, and force control subtasks, the target agent matches operation tools such as trajectory motion, gripper control, and force control detection from the robotic arm operation tool set. Each subtask is then bound to a functionally compatible target tool, and all matched target tools are aggregated to generate a target toolset.

[0044] The target agent constructs a task execution topology based on the pre-constraints, execution order, and dependencies between each sub-task. Following the principles of workflow rationality, hardware motion safety, and operational logic fluency, various target tools in the target toolkit are sequentially arranged. For execution logic involving visual perception as a pre-process and motion execution as a post-process, perception tools such as visual acquisition and depth detection are prioritized, followed by operation tools such as robotic arm approach, gripper opening and closing, and trajectory movement, ensuring a closed-loop workflow with perception preceding execution. After sorting, the target agent encapsulates the tool call sequence, generating a tool call sequence with execution timing, call parameters, and dependency identifiers. This tool call sequence explicitly records the timing of each tool call, input parameter format, and execution order, providing an accurate basis for subsequent generation of robotic arm target control commands.

[0045] It is evident that by breaking down user intent into subtasks, accurately matching the corresponding tools, and then generating a tool call sequence based on dependencies, the structured decomposition and automated orchestration of complex tasks are achieved, freeing users from the constraints of fixed scripts and improving the flexibility and robustness of task execution.

[0046] S105. Generate target control instructions based on the tool call sequence to drive the robotic arm to perform operation tasks corresponding to the natural language information.

[0047] In some embodiments, S105, generating target control instructions based on the tool invocation sequence, includes: E1. Determine the execution sequence number and execution parameters of each target tool in the tool call sequence to obtain the execution parameter sequence; E2. Convert the sequence of execution parameters into standard control instructions that the robotic arm can execute, and obtain the target control instructions.

[0048] The execution sequence number indicates the order in which each target tool is called in the task execution process. The execution parameters include the input parameters required for each target tool to execute, such as the target space coordinates of the robotic arm movement, the movement speed, the force control threshold, and the gripper opening and closing force.

[0049] For each invoked target tool, the target agent determines the corresponding execution parameters by combining the current task requirements, visual perception data, session context, and the physical constraints of the robotic arm. For visual perception tools, execution parameters include perception parameters such as acquisition frame rate, exposure parameters, detection depth range, and recognition threshold; for robotic arm operation tools, execution parameters include control parameters such as motion coordinates, motion speed, motion acceleration, gripper opening and closing angle, force control threshold, and motion trajectory curvature. The target agent verifies, filters, and normalizes the parameters of all target tools, eliminating illegal parameters that exceed hardware safety thresholds, and summarizing and encapsulating all compliant execution parameters to form a structured sequence of execution parameters.

[0050] The target intelligent agent parses, encodes, and converts semantic tool parameters in the parameter sequence into low-level binary instructions or SDK standard instruction formats that can be recognized by the device control layer, shielding the communication protocol differences between different models of robotic arms and camera devices. Specifically, motion parameters are converted into trajectory motion instructions, gripper parameters into opening and closing control instructions, and force control parameters into force control adjustment instructions. After instruction conversion, all low-level instructions undergo timing encapsulation, instruction verification, and format encryption to generate target control instructions adapted to the current task. These target control instructions can be directly recognized and parsed by the device control layer to drive the robotic arm to complete physical operations such as grasping, placing, moving, and force control.

[0051] As can be seen, by determining the sequence of execution parameters based on the execution number and execution parameters of each target tool and converting them into standard control instructions that the robotic arm can recognize, the upper-level semantic tool call sequence is converted into the lower-level hardware instructions, ensuring the legality, security and executability of the control instructions, and opening up a closed-loop link between intelligent decision-making and physical execution.

[0052] In some embodiments, the method further includes: The tool call results and task execution status of the target intelligent agent are streamed and pushed to the human-computer interaction interface for real-time display. Write back the tool execution results and robotic arm status of the target agent to the session context to trigger subsequent reasoning or exception handling of the target agent.

[0053] Throughout the entire process of the robotic arm performing its tasks, the system continuously collects the tool invocation results and task execution status of the target intelligent agent. Tool invocation results include, but are not limited to: tool invocation name, invocation sequence, input and output parameters, and tool execution return data. Task execution status includes, but is not limited to: model inference progress, gripper opening and closing status, task completion progress, and equipment anomaly alarm information. The system utilizes SSE streaming technology built into the presentation layer to segment and encode the collected data, pushing it frame by frame to achieve streaming output of tool invocation results and task execution status. Simultaneously, it incorporates a WebSocket bidirectional communication mechanism to ensure low-latency transmission of hardware status messages. After receiving the real-time data stream, the human-machine interface visualizes the text information, status logs, and equipment data, displaying them through a combination of text logs, status icons, and real-time visuals.

[0054] After the robotic arm completes its task, it transmits the tool execution results (such as gripper closure, robotic arm reaching the target position, force control detection values, etc.) and the robotic arm status (such as joint angles, end effector pose, abnormal alarm information, etc.) back to the target agent via the MCP protocol. Upon receiving this information, the target agent structures it into the current session context as input for subsequent reasoning. When the tool execution result matches expectations, the target agent continues to execute subsequent steps in the tool call sequence based on the updated status information in the session context. When an execution anomaly is detected (such as grasping failure, collision alarm, force control exceeding limits, etc.), the target agent autonomously reasons based on the anomaly information in the session context, dynamically adjusting the subsequent tool call sequence, such as replanning the grasping path, adjusting force control parameters, or performing anomaly recovery operations, forming a closed-loop control system of perception-reasoning-execution-feedback.

[0055] It is evident that not only are tool call results and task status pushed to the human-computer interaction interface in real time through streaming output, improving the transparency of interaction and user experience, but the tool execution results and robotic arm status are also written back to the session context to trigger subsequent reasoning or anomaly handling of the target intelligent agent, forming a closed-loop control of perception-reasoning-execution-feedback, and enhancing the system's anomaly adaptation capability.

[0056] like Figure 4As shown in the illustration, this application provides a flowchart of the reasoning execution of a target intelligent agent. The system receives text or voice input from the user. If the input is voice, it is first converted to text format by a speech-to-text module. All inputs are ultimately sent to the target intelligent agent in text form. The target intelligent agent performs comprehensive processing by combining three types of information: retrieving historical conversation records to construct a conversational context for understanding the language environment; retrieving auxiliary knowledge from the knowledge base to provide professional constraints for reasoning; and relying on the MCP protocol to call the device control interface to complete the robotic arm operation. The reasoning process of the target intelligent agent, the results of tool calls, and the device execution status are presented to the user in real time in a visual manner, forming an interactive closed loop.

[0057] It should be understood that although the steps in the flowcharts of the above embodiments are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the above embodiments may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages in other steps.

[0058] Based on the same inventive concept, this application also provides an intelligent control device for a robotic arm. The solution provided by this device is similar to the solution described in the above method. Therefore, the specific limitations of one or more embodiments of the intelligent control device for a robotic arm provided below can be found in the limitations of the intelligent control method for a robotic arm described above, and will not be repeated here.

[0059] like Figure 5 As shown, this application provides a robotic arm intelligent control device 500, applied to a target intelligent agent in a robotic arm intelligent control system. The system also includes a robotic arm, and the target intelligent agent is a large language model intelligent agent. The device 500 includes: Module 501 is used to obtain the natural language information and historical conversation records of the target user; Analysis module 502 is used to perform end-to-end autonomous reasoning analysis based on natural language information and historical conversation records to obtain the target user's intent; The first determination module 503 is used to initiate a tool discovery request to the tool registry through the MCP protocol, and determine the robotic arm operation tool set and the visual perception tool set according to the tool discovery request. The second determining module 504 is used to determine the tool calling sequence based on the robotic arm operation tool set, the visual perception tool set, and the target user's intent; The generation module 505 is used to generate target control instructions based on the tool call sequence to drive the robotic arm to perform operation tasks corresponding to natural language information.

[0060] In some embodiments, the intelligent control system for the robotic arm further includes a human-machine interface and a session management module; In obtaining the natural language information and historical conversation records of the target user, module 501 is specifically used for: The system receives interactive information input from the target user through a human-computer interaction interface; the interactive information includes text information or voice information. Determine natural language information based on interaction information; Retrieve the target user's historical session records from the session management module.

[0061] In some embodiments, the intelligent control system for the robotic arm also includes a knowledge base module; In terms of end-to-end autonomous reasoning analysis based on natural language information and historical conversation records to obtain the target user's intent, the analysis module 502 is specifically used for: Construct a session context based on historical session records; Retrieve auxiliary knowledge information corresponding to the natural language information from the knowledge base module; Semantic parsing is performed on natural language information to obtain semantic parsing results; Based on the conversation context, auxiliary knowledge information, and semantic parsing results, the target user's intent is determined.

[0062] In some embodiments, in determining the robotic arm manipulation toolset and the visual perception toolset respectively based on the tool discovery request, the first determining module 503 is specifically configured to: Based on the tool discovery request, determine the list of callable tools corresponding to the tool registry; From the list of available tools, retrieve at least one robotic arm operation tool and at least one visual perception tool to form a robotic arm operation tool set and a visual perception tool set, respectively.

[0063] In some embodiments, the second determining module 504 is specifically configured to: determine the tool invocation sequence based on the robotic arm operation toolset, the visual perception toolset, and the target user's intent. The target user's intent is broken down into multiple sub-tasks; Based on the execution logic of each subtask in the multiple subtasks, the corresponding target tools are matched from the robotic arm operation tool set and the visual perception tool set respectively to obtain the target tool set; Based on the execution order and dependencies of each subtask in the multiple subtasks, the target tools in the target toolset are sorted to obtain the tool call sequence.

[0064] In some embodiments, in generating target control instructions based on a tool invocation sequence, the generation module 505 is specifically configured to: Determine the execution sequence number and execution parameters of each target tool in the tool call sequence to obtain the execution parameter sequence; The sequence of execution parameters is converted into standard control instructions that the robotic arm can execute, thus obtaining the target control instructions.

[0065] In some embodiments, the device 500 is further configured to: The tool call results and task execution status of the target intelligent agent are streamed and pushed to the human-computer interaction interface for real-time display. Write back the tool execution results and robotic arm status of the target agent to the session context to trigger subsequent reasoning or exception handling of the target agent.

[0066] Each module in the aforementioned intelligent control device 500 for the robotic arm can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device in hardware form, or stored in the memory of a computer device in software form, so that the processor can call and execute the operations corresponding to each module.

[0067] In some embodiments, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 6 As shown, the computer device includes a processor, memory, input / output (I / O) interfaces, and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is also connected to the system bus via the I / O interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The database stores data related to the intelligent control method for the robotic arm. The I / O interfaces are used for exchanging information between the processor and external devices. The communication interface is used for communication with external terminals via a network connection. When the computer program is executed by the processor, it implements the steps in the aforementioned intelligent control method for the robotic arm.

[0068] In some embodiments, a computer device is provided, which may be a terminal, and its internal structure diagram may be as follows: Figure 7 As shown, the computer device includes a processor, memory, input / output interface, communication interface, display unit, and input device. The processor, memory, and input / output interface are connected via a system bus, and the communication interface, display unit, and input device are also connected to the system bus via the input / output interface. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage media. The input / output interface is used for exchanging information between the processor and external devices. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, mobile cellular networks, NFC (Near Field Communication), or other technologies. When the computer program is executed by the processor, it implements the steps in the aforementioned intelligent control method for the robotic arm. The display unit is used to form a visually visible image and can be a display screen, projection device, or virtual reality imaging device. The display screen can be an LCD screen or an e-ink screen; the input device of the computer device can be a touch layer covering the display screen, or buttons, trackballs or touchpads set on the casing of the computer device, or external keyboards, touchpads or mice, etc.

[0069] Those skilled in the art will understand that Figure 6 or Figure 7 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0070] In some embodiments, a computer device is provided, the computer device including a memory and a processor, the memory storing a computer program, the processor executing the computer program to implement the steps in the above method embodiments.

[0071] In some embodiments, such as Figure 8 The diagram shows the internal structure of a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps described in the above-described method embodiments.

[0072] In some embodiments, a computer program product is provided, which includes a computer program that, when executed by a processor, implements the steps in the above method embodiments.

[0073] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0074] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0075] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0076] The above embodiments are merely illustrative of several implementation methods of this application, and their descriptions are relatively specific and detailed. However, they should not be construed as limiting the scope of this application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A method for intelligent control of a robotic arm, characterized in that, A target intelligent agent is applied to a robotic arm intelligent control system, the system further including a robotic arm, the target intelligent agent being a large language model intelligent agent, the method comprising: Obtain the target user's natural language information and historical conversation records; Based on the natural language information and the historical conversation records, end-to-end autonomous reasoning analysis is performed to obtain the target user's intent; A tool discovery request is initiated to the tool registry via the MCP protocol, and the robotic arm operation tool set and the visual perception tool set are determined according to the tool discovery request. The tool invocation sequence is determined based on the robotic arm operation toolset, the visual perception toolset, and the target user intent; The target control instructions are generated according to the tool call sequence to drive the robotic arm to perform the operation task corresponding to the natural language information.

2. The method according to claim 1, characterized in that, The system also includes a human-computer interaction interface and a session management module; The acquisition of the target user's natural language information and historical conversation records includes: The human-computer interaction interface receives interactive information input by the target user; the interactive information includes text information or voice information. Natural language information is determined based on the interaction information; The historical session records of the target user are retrieved from the session management module.

3. The method according to claim 2, characterized in that, The system also includes a knowledge base module; The step of performing end-to-end autonomous reasoning analysis based on the natural language information and the historical conversation records to obtain the target user's intent includes: Construct a session context based on the historical session records; Retrieve auxiliary knowledge information corresponding to the natural language information from the knowledge base module; The natural language information is semantically parsed to obtain the semantic parsing result; The target user's intent is determined based on the session context, the auxiliary knowledge information, and the semantic parsing results.

4. The method according to claim 2, characterized in that, The step of determining the robotic arm operation tool set and the visual perception tool set according to the tool discovery request includes: Based on the tool discovery request, determine the list of callable tools corresponding to the tool registry; From the list of available tools, at least one robotic arm operation tool and at least one visual perception tool are retrieved to form a robotic arm operation tool set and a visual perception tool set, respectively.

5. The method according to claim 4, characterized in that, The step of determining the tool invocation sequence based on the robotic arm operation toolset, the visual perception toolset, and the target user intent includes: The target user's intent is broken down into multiple sub-tasks; Based on the execution logic of each of the multiple subtasks, the corresponding target tools are matched from the robotic arm operation tool set and the visual perception tool set respectively to obtain the target tool set; Based on the execution order and dependencies of each subtask in the plurality of subtasks, the target tools in the target toolset are sorted to obtain a tool call sequence.

6. The method according to claim 5, characterized in that, The step of generating target control instructions based on the tool call sequence includes: Determine the execution sequence number and execution parameters of each target tool in the tool call sequence to obtain the execution parameter sequence; The sequence of execution parameters is converted into standard control instructions that the robotic arm can execute, thus obtaining the target control instructions.

7. The method according to any one of claims 3-6, characterized in that, The method further includes: The tool call results and task execution status of the target intelligent agent are streamed and pushed to the human-computer interaction interface for real-time display. The tool execution results and robotic arm status of the target agent are written back to the session context to trigger subsequent reasoning or exception handling of the target agent.

8. An intelligent control device for a robotic arm, characterized in that, A target intelligent agent applied to a robotic arm intelligent control system, the system further comprising a robotic arm, the target intelligent agent being a large language model intelligent agent, the device comprising: The acquisition module is used to obtain the target user's natural language information and historical conversation records; The analysis module is used to perform end-to-end autonomous reasoning analysis based on the natural language information and the historical conversation records to obtain the target user's intent; The first determination module is used to initiate a tool discovery request to the tool registry through the MCP protocol, and determine the robotic arm operation tool set and the visual perception tool set according to the tool discovery request. The second determining module is used to determine the tool calling sequence based on the robotic arm operation tool set, the visual perception tool set, and the target user intent; The generation module is used to generate target control instructions based on the tool call sequence to drive the robotic arm to perform operation tasks corresponding to the natural language information.

9. A computer device comprising a memory and a processor, the memory storing a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 7.

Images