Method, device and system for controlling an agent, and electronic device
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- BEIJING LUSTER LIGHTTECH
- Filing Date
- 2026-02-24
- Publication Date
- 2026-07-14
Smart Images
Figure CN122387697A_ABST
Abstract
Description
Technical Field
[0001] This application belongs to the field of intelligent agent technology, and in particular relates to a control method, device, system and electronic device for an intelligent agent. Background Technology
[0002] Currently, the development of cross-domain control systems such as vision-guided systems typically relies on step-by-step development and manual integration. For example, the vision team and the motion control team develop independently using their respective tools, and then bridge the gap by agreeing on communication protocols, data formats, and signal interaction points, and writing additional communication code. This development model struggles to meet the demands for efficient agent collaboration and system flexibility in complex and dynamic scenarios. Summary of the Invention
[0003] This invention aims to address at least one of the technical problems existing in the prior art. To this end, this invention proposes a control method, device, system, and electronic device for intelligent agents, which can reduce development collaboration and debugging costs, and improve the task adaptability, overall flexibility, and reusability of intelligent agents in complex and ever-changing scenarios.
[0004] Firstly, this application provides a method for controlling an intelligent agent, the method comprising:
[0005] Receive and parse the control information input by the user to the intelligent agent to obtain at least two cross-domain sub-task processes of the intelligent agent; Based on cross-domain collaboration rules, data conversion interfaces and event synchronization interfaces are generated between the various sub-task processes. The cross-domain collaboration rules are used to define the collaborative working methods between the various sub-task processes in task decomposition, data conversion and event synchronization. The agent is controlled to run based on each of the subtask processes and the data conversion interface and event synchronization interface between them.
[0006] According to the control method of the intelligent agent in this application, by introducing cross-domain collaboration rules, the control information input by the user to the intelligent agent is automatically parsed into multiple cross-domain sub-task processes, and corresponding data conversion interfaces and event synchronization interfaces are generated. This realizes semantic alignment and execution collaboration between cross-domain processes such as vision and motion, thereby reducing development collaboration and debugging costs, and improving the task adaptability, overall flexibility and reusability of the intelligent agent in complex and ever-changing scenarios.
[0007] According to one embodiment of this application, controlling the operation of the intelligent agent based on the data conversion interface and the event synchronization interface between each subtask process and the subtask processes includes: Based on each of the subtask processes and the data conversion interface and event synchronization interface between the subtask processes, a collaborative control diagram of the agent is generated. The collaborative control diagram is used to indicate the way in which the subtask processes work together in task decomposition, data conversion and event synchronization. The operation of the intelligent agent is controlled based on the aforementioned collaborative control diagram.
[0008] According to one embodiment of this application, controlling the operation of the intelligent agent based on the cooperative control diagram includes: The collaborative control graph is compiled into an executable program corresponding to each of the sub-task processes; The communication configuration files between each of the executable programs and each of the subtask processes are executed to control the operation of the agent.
[0009] According to one embodiment of this application, after generating the cooperative control graph of the agent, the method further includes: The collaborative control graph is visualized, and the operation of the intelligent agent is simulated based on the visualization results.
[0010] According to one embodiment of this application, at least two of the sub-task processes are divided into a visual domain task process and a motion control domain task process.
[0011] According to one embodiment of this application, the cross-domain collaboration rule is further used to define the exception handling methods between the various sub-task processes, and the control of the intelligent agent's operation includes: During the operation of the intelligent agent, if any of the sub-task processes encounters an anomaly, the other related sub-task processes will be stopped and an alarm mechanism will be triggered.
[0012] Secondly, this application provides a control device for an intelligent agent, comprising: The first processing module is used to receive and parse the control information input by the user to the intelligent agent, and obtain at least two cross-domain sub-task processes of the intelligent agent; The second processing module is used to generate data conversion interfaces and event synchronization interfaces between the various sub-task processes based on cross-domain collaboration rules. The cross-domain collaboration rules are used to define the collaborative working methods between the various sub-task processes in task decomposition, data conversion and event synchronization. The third processing module is used to control the operation of the intelligent agent based on each of the sub-task processes and the data conversion interface and event synchronization interface between the sub-task processes.
[0013] According to the control device of the intelligent agent in this application, by introducing cross-domain collaboration rules, the control information input by the user to the intelligent agent is automatically parsed into multiple cross-domain sub-task processes, and corresponding data conversion interfaces and event synchronization interfaces are generated. This realizes semantic alignment and execution collaboration between cross-domain processes such as vision and motion, thereby reducing development collaboration and debugging costs, and improving the task adaptability, overall flexibility and reusability of the intelligent agent in complex and ever-changing scenarios.
[0014] Thirdly, this application provides a control system for an intelligent agent, used to execute the control method for the intelligent agent described in the first aspect above.
[0015] According to the intelligent agent control system of this application, by introducing cross-domain collaboration rules, the control information input by the user to the intelligent agent is automatically parsed into multiple cross-domain sub-task processes, and corresponding data conversion interfaces and event synchronization interfaces are generated. This realizes semantic alignment and execution collaboration between cross-domain processes such as vision and motion, thereby reducing development collaboration and debugging costs, and improving the task adaptability, overall flexibility and reusability of the intelligent agent in complex and ever-changing scenarios.
[0016] According to one embodiment of this application, the control system of the intelligent agent includes: A distributed compilation module is used to compile the collaborative control graph into an executable program corresponding to each of the subtask processes.
[0017] Fourthly, this application provides an electronic device including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the control method of the intelligent agent as described in the first aspect above.
[0018] Fifthly, this application provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the control method of the intelligent agent as described in the first aspect above.
[0019] In a sixth aspect, this application provides a computer program product, including a computer program that, when executed by a processor, implements the control method for an intelligent agent as described in the first aspect above.
[0020] Additional aspects and advantages of this application will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of this application. Attached Figure Description
[0021] The above and / or additional aspects and advantages of this application will become apparent and readily understood from the description of the embodiments taken in conjunction with the following drawings, in which: Figure 1This is a flowchart illustrating the control method for an intelligent agent provided in an embodiment of this application; Figure 2 This is a schematic diagram of the structure of the control device for the intelligent agent provided in the embodiments of this application; Figure 3 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application. Detailed Implementation
[0022] The technical solutions of the embodiments of this application will be clearly described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application are within the scope of protection of this application.
[0023] The terms "first," "second," etc., used in the specification and claims of this application are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such use of data can be interchanged where appropriate so that embodiments of this application can be implemented in orders other than those illustrated or described herein, and the objects distinguished by "first," "second," etc., are generally of the same class and the number of objects is not limited; for example, a first object can be one or more. Furthermore, in the specification and claims, "and / or" indicates at least one of the connected objects, and the character " / " generally indicates that the preceding and following objects are in an "or" relationship.
[0024] The following description, in conjunction with the accompanying drawings, details the control method, control device, control system, electronic device, and readable storage medium for the intelligent agent provided in this application, through specific embodiments and application scenarios.
[0025] The control method for the intelligent agent can be applied to the terminal, and can be executed by the hardware or software in the terminal.
[0026] The intelligent agent control method provided in this application embodiment can be executed by an electronic device or a functional module or functional entity in an electronic device that can implement the intelligent agent control method. The electronic devices mentioned in this application embodiment include, but are not limited to, mobile phones, tablets, computers, cameras and wearable devices. The following uses an electronic device as the execution subject to illustrate the intelligent agent control method provided in this application embodiment.
[0027] In this embodiment, the intelligent agent is an intelligent entity that can perceive the environment, understand the task, make autonomous decisions, and perform actions to achieve the goal, and can be a robot.
[0028] like Figure 1As shown, the control method of the intelligent agent includes steps 110, 120 and 130.
[0029] Step 110: Receive and parse the control information input by the user to the agent to obtain at least two cross-domain sub-task processes for the agent.
[0030] Among them, control information refers to instructions or commands input by the user, which are used to guide the intelligent agent to perform tasks; the subtask process is a sequence of steps that can be executed independently after the overall task is decomposed into different domains.
[0031] It should be noted that cross-domain tasks involve two or more modalities that are fundamentally different in function, representation or processing mechanism. For example, the visual domain is responsible for perceiving and understanding images and the motion domain is responsible for planning and executing actions. The two need to work together to complete the task but have different data formats, semantic spaces and operational logic.
[0032] In this step, the agent receives control information input by the user and, through semantic understanding and task decomposition, transforms it into at least two sub-task processes that operate in different domains and have clear objectives and execution logic.
[0033] The input form of control information can be diverse, including natural language instructions, structured instruction codes, voice instructions, or graphical operation instructions. The input control information is first preprocessed. For example, for natural language instructions, natural language processing steps such as word segmentation, part-of-speech tagging, and semantic role recognition are used to extract the core elements of the task, the execution object, action requirements, and constraints. For voice instructions, speech-to-text conversion is completed first, and then the semantic parsing process is executed. For graphical operation instructions, operation nodes, relationships, and parameter configurations are parsed to ensure accurate identification and interpretation of control information.
[0034] The task decomposition process follows the principles of domain independence and collaborative compatibility. Domain independence requires that each subtask process belong to only one specific domain, without cross-domain functional overlap, ensuring that each subtask process can be developed, tested, and optimized independently. Collaborative compatibility requires that the decomposed subtask processes can be connected through interfaces, without functional conflicts or semantic gaps. The decomposition process can employ a combination of rule-driven decomposition and intelligent learning decomposition. Rule-driven decomposition, based on preset domain division rules and task mapping tables, automatically matches common overall tasks to subtask processes within the corresponding domain. For example, the overall task of grasping an object can be directly decomposed into a visual domain object detection and localization subtask process and a motion domain path planning and action execution subtask process based on preset rules. Intelligent learning decomposition, on the other hand, adaptively decomposes complex and unknown tasks based on historical task decomposition data and model training results. By mining the correlation features between task elements and domain functions, it generates the optimal subtask process decomposition scheme.
[0035] The generation of subtask processes includes complete execution steps, input and output parameters, execution conditions, and execution time limits. For example, the target detection and localization subtask process in the visual domain explicitly includes execution steps such as image acquisition, image preprocessing, feature extraction, target recognition, pose calculation, and result output. The input parameters are the raw image data and calibration parameters acquired by the camera, and the output parameters are the spatial coordinates, pose angle, and recognition confidence of the target object. The execution conditions are that the camera is working normally and the lighting conditions meet the preset threshold. The execution time limit matches the overall task's time requirements to avoid the timeout of a single subtask process affecting the overall task progress. In addition, for complex tasks containing multiple cross-domain subtask processes, the dependencies between subtask processes are also marked, including sequential dependencies, parallel dependencies, or conditional dependencies. For example, path planning can only start after target detection is completed, which is a sequential dependency; visual data acquisition and device status monitoring can be executed simultaneously, which is a parallel dependency; if the target recognition confidence reaches the standard, grasping is performed; otherwise, detection is repeated, which is a conditional dependency. This provides a foundation for subsequent interface generation and collaborative control.
[0036] Step 120: Based on cross-domain collaboration rules, generate data conversion interfaces and event synchronization interfaces between various sub-task processes.
[0037] Among them, cross-domain collaboration rules are used to define the way in which various sub-task processes work together in terms of task decomposition, data transformation and event synchronization. That is, cross-domain collaboration rules are a set of preset or learned strategies and interface specifications that clarify the collaboration mechanisms of each sub-task process, such as visual perception, motion control and speech understanding, in terms of how tasks are decomposed and allocated, how heterogeneous data is mapped and transformed (e.g., converting image recognition results into robot motion parameters), and how key events are triggered and synchronized (e.g., the completion of target detection triggers the start of path planning). This ensures that the intelligent agent operates efficiently, consistently and reliably in multi-domain fusion scenarios.
[0038] The data conversion interface is used to map and adapt heterogeneous data formats and semantics between different domains, while the event synchronization interface is used to coordinate the execution sequence of each subtask process to ensure that the operations corresponding to the intelligent agent's task execution can be correctly triggered according to the preset dependencies.
[0039] For example, the overall task flow is as follows: after the robot sees a cup on the table, it autonomously moves over and grabs it. The vision domain sub-task flow captures images through a camera, detects and identifies the cup, and outputs its position and pose in the image. Through camera calibration and depth information, the pixel coordinates are converted into a three-dimensional spatial position in the robot's base coordinate system. The data conversion interface converts the three-dimensional target position and grasping orientation output by the vision domain into target pose instructions that the motion control domain can understand, completing semantic and format adaptation. The event synchronization interface is used to trigger the motion domain sub-task flow to start path planning and grasping actions when the vision system confirms that the cup has been successfully located and the confidence level meets the standard. If the vision does not detect the cup, no motion is triggered.
[0040] In this step, based on cross-domain collaboration rules, by parsing the semantic mapping and temporal dependencies between subtask processes, a data conversion interface adapted to the data format and an event synchronization interface that triggers the execution order are generated.
[0041] Step 130: Control the operation of the intelligent agent based on the data conversion interface and event synchronization interface between each subtask process and between each subtask process.
[0042] In this step, all subtask processes and their corresponding data conversion interfaces and event synchronization interfaces are integrated into a unified intelligent agent execution framework. During runtime, the intelligent agent activates each subtask process sequentially or concurrently based on the dependencies defined by the event synchronization interface. At the same time, cross-domain information is transmitted and adapted in real time through the data conversion interface to ensure that processes in different domains work collaboratively under the premise of semantic consistency and correct timing.
[0043] The unified execution framework has core functions such as task scheduling, resource management, data interaction, and status monitoring. The task scheduling function formulates an execution plan, allocates execution resources, and controls the start, pause, resumption, and termination of sub-task processes based on the dependencies and priorities of sub-task processes. For example, for sequentially dependent sub-task processes, they are started and executed in the order of dependency; for parallel dependent sub-task processes, independent execution threads are allocated and started simultaneously; for conditionally dependent sub-task processes, the start decision is made based on the condition judgment result.
[0044] The resource management function is responsible for the unified management and scheduling of the hardware and software resources of the intelligent agent. Hardware resources include processors, memory, sensors, actuators, etc., while software resources include algorithm models, program code, rule bases, etc. Resource management achieves load balancing to avoid overloading of a single resource. For example, when the processor load reaches a preset threshold, the execution of low-priority sub-tasks is paused to release processor resources. Simultaneously, resource management has fault detection and redundancy backup functions. When a hardware resource fails, it automatically switches to a redundant resource to ensure continuous task execution. For example, when the main camera fails, it switches to a backup camera to complete the image acquisition task. The data interaction function, based on a data conversion interface, enables real-time data transmission and interaction between cross-domain sub-tasks, constructing a data interaction link to ensure the accuracy, real-time performance, and security of data transmission. Data must be encrypted during data interaction to prevent data leakage or tampering. The data interaction function also needs a data caching mechanism. When the network is interrupted or data transmission is delayed, critical data is cached and transmission continues after the network is restored to avoid data loss.
[0045] The status monitoring function is responsible for real-time monitoring of the execution status, interface running status, and resource usage status of each subtask process. It collects monitoring data such as execution progress, execution results, interface response time, and resource utilization, and generates monitoring logs. If abnormal states are detected, the exception handling mechanism is triggered promptly. For example, subtask execution timeouts, interface response failures, and resource utilization exceeding thresholds are all considered abnormal states. The status monitoring function quickly locates the location and cause of the exception, providing support for exception handling. Furthermore, the unified execution framework is configurable and scalable, supporting dynamic configuration of subtask processes, interface parameters, and resource allocation strategies according to different task requirements. It also supports flexible integration of subtask processes and interfaces from new domains, adapting to complex scenarios involving multi-domain integration.
[0046] According to the control method of the intelligent agent provided in the embodiments of this application, by introducing cross-domain collaboration rules, the control information input by the user to the intelligent agent is automatically parsed into multiple cross-domain sub-task processes, and corresponding data conversion interfaces and event synchronization interfaces are generated. This realizes semantic alignment and execution collaboration between cross-domain processes such as vision and motion, thereby reducing development collaboration and debugging costs, and improving the task adaptability, overall flexibility and reusability of the intelligent agent in complex and ever-changing scenarios.
[0047] In some embodiments, the operation of the intelligent agent is controlled based on each subtask process and the data conversion interface and event synchronization interface between the subtask processes, including: Based on each subtask process and the data conversion interface and event synchronization interface between each subtask process, a collaborative control diagram of the intelligent agent is generated. Control the operation of intelligent agents based on collaborative control graphs.
[0048] The collaborative control diagram is used to indicate how the various sub-task processes work together in terms of task decomposition, data transformation, and event synchronization. In other words, the collaborative control diagram is a blueprint to guide the operation of the intelligent agent.
[0049] In this embodiment, a collaborative control diagram for the agent is first constructed based on each sub-task process and the data conversion and event synchronization interfaces between these sub-task processes. The collaborative control diagram depicts the task decomposition strategy, data exchange format and timing, and event synchronization mechanism of each sub-task process during execution, providing clear guidance for seamless collaboration between different functional modules. Subsequently, the generated collaborative control diagram guides the agent's operation, ensuring that each sub-task process can be executed efficiently and orderly according to predetermined cross-domain collaboration rules, and can dynamically adapt to environmental changes or task requirements, thereby achieving intelligent response and execution of complex tasks.
[0050] During the generation of the collaborative control diagram, the execution logic, interface parameters, and dependencies of each sub-task process are integrated. A visual modeling approach is used to construct the collaborative control diagram, clearly presenting the data flow path and event triggering sequence. The collaborative control diagram includes sub-task nodes, interface nodes, logic judgment nodes, and exception handling nodes. Among them, interface nodes explicitly define data transformation rules and event synchronization protocols, and logic judgment nodes adapt to conditional dependency scenarios, ensuring that the diagram structure completely covers the entire task execution process and provides a standardized blueprint for subsequent compilation and operation.
[0051] In some embodiments, controlling the operation of an agent based on a cooperative control graph includes: Compile the collaborative control chart into an executable program corresponding to each subtask process; Execute the communication configuration files between each executable program and each subtask process to control the operation of the agent.
[0052] Among them, the executable program is a code module generated according to the subtask process that can be run directly to complete a specific function, and the communication configuration file is a file that describes the format, protocol and interface address of data transmission between subtask processes.
[0053] In this embodiment, the cooperative control graph is first compiled to generate executable programs corresponding to each sub-task process, and a communication configuration file for coordinating the interaction between these programs is automatically generated. Then, by executing these executable programs in parallel or sequentially, and establishing real-time data exchange and event synchronization based on the communication configuration file, the various cross-domain modules of the agent are driven to complete the overall task according to the logic defined by the cooperative control graph.
[0054] In some embodiments, after generating the cooperative control graph of the agents, the method further includes: The collaborative control diagram is visualized, and the operation of the control agent is simulated based on the visualization results.
[0055] In this embodiment, after generating the cooperative control graph, the cooperative control graph is visualized, and the operation of the intelligent agent is simulated and controlled based on the visualization results for testing and optimization.
[0056] In some embodiments, at least two sub-task processes are divided into a visual domain task process and a motion control domain task process.
[0057] In this embodiment, at least two cross-domain subtask processes may include visual domain task processes, such as object detection and pose estimation, and motion control domain task processes, such as path planning and robotic arm trajectory execution, which are responsible for environmental perception and physical action execution, respectively.
[0058] In some embodiments, cross-domain collaboration rules are also used to define exception handling methods between various sub-task processes and control the operation of the intelligent agent, including: During the operation of the control agent, if any subtask process encounters an anomaly, the control agent will stop the execution of other related subtask processes and trigger an alarm mechanism.
[0059] In this embodiment, the cross-domain collaboration rules also define how other related sub-task processes should respond when an exception occurs in a certain sub-task process, such as stopping execution, rolling back or retrying, and whether to trigger alarms and other collaborative fault tolerance strategies.
[0060] If any subtask process encounters an anomaly during the operation of the intelligent agent, the system will automatically stop other related subtask processes and trigger an alarm mechanism to prevent error propagation, ensure system security, and improve the timeliness and controllability of fault response.
[0061] The control method for an intelligent agent provided in this application can be executed by a control device for the intelligent agent. This application uses an example of a control device for the intelligent agent executing the control method to illustrate the control device for the intelligent agent provided in this application.
[0062] This application also provides a control device for an intelligent agent.
[0063] like Figure 2 As shown, the control device for this intelligent agent includes: The first processing module 210 is used to receive and parse the control information input by the user to the intelligent agent, and obtain at least two cross-domain sub-task processes of the intelligent agent; The second processing module 220 is used to generate data conversion interfaces and event synchronization interfaces between various sub-task processes based on cross-domain collaboration rules. The cross-domain collaboration rules are used to define the way in which various sub-task processes work together in task decomposition, data conversion and event synchronization. The third processing module 230 is used to control the operation of the intelligent agent based on each subtask process and the data conversion interface and event synchronization interface between each subtask process.
[0064] According to the control device for the intelligent agent provided in the embodiments of this application, by introducing cross-domain collaboration rules, the control information input by the user to the intelligent agent is automatically parsed into multiple cross-domain sub-task processes, and corresponding data conversion interfaces and event synchronization interfaces are generated. This realizes semantic alignment and execution collaboration between cross-domain processes such as vision and motion, thereby reducing development collaboration and debugging costs, and improving the task adaptability, overall flexibility and reusability of the intelligent agent in complex and ever-changing scenarios.
[0065] In some embodiments, the third processing module 230 is used to generate a collaborative control diagram of the agent based on each subtask process and the data conversion interface and event synchronization interface between the subtask processes. The collaborative control diagram is used to indicate the way in which the subtask processes work together in task decomposition, data conversion and event synchronization. Control the operation of intelligent agents based on collaborative control graphs.
[0066] In some embodiments, the third processing module 230 is used to compile the collaborative control graph into an executable program corresponding to each subtask process; Execute the communication configuration files between each executable program and each subtask process to control the operation of the agent.
[0067] In some embodiments, the third processing module 230 is further configured to visualize the collaborative control graph and simulate the operation of the control agent based on the visualization results.
[0068] In some embodiments, at least two sub-task processes are divided into a visual domain task process and a motion control domain task process.
[0069] In some embodiments, the cross-domain collaboration rules are also used to define the exception handling methods between various sub-task processes. The third processing module 230 is used to control other related sub-task processes to stop execution and trigger an alarm mechanism when any sub-task process encounters an exception during the operation of the control agent.
[0070] The control device for the intelligent agent in this application embodiment can be an electronic device or a component within an electronic device, such as an integrated circuit or a chip. The electronic device can be a terminal or other devices besides a terminal. For example, the electronic device can be a mobile phone, tablet computer, laptop computer, PDA, in-vehicle electronic device, mobile internet device (MID), augmented reality (AR) / virtual reality (VR) device, robot, wearable device, ultra-mobile personal computer (UMPC), netbook, or personal digital assistant (PDA), etc. It can also be a server, network attached storage (NAS), personal computer (PC), television (TV), ATM, or self-service machine, etc. This application embodiment does not specifically limit the scope of the device.
[0071] The control device for the intelligent agent in this application embodiment can be a device with an operating system. This operating system can be a Microsoft (Windows) operating system, an Android operating system, an iOS operating system, or other possible operating systems; this application embodiment does not specifically limit the specific operating system.
[0072] The control device for the intelligent agent provided in this application embodiment can achieve... Figure 1 The various processes implemented in the method implementation examples will not be described again here to avoid repetition.
[0073] This application also provides a control system for an intelligent agent.
[0074] The control system of the intelligent agent is used to execute the control method of the intelligent agent described above.
[0075] The control system for the intelligent agent provided in the embodiments of this application introduces cross-domain collaboration rules, which automatically parses the control information input by the user to the intelligent agent into multiple cross-domain sub-task processes, and generates corresponding data conversion interfaces and event synchronization interfaces. This realizes semantic alignment and execution collaboration between cross-domain processes such as vision and motion, thereby reducing development collaboration and debugging costs, and improving the task adaptability, overall flexibility and reusability of the intelligent agent in complex and ever-changing scenarios.
[0076] In some embodiments, the control system of the intelligent agent includes: The distributed compilation module is used to compile the collaborative control graph into executable programs corresponding to each subtask process.
[0077] Among them, the distributed compilation module is a component that can automatically convert the collaborative control graph and generate executable programs corresponding to each subtask process. Its compilation process can be distributed and deployed on multiple computing nodes to support efficient code generation and deployment in cross-domain and heterogeneous environments.
[0078] The following describes a specific embodiment of the control method of an intelligent agent executed by a control system of an intelligent agent.
[0079] Cross-domain intelligent orchestration is achieved by introducing a collaborative bridging core layer.
[0080] Step 1: Construct a unified collaborative knowledge base. On top of the two independent knowledge bases for vision and operation control, build a collaborative bridging rule base, which stores cross-domain collaborative rules and defines cross-domain task mapping, cross-domain data transformation rules, and cross-domain event synchronization rules.
[0081] Cross-domain task mapping refers to the mapping relationship between high-level tasks (i.e., the overall task flow, such as "grabbing") and visual sub-task flows ("localization") and motion control sub-task flows ("movement-grabbing").
[0082] Cross-domain data transformation rules define how to automatically convert visual outputs, such as pixel coordinates (px, py) and angles theta, into motion control inputs, such as millimeter coordinates (Xmm, Ymm) and rotation angles Rz in machine coordinates, using calibration parameters. The rules can be simple linear transformations or calls to a calibration transformation function.
[0083] Cross-domain event synchronization rules define how visual events (such as "detection complete", "OK / NG") trigger or change the state of the operation and control process (such as "start movement" or "enter branch").
[0084] Step 2: The integrated task of receiving and parsing involves receiving the user's natural language description of the complete work unit, i.e., control information (e.g., "Use the camera to locate the workpiece on the conveyor belt, guide the robot to grab it and place it in the material box; if the workpiece type is incorrect, place it in the scrap area"). The large language model parses this description, identifying the composite needs involving both "seeing" and "moving".
[0085] Step 3: Cross-domain task decomposition and collaborative planning. Based on the collaborative knowledge base, the large language model performs the following key steps: Task decomposition breaks down the integrated task process into visual sub-task processes and motion control sub-task processes.
[0086] The interfaces are automatically generated based on data conversion rules, i.e., cross-domain collaboration rules. This automatically generates a data bridging interface (data conversion interface) connecting two sub-task processes, and an event synchronization interface (event synchronization interface). For example, a data conversion node is automatically created, whose internal logic calls a hand-eye calibration conversion function, taking the visual output coordinates as input and the robot's target coordinates as output.
[0087] Global anomaly flow planning defines cross-domain anomaly handling strategies. If vision fails to recognize three times consecutively, the operation and control process is notified to alarm and the system is shut down.
[0088] Step 4: Generate a structured collaborative workflow blueprint, i.e., a collaborative control diagram. Output a three-layer structured blueprint, including a visual workflow segment describing the image processing chain; a motion control workflow segment describing the action control chain; and a collaborative bridging segment that explicitly defines all interfaces between the above two workflow segments. This includes: a data bridging list (source, target, transformation function), an event synchronization list (source event, target action), and global variable and state definitions.
[0089] Step 5: Integrated Visualization and Collaborative Simulation. The system renders the three-layer blueprint in a unified manner, displaying the visual flow, operational control flow, and the connecting lines between them in a single view, representing data / event flow. Collaborative simulation can be performed to simulate how visual results affect control actions in real time.
[0090] Step Six: Distributed Compilation and Deployment. Equipped with a distributed compilation module, the unified collaborative blueprint is compiled in parallel into two independent deployable components: the vision platform project file and the motion controller program file. It also automatically generates and configures the communication configuration files between the two, such as the IP address, port, and data packet format of the Socket server / client.
[0091] The intelligent agent control method provided in this application provides a unified description, cross-domain decomposition, automatic interface generation, and integrated process orchestration for complex tasks that include vision and motion control.
[0092] It also constructs and uses a collaborative bridging rule library for cross-domain task mapping, data transformation rules, and event synchronization rules.
[0093] The three-tiered collaborative blueprint structure includes a structured blueprint data model comprising a vision segment, a control segment, and a collaborative bridging segment that specifically defines all interactions between the two.
[0094] Based on task requirements and rule base, specific technical solutions are automatically generated for data conversion and event synchronization interfaces between vision and operation control processes.
[0095] A distributed compilation system is a compiler system that can compile a single collaborative blueprint into multiple executable programs that belong to different platforms and can work together.
[0096] In related technologies, vision-guided control (such as visual positioning and grasping) typically employs a phased development and manual integration approach. This involves independent development, communication interfacing, and separate development processes and programs by the vision team and motion control team using their respective tools. Upon completion, both parties agree on communication protocols (such as TCP / IP, Modbus), data formats (such as byte order), and signal interaction points (such as image capture completion signals), manually bridging these interfaces by writing additional communication code. Integrated development environments (IDEs) are also used: some high-end robot manufacturers provide programming environments with built-in vision functions, but their vision processing capabilities are relatively fixed, their scalability is poor, and they are usually locked to specific hardware.
[0097] In related technologies, development is fragmented, resulting in extremely high collaboration costs. The vision and operation control teams need to communicate and debug repeatedly, and interface definitions are prone to errors, posing a major risk factor for project delays. Debugging is complex, and problem localization is difficult: during system operation, problems may originate from any link in the vision, communication, or control chain. The lack of cross-system debugging tools makes troubleshooting akin to guessing a black box. There is a lack of top-level collaboration logic: existing technologies only solve communication problems, not collaboration logic. For example, how the control flow should respond and retry after visual recognition failure still requires manual programming. The system is rigid and difficult to reuse; the communication and collaboration logic, customized for specific scenarios, is tightly coupled, making it difficult to quickly port to other similar but not identical application scenarios.
[0098] The control method for intelligent agents provided in this application can solve the problems of difficulty in collaboration, complexity in integration and debugging, lack of unified intelligent planning, and inability to perform integrated automatic orchestration from a top-level task perspective in application development.
[0099] It can eliminate collaboration gaps, achieving end-to-end automatic generation from top-level task description to final executable code, eliminating the expensive and error-prone manual integration process. It improves system reliability and robustness by using explicitly defined collaborative bridging segments and global exception policies, making cross-system collaboration logic clear, predictable, and maintainable. It achieves truly flexible collaborative units, packaging vision and motion control combinations into a rapidly configurable intelligent collaborative unit, greatly simplifying deployment and changeover at production line workstations.
[0100] In some embodiments, such as Figure 3 As shown, this application embodiment also provides an electronic device 300, including a processor 301, a memory 302, and a computer program stored in the memory 302 and executable on the processor 301. When the program is executed by the processor 301, it implements the various processes of the above-described intelligent agent control method embodiment and can achieve the same technical effect. To avoid repetition, it will not be described again here.
[0101] It should be noted that the electronic devices in the embodiments of this application include the mobile electronic devices and non-mobile electronic devices described above.
[0102] This application also provides a non-transitory computer-readable storage medium storing a computer program. When the computer program is executed by a processor, it implements the various processes of the above-described intelligent agent control method embodiments and achieves the same technical effect. To avoid repetition, it will not be described again here.
[0103] The processor is the processor in the electronic device described in the above embodiments. The readable storage medium includes computer-readable storage media, such as computer read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk.
[0104] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the above-described control method for an intelligent agent.
[0105] The processor is the processor in the electronic device described in the above embodiments. The readable storage medium includes computer-readable storage media, such as computer read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk.
[0106] This application embodiment also provides a chip, which includes a processor and a communication interface. The communication interface is coupled to the processor. The processor is used to run programs or instructions to implement the various processes of the above-described intelligent agent control method embodiment and can achieve the same technical effect. To avoid repetition, it will not be described again here.
[0107] It should be understood that the chip mentioned in the embodiments of this application may also be referred to as a system-on-a-chip, system chip, chip system, or system-on-a-chip, etc.
[0108] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element. Furthermore, it should be noted that the scope of the methods and apparatuses in the embodiments of this application is not limited to performing functions in the order shown or discussed, but may also include performing functions substantially simultaneously or in the reverse order, depending on the functions involved. For example, the described methods may be performed in a different order than described, and various steps may be added, omitted, or combined. Additionally, features described with reference to certain examples may be combined in other examples.
[0109] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the related technology, can be embodied in the form of a computer software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of this application.
[0110] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of this application without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of this application.
[0111] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "illustrative embodiment," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.
[0112] Although embodiments of this application have been shown and described, those skilled in the art will understand that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of this application, the scope of which is defined by the claims and their equivalents.
Claims
1. A control method for an intelligent agent, characterized in that, include: Receive and parse the control information input by the user to the intelligent agent to obtain at least two cross-domain sub-task processes of the intelligent agent; Based on cross-domain collaboration rules, data conversion interfaces and event synchronization interfaces are generated between the various sub-task processes. The cross-domain collaboration rules are used to define the collaborative working methods between the various sub-task processes in task decomposition, data conversion and event synchronization. The operation of the intelligent agent is controlled based on each of the sub-task processes and the data conversion interface and event synchronization interface between each of the sub-task processes.
2. The control method for an intelligent agent according to claim 1, characterized in that, The control of the agent's operation based on the data conversion interface and event synchronization interface between each subtask process and the subtask processes includes: Based on each of the subtask processes and the data conversion interface and event synchronization interface between the subtask processes, a collaborative control diagram of the agent is generated. The collaborative control diagram is used to indicate the way in which the subtask processes work together in task decomposition, data conversion and event synchronization. The operation of the intelligent agent is controlled based on the aforementioned collaborative control diagram.
3. The control method for an intelligent agent according to claim 2, characterized in that, The control of the agent's operation based on the cooperative control diagram includes: The collaborative control graph is compiled into an executable program corresponding to each of the sub-task processes; The communication configuration files between each of the executable programs and each of the subtask processes are executed to control the operation of the agent.
4. The control method for an intelligent agent according to claim 2, characterized in that, After generating the cooperative control graph of the agent, the method further includes: The collaborative control graph is visualized, and the operation of the intelligent agent is simulated based on the visualization results.
5. The control method for an intelligent agent according to any one of claims 1-4, characterized in that, At least two of the aforementioned sub-task processes are divided into a visual domain task process and a motion control domain task process.
6. The control method for an intelligent agent according to any one of claims 1-4, characterized in that, The cross-domain collaboration rules are also used to define the exception handling methods between each of the sub-task processes, and the control of the intelligent agent's operation includes: During the operation of the intelligent agent, if any of the sub-task processes encounters an anomaly, the other related sub-task processes will be stopped and an alarm mechanism will be triggered.
7. A control device for an intelligent agent, characterized in that, include: The first processing module is used to receive and parse the control information input by the user to the intelligent agent, and obtain at least two cross-domain sub-task processes of the intelligent agent; The second processing module is used to generate data conversion interfaces and event synchronization interfaces between the various sub-task processes based on cross-domain collaboration rules. The cross-domain collaboration rules are used to define the collaborative working methods between the various sub-task processes in task decomposition, data conversion and event synchronization. The third processing module is used to control the operation of the intelligent agent based on each of the sub-task processes and the data conversion interface and event synchronization interface between the sub-task processes.
8. A control system for an intelligent agent, characterized in that, A control method for performing the intelligent agent as described in any one of claims 1-6.
9. The control system for an intelligent agent according to claim 8, characterized in that, include: A distributed compilation module is used to compile the collaborative control graph into an executable program corresponding to each of the subtask processes.
10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the control method for the intelligent agent as described in any one of claims 1-6.