Task planning and execution method and system of heterogeneous robots, storage medium and computer device

Abstract

The application provides a heterogeneous robot task planning and execution method, system, storage medium and computer device. The method is applied to a general control system. The general control system is connected with different types of robots through an adapter and can realize unified management of different types of robots. Specifically, by introducing a meta-action / meta-skill library decoupled from the robot hardware and standardized task instructions, the application realizes the abstraction of the task planning and execution logic to the robot hardware type, avoids the repeated work of developing control codes for different robots, and significantly reduces the development cost and maintenance difficulty of the heterogeneous robot system. Meanwhile, the target planning scheme of the application is verified by artificial verification to ensure the rationality of the task logic. The 'hardware-instruction' conversion mechanism of the adapter can adapt different types of target robots to the same set of standardized instructions, greatly improving the compatibility of the system to heterogeneous hardware and the flexibility of task execution.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a method, system, storage medium and computer equipment for task planning and execution of heterogeneous robots. Background Technology

[0002] In real-world scenarios such as industrial production, logistics warehousing, and emergency rescue, heterogeneous robot systems (composed of robots with different structures, functions, and motion methods) are increasingly widely used. However, existing heterogeneous robot task planning and execution solutions have significant limitations: On the one hand, most systems rely on control logic customized for specific robot hardware. The control command formats and action definitions of different types of robots (such as wheeled mobile robots, robotic arms, and drones) vary greatly, making it difficult for general control systems to be directly compatible with multiple types of robots. This requires writing separate adaptation code for each type of robot, resulting in high development costs and poor scalability. On the other hand, task planning schemes are often strongly tied to the hardware characteristics of specific robots. When the robot hardware configuration changes, the planning scheme needs to be redesigned and verified, reducing the flexibility and efficiency of task execution. Summary of the Invention

[0003] The purpose of this application is to at least address one of the aforementioned technical deficiencies, particularly the high development costs, poor scalability, and low flexibility and efficiency of task execution in existing heterogeneous robot task planning and execution schemes.

[0004] This application provides a task planning and execution method for heterogeneous robots. The method is applied to a general control system, which communicates with different types of robots via an adapter. The method includes:

[0005] In response to a task planning request, a target planning scheme is obtained, which is obtained after manual verification;

[0006] Based on the target planning scheme and a predefined meta-action / meta-skill library that is decoupled from robot hardware, motion choreography is performed to form standardized task instructions that are independent of the robot.

[0007] Based on the type of the target robot to be controlled, the standardized task instructions are converted into the underlying control instructions of the target robot through the corresponding adapter;

[0008] The target robot is controlled to perform tasks according to the underlying control instructions.

[0009] Optionally, obtaining the target planning scheme includes:

[0010] The task library is used to search for a task planning scheme that matches the task planning request. The task library stores task planning schemes that have been planned for different scenarios and verified by humans.

[0011] If it exists, then the task planning scheme shall be used as the target planning scheme;

[0012] If it does not exist, an initial planning scheme is generated based on the task planning request, and the initial planning scheme is manually verified and then used as the target planning scheme.

[0013] Optionally, generating an initial planning scheme based on the task planning request includes:

[0014] Based on the task planning request, determine the scenario characteristics, task description, and task execution time of the task to be executed;

[0015] The pre-configured task planning model is invoked, and an initial planning scheme corresponding to the task to be executed is generated based on the scenario characteristics, the task description, and the task execution time.

[0016] Optionally, the step of manually verifying the initial planning scheme and then using it as the target planning scheme includes:

[0017] The initial planning scheme is visualized so that the operator can modify the initial planning scheme in the display interface according to the preset correction method;

[0018] The revised initial planning scheme will be used as the target planning scheme.

[0019] Optionally, the step of choreographing actions based on the target planning scheme and a predefined meta-action / meta-skill library decoupled from robot hardware to form standardized task instructions independent of the robot includes:

[0020] Obtain the subtask descriptions corresponding to each subtask included in the target planning scheme;

[0021] For each subtask, determine whether there is a composite action sequence in the composite action library that matches the subtask description of that subtask. The composite action library stores composite actions that have been manually verified and arranged for different subtask descriptions and a predefined meta-action / meta-skill library that is decoupled from robot hardware.

[0022] If present, then a standardized task instruction independent of the robot is formed based on the composite action sequence;

[0023] If it does not exist, then motion orchestration is performed based on the subtask description of the subtask and a predefined meta-motion / meta-skill library that is decoupled from the robot hardware, forming standardized task instructions that are independent of the robot.

[0024] Optionally, the motion orchestration based on the subtask description and a predefined, decoupled meta-action / meta-skill library from the robot hardware forms standardized task instructions independent of the robot, including:

[0025] Based on the subtask description of the subtask and the basic descriptions of each meta-action / meta-skill in the predefined meta-action / meta-skill library that is decoupled from the robot hardware, the list of meta-actions / meta-skills corresponding to the subtask description and the order of meta-actions / meta-skills are selected from the meta-action / meta-skill library.

[0026] Determine the security check action corresponding to each meta-action / meta-skill in the meta-action / meta-skill list;

[0027] Based on the safety check actions corresponding to each meta-action / meta-skill, and the sequence of each meta-action / meta-skill, composite actions are choreographed to form standardized task instructions independent of the robot.

[0028] Optionally, the step of choreographing composite actions based on the safety check actions corresponding to each meta-action / meta-skill and the sequence of each meta-action / meta-skill to form standardized task instructions independent of the robot includes:

[0029] Obtain real-time environmental information of the target robot to be controlled;

[0030] Based on the real-time environmental information, the safety check actions corresponding to each meta-action / meta-skill, and the sequence of each meta-action / meta-skill, composite actions are arranged to form standardized task instructions that are independent of the robot.

[0031] The standardized task instructions include sequential execution instructions and cyclic execution instructions, and the condition triggering mechanism of the standardized task instructions includes cyclic exit conditions and action triggering conditions.

[0032] Optionally, the method further includes:

[0033] During the execution of the target robot's task, the execution status of the target robot is monitored, and when an abnormality is detected in the execution status, the execution status of the target robot is intervened according to a preset abnormality handling strategy.

[0034] Optionally, the intervention in the execution state of the target robot according to a preset anomaly handling strategy includes:

[0035] Determine the policy type of the preset exception handling strategy, wherein the policy type includes local policy and global policy;

[0036] If the strategy type is a local strategy, then the execution state of the target robot is intervened according to the local strategy;

[0037] If the strategy type is a global strategy, then the execution state of the target robot is intervened according to the global strategy.

[0038] Optionally, intervening in the execution state of the target robot according to the local strategy includes:

[0039] Determine the target sub-task to be executed when the execution state of the target robot is detected to be abnormal, and the target composite action corresponding to the target sub-task;

[0040] According to the local strategy, the target robot can be controlled to re-execute the target composite action, skip the target composite action and execute the next composite action, or stop executing the target composite action and return the execution failure result of the target subtask.

[0041] Optionally, intervening in the execution state of the target robot according to the global strategy includes:

[0042] Control the target robot to re-execute the task, re-execute the currently abnormal sub-task, or skip the currently abnormal sub-task and continue to execute the next sub-task, and monitor the execution status of the target robot;

[0043] If the execution state of the target robot continues to be abnormal, the target robot is controlled to capture an image of the current scene, and an abnormality handling action sequence is generated based on the current scene image, the set of safety meta-actions corresponding to the current scene, and the complete information of the current task. After the target robot executes the abnormality handling action sequence, it continues to execute subsequent sub-tasks.

[0044] If an anomaly is detected in the target robot while it is executing the anomaly handling action sequence, manual intervention is requested.

[0045] This application also provides a general control system that communicates with different types of robots via an adapter, the system comprising:

[0046] The task planning module is used to obtain a target planning scheme in response to a task planning request. The target planning scheme is obtained after manual verification.

[0047] The motion orchestration module is used to orchestrate motions based on the target planning scheme and a predefined meta-motion / meta-skill library that is decoupled from the robot hardware, forming standardized task instructions that are independent of the robot.

[0048] The instruction generation module is used to convert the standardized task instructions into the underlying control instructions of the target robot according to the type of the target robot to be controlled, through the corresponding adapter;

[0049] The task execution module is used to control the target robot to perform tasks according to the underlying control instructions.

[0050] This application also provides a computer-readable storage medium storing computer-readable instructions, which, when executed by one or more processors, cause the one or more processors to perform the steps of the task planning and execution method for heterogeneous robots as described in any of the above embodiments.

[0051] This application also provides a computer device, including: one or more processors, and memory;

[0052] The memory stores computer-readable instructions, which, when executed by the one or more processors, perform the steps of the task planning and execution method for heterogeneous robots as described in any of the above embodiments.

[0053] As can be seen from the above technical solutions, the embodiments of this application have the following advantages:

[0054] This application provides a task planning and execution method, system, storage medium, and computer equipment for heterogeneous robots. The method is applied to a general control system that communicates with different types of robots via an adapter, enabling unified management of various robot types. Specifically, this application introduces a meta-action / meta-skill library and standardized task instructions decoupled from the robot hardware, achieving "abstraction" of task planning and execution logic for robot hardware types. This avoids the repetitive work of developing separate control code for different robots, significantly reducing the development cost and maintenance difficulty of heterogeneous robot systems. Furthermore, the target planning scheme of this application has been manually verified to ensure the rationality of the task logic. The adapter's "hardware-instruction" conversion mechanism allows the same set of standardized instructions to be adapted to different types of target robots, such as wheeled mobile robots, robotic arms, and drones, greatly improving the system's compatibility with heterogeneous hardware and the flexibility of task execution. Attached Figure Description

[0055] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0056] Figure 1 A flowchart illustrating a task planning and execution method for a heterogeneous robot provided in an embodiment of this application;

[0057] Figure 2 A schematic diagram illustrating the process of determining standardized task instructions provided in an embodiment of this application;

[0058] Figure 3 A schematic diagram illustrating the process of forming standardized task instructions independent of robots, provided for embodiments of this application;

[0059] Figure 4 A schematic diagram of the structure of a general control system provided in an embodiment of this application;

[0060] Figure 5 This is a schematic diagram of the internal structure of a computer device provided in an embodiment of this application. Detailed Implementation

[0061] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0062] In one embodiment, as shown in the figure Figure 1 This application provides a flowchart illustrating a task planning and execution method for a heterogeneous robot, as shown in the embodiments of this application. The method is applied to a general-purpose control system, which communicates with different types of robots via an adapter. The method may include:

[0063] S110: In response to a task planning request, obtain the target planning scheme.

[0064] In this step, the task planning request can be manually entered by the user through the interactive interface of the general control system, or it can be automatically triggered by the associated business system (such as the industrial production scheduling system or the logistics and warehousing management system). After receiving the request, the system can first parse the request content and extract the core elements of the task to be executed, such as "in the warehousing scenario, move 3 boxes of parts from the shelf in area A to the assembly station in area B", thus clarifying the task's objectives, scenario, and key constraints.

[0065] The process then proceeds to obtain the target planning scheme: If a manually verified similar task planning scheme (such as a standardized process including "shelf positioning → part picking → path navigation → assembly table placement") is already stored in the task library, that scheme is directly retrieved as the target planning scheme; if no matching scheme is found in the task library, the system will call a pre-trained task planning model, combining parameters such as the spatial layout of the warehouse scene, the weight and volume of the parts, and the robot's load capacity, to generate an initial planning scheme (such as planning the shortest obstacle avoidance path from area A to area B, and setting the force threshold for the robotic arm to pick up parts), and display it to the operator through a visual interface. The operator can modify the initial scheme in the interface, such as adjusting the stopping points in the path and modifying the execution order of the picking actions. After the modified scheme is verified by the system, it is officially stored in the task library as the target planning scheme and marked as "verified" so that it can be directly reused in subsequent similar tasks.

[0066] S120: Based on the target planning scheme and a predefined meta-action / meta-skill library that is decoupled from the robot hardware, motion programming is performed to form standardized task instructions that are independent of the robot.

[0067] In this step, after obtaining the target planning scheme through S110, this application can perform motion choreography based on the target planning scheme and a predefined meta-motion / meta-skill library that is decoupled from the robot hardware, forming standardized task instructions that are independent of the robot.

[0068] Specifically, when choreographing actions, this application first breaks down the target planning scheme into a structured form, extracting each sub-task and its corresponding action requirements. For example, the target planning scheme for "warehouse parts handling" can be broken down into four sub-tasks: "shelf positioning," "parts gripping," "path navigation," and "assembly table placement." Each sub-task corresponds to specific action requirements. For instance, "shelf positioning" requires the identification and precise docking of the target shelf's coordinates, while "parts gripping" requires the stable clamping and attitude adjustment of the parts.

[0069] Next, the system can call a predefined library of meta-actions / meta-skills. These meta-actions / meta-skills are decoupled from the robot hardware, such as "coordinate positioning," "robotic arm joint drive," "mobile chassis path planning," and "end-effector gripping control." Each meta-action / meta-skill includes standardized parameter definitions (e.g., positioning accuracy ±5mm, gripping force range 10-50N) and execution logic descriptions. For each decomposed subtask, the system can first search the composite action library to determine if a verified composite action sequence matching the subtask's action requirements exists. Taking the "part gripping" subtask as an example, if the compound action library already stores a standardized compound action sequence of "robotic arm extension → end effector opening → contact with part → pressure sensor detection → end effector closing → robotic arm lifting", then the sequence is directly called. If there is no matching sequence in the compound action library, the system can filter the corresponding meta-actions / meta-skills from the meta-action / meta-skill library based on the action requirements of the subtask. For example, for the requirement of "part gripping", three meta-actions are filtered out: "robotic arm joint angle adjustment", "end effector opening and closing control" and "pressure sensor data acquisition". Then, the order of the meta-actions is determined according to the logical dependency of the actions (such as needing to adjust the robotic arm angle to above the part first, and then control the end effector to open).

[0070] After determining the meta-action sequence, the system can also dynamically configure the parameters of the meta-actions by combining real-time environmental information of the current task scenario (such as the distribution of obstacles in the storage space and the real-time positional deviation of the parts). For example, if a deviation of 10mm is detected between the actual position of a part and the planned position, the system will automatically adjust the parameters of the "robotic arm joint angle adjustment" to correct the target angle to the value after the deviation. At the same time, the system can add conditional triggering mechanisms to the choreographed action sequence, including loop exit conditions (such as in the "path navigation" sub-task, when the robot reaches the assembly table coordinates within ±10mm, it exits the navigation loop) and action triggering conditions (such as in the "part gripping" sub-task, when the pressure value detected by the pressure sensor reaches 20N, it triggers the subsequent action of closing the end effector). The final standardized task instructions include sequential execution instructions (such as executing them in the order of "shelf positioning → part gripping → path navigation → assembly table placement") and cyclic execution instructions (such as cyclically detecting obstacles during the "path navigation" process until the target position is reached). The format of the instructions is independent of the robot hardware type and only contains general information such as action type, parameter range, and triggering conditions.

[0071] It is understood that the meta-action / meta-skill library in this application contains all the basic actions a robot can perform, such as walking, raising its hand, and closing its end effector for grasping. These actions together constitute the robot's most basic action unit. When a robot leaves the factory, it usually comes pre-installed with a series of basic actions. This application refers to these basic actions as meta-actions, which constitute the robot's most fundamental capabilities. Based on the robot's fundamental capabilities, algorithmic models trained for specific targets, or capabilities formed by combining algorithmic models with meta-actions, such as obstacle avoidance grasping by a robotic arm, vision-based following, and VLA-based unordered grasping, constitute the basic capabilities for completing specific targets. This application refers to these as meta-skills. Meta-actions and meta-skills constitute the basic modules of a robot's capabilities. Based on the combination and arrangement of these modules, a sequence of actions can be formed for the robot to complete a specific task.

[0072] Furthermore, each meta-action / meta-skill in this application contains five attributes: code represents the encoding of the meta-action / meta-skill, name represents the name of the meta-action / meta-skill, type represents the type of the meta-action / meta-skill, and in this application, a total of 5 types of actions are defined: ExecutionAction: representing the robot's basic action, which can be executed directly, VIPerceptor: representing visual perception skills, VoicePerceptor: representing voice perception skills, PathPlan: representing path planning and decision-making skills, ArmMovePlan: representing arm planning and decision-making skills; category represents the category of the meta-action / meta-skill, which includes two categories: BuildingAction: representing meta-action / meta-skill, ComposeAction: representing compound actions; description is a basic description of the meta-action / meta-skill capability.

[0073] The composite action in this application is a hierarchical encapsulation based on the meta-action / meta-skill unit. For example, the three meta-action / meta-skill units of "rotating the robotic arm joint to the target angle," "opening the end effector gripper," and "visually locating the object to be grasped" are arranged in the logical sequence of "first visual positioning → then joint rotation → finally gripper opening" and encapsulated into the composite action unit "robotic arm prepares to grasp." This composite action unit retains the execution logic of the underlying meta-action / meta-skill but hides the specific hardware control details from the outside. When using it, users only need to focus on the target effect of the composite action without having to disassemble its internal structure.

[0074] Meanwhile, the composite actions in this application support binding specific parameters to upper-layer applications. For example, for the composite action of "robotic arm preparing to grasp", users can bind parameters such as "target object recognition type (e.g., 'plastic bottle')" and "robotic arm joint target angle (e.g., 'shoulder joint 90°, elbow joint 45°')" to it during the task orchestration stage. These parameters can be passed to the underlying meta-action / meta-skill unit during task execution to achieve personalized adaptation of composite actions.

[0075] S130: Based on the type of the target robot to be controlled, the standardized task instructions are converted into the target robot's low-level control instructions through the corresponding adapter.

[0076] In this step, after forming standardized task instructions independent of the robot through S120, this application can convert the standardized task instructions into the underlying control instructions of the target robot according to the type of the target robot to be controlled through the corresponding adapter.

[0077] It is understandable that different types of robots differ in hardware architecture, communication protocols, and control interfaces. For example, wheeled mobile robots use the ROS (Robot Operating System) topic communication protocol, robotic arms use the Modbus TCP protocol, and drones support the MQTT protocol. Furthermore, the underlying control command formats (such as joint angle commands, speed commands, and position commands) also vary among robots. Therefore, this application uses adapters to achieve the adaptation and conversion between standardized task commands and the underlying control commands of the target robot. Each adapter is designed for a specific type of robot and incorporates core functional modules such as protocol parsing, command mapping, and parameter conversion.

[0078] Specifically, the system first identifies the type of the target robot to be controlled (e.g., identifying it as an industrial robotic arm based on the device identifier "Robot-Type: IndustrialArm-001" reported when the robot connects), and then calls the corresponding adapter. After receiving the standardized task instruction, the adapter parses the instruction, extracting core information such as the action type, parameter range, and trigger conditions. Taking the standardized instruction "robotic arm prepares to grasp" as an example, the core information obtained after parsing includes: action type "compound action - prepare to grasp," parameters "target object recognition type: plastic bottle, robotic arm joint target angle: shoulder joint 90°, elbow joint 45°," and trigger condition "execute after successful visual positioning."

[0079] Next, the adapter can convert the parsed information into low-level control commands for the target robot based on a predefined "standardized command-low-level command" mapping table. For the industrial robotic arm adapter, the low-level command sequence corresponding to "compound action-ready gripping" in the mapping table is as follows: First, send the vision sensor control command "SET_VISION_MODE: OBJECT_RECOGNITION, TARGET: PLASTIC_BOTTLE" (set the vision mode to object recognition, the target to a plastic bottle). After the vision sensor returns "RECOGNITION_SUCCESS" (recognition successful) feedback, send the joint angle control command "SET_JOINT_ANGLE: SHOULDER=90, ELBOW=45" (set the shoulder joint angle to 90° and the elbow joint angle to 45°). Finally, send the end effector control command "SET_GRIPPER_STATE: OPEN, SPEED=50%" (set the gripper to open at 50% speed). At the same time, the adapter will adapt the format of the converted underlying instructions. For example, the underlying instructions of industrial robotic arms need to adopt the format of "instruction code + parameter + check bit". The adapter will automatically add the instruction code (such as vision control instruction code is 0x01, joint control instruction code is 0x02) and CRC check bit to ensure that the instructions comply with the robot's communication protocol requirements.

[0080] Furthermore, the adapter in this application also features real-time parameter calibration. During the conversion process, if a difference is detected between the target robot's hardware parameters and those in the standardized instructions (e.g., the standardized instructions require a joint angle accuracy of ±5mm, while the target robot's actual accuracy is ±8mm), the adapter will call the parameter calibration model and adjust the parameters based on the robot's hardware performance curve (e.g., correcting the joint target angle from 90° to 90.5° to compensate for the accuracy deviation), and embed the calibrated parameters into the underlying control instructions. After adjustment, the adapter will send the underlying control instructions to the target robot via a communication interface (e.g., Ethernet port, serial port) and monitor the robot's instruction feedback information in real time (e.g., "Instruction received successfully" "Joint angle has reached the target value"). If no feedback is received within a preset time (e.g., exceeding 500ms), the adapter will automatically resend the instruction. If there is still no feedback after more than 3 resends, the system will trigger an alarm and upload the abnormal information to the task monitoring module so that the operator can promptly troubleshoot the communication failure.

[0081] S140: Controls the target robot to perform tasks according to the underlying control instructions.

[0082] In this step, after the standardized task instructions are converted into the target robot's underlying control instructions through S130, this application can control the target robot to perform tasks based on the underlying control instructions.

[0083] Specifically, after receiving the underlying control commands, the target robot can execute actions sequentially according to the order and logic of the commands. Taking a parts handling task in a warehouse scenario as an example, after receiving the underlying instructions for "shelf positioning → parts gripping → path navigation → assembly table placement", the industrial robotic arm first activates the vision perception module to execute the shelf positioning instruction. It collects image information of the shelves in area A through a camera, matches it with the pre-stored shelf feature library, determines the three-dimensional coordinates of the shelves, and feeds it back to the control system. Then, it executes the parts gripping instruction. The robotic arm adjusts the joint angles according to the position of the parts located by vision, and the end gripper closes to grip the parts according to the set force threshold. The pressure sensor monitors the gripping pressure in real time to ensure that the parts are stably gripped without damage. Next, the wheeled mobile robot receives the path navigation instruction. Combining the map data of the warehouse space and real-time obstacle detection information (such as avoiding temporarily stacked goods by scanning with LiDAR), it plans the optimal movement path and starts the chassis drive system to move towards the assembly table in area B at a preset speed. When it reaches the assembly table coordinates within ±10mm, the system triggers the path navigation loop exit condition, the mobile robot stops moving, the robotic arm executes the assembly table placement instruction, slowly adjusts the posture of the end gripper, and places the parts smoothly in the designated area of ​​the assembly table. After completion, the gripper opens and resets, and sends task completion feedback to the control system.

[0084] During task execution, the system can monitor the robot's motion status in real time. It collects motion execution data through the robot's built-in sensors (such as joint angle sensors, speed sensors, and force sensors), including joint angle deviations of the robotic arm, the deviation between the actual and planned paths, and changes in pressure during part grasping. This data is then compared with preset thresholds. If an anomaly is detected, such as a joint angle deviation exceeding ±2°, a path blockage caused by an unidentified static obstacle, or a part grasping pressure below 10N (potentially causing part detachment), the system immediately triggers an anomaly handling mechanism: first, it suspends the current action and sends an alarm message to the operator containing the anomaly type, location, and suggested solution; if the anomaly is automatically recoverable (e.g., in the case of path blockage, the system re-invokes the path planning decision-making skill to generate alternative paths), the remaining task continues after anomaly handling; if the anomaly cannot be automatically recovered (e.g., a robotic arm sensor malfunction), the task execution is terminated, and anomaly details are recorded in the task log for subsequent analysis and maintenance.

[0085] Furthermore, upon completion of the task, the system can generate a task execution report, which includes key indicators such as the start and end times of the task, the execution time of each subtask, the completion rate of the robot's actions, the number of anomalies and their handling results, and the final positional accuracy of the parts being transported. This report serves two purposes: firstly, it evaluates the efficiency and quality of task execution; secondly, it supplements the task library with experiential data. If similar tasks are encountered subsequently, the system can refer to the parameter configurations in this report (such as the optimal force threshold for part grasping and obstacle avoidance strategies for path navigation) to optimize the planning scheme, thereby achieving continuous iterative upgrades to task planning and execution capabilities.

[0086] In the above embodiments, the general control system communicates with different types of robots through adapters, enabling unified management of different types of robots. Specifically, this application introduces a meta-action / meta-skill library and standardized task instructions decoupled from the robot hardware, achieving "abstraction" of task planning and execution logic for robot hardware types. This avoids the repetitive work of developing separate control code for different robots, significantly reducing the development cost and maintenance difficulty of heterogeneous robot systems. Simultaneously, the target planning scheme of this application has been manually verified to ensure the rationality of the task logic, and the adapter's "hardware-instruction" conversion mechanism allows the same set of standardized instructions to be adapted to different types of target robots such as wheeled mobile robots, robotic arms, and drones, greatly improving the system's compatibility with heterogeneous hardware and the flexibility of task execution.

[0087] In one embodiment, obtaining the target planning scheme in S110 may include:

[0088] S111: Search the task library for a task planning scheme that matches the task planning request. The task library stores task planning schemes that are planned for different scenarios and have been manually verified.

[0089] S112: If it exists, then the task planning scheme shall be used as the target planning scheme.

[0090] S113: If it does not exist, an initial planning scheme is generated based on the task planning request, and the initial planning scheme is manually verified and used as the target planning scheme.

[0091] In this embodiment, the construction of the task library follows the principle of "scenario-based classification + dynamic update". For example, the storage directory is divided according to typical scenarios such as warehousing and handling, assembly and production, and inspection and security. The task planning scheme under each scenario is marked with the applicable robot type combination (such as "industrial robotic arm + wheeled mobile robot" and "drone + ground inspection robot"), task complexity level (such as simple task / medium task / complex task) and key parameter configuration template.

[0092] When the system receives a task planning request for "parts cross-area handling" from the warehouse management system, it can first extract the core features of the request: the task scenario is "warehouse handling", the target robot combination is "industrial robotic arm (model IRB6700) + wheeled mobile robot (model AGV-R2)", and the task objective is "to move the M20 bolts on the 3rd layer of the shelf in area A to the assembly table in area B". Subsequently, the system performs a fuzzy search in the task library based on these characteristics. If it finds a "Warehouse Parts Handling Task Solution V2.1" that was verified by the operator two months ago—whose applicable scenario and robot combination perfectly match the current request, and whose part type (bolt) and shelf level (3rd level) in the task target differ from the current request within a preset threshold (e.g., part size deviation ≤10%)—then it directly uses this solution as the target planning solution and automatically calls the parameter adaptation module to update "part model M18" to "M20" in the solution, and fine-tune "shelf coordinates X=15.2m, Y=8.7m" to the actual coordinates X=15.3m, Y=8.6m of the current shelf in area A, ensuring that the solution is fully adapted to the current task environment.

[0093] If no matching solution is found in the task library, the system will initiate the initial planning solution generation process. This process is goal-oriented and first calls the meta-action / meta-skill library to filter out the basic capability modules required to complete the task. For example, "warehouse parts handling" requires calling the meta-skills of "visual positioning (shelves / parts)," "joint motion planning," and "end-effector grasping" of industrial robotic arms, as well as the meta-actions of "path navigation (warehouse environment)" and "obstacle avoidance" of wheeled mobile robots. Next, the system can use the task logic reasoning engine to arrange these modules into an initial action sequence based on preset "action dependency rules" (such as "part gripping must be performed after visual positioning is completed" and "mobile robot path navigation must be started after the robotic arm grips the part") in chronological order: "Industrial robotic arm visually positions shelf A → visually positions M20 bolt on the 3rd layer of shelf → adjusts joint angle to gripping position → end gripper closes and grips → wheeled mobile robot plans path from area A to area B → mobile robot reaches robotic arm coordinates → robotic arm places part on mobile robot carrier → mobile robot navigates to assembly table in area B → robotic arm grips part from carrier → places part in designated position on assembly table → gripper resets."

[0094] After the initial solution is generated, the system can automatically generate a verification report, including but not limited to the logical flowchart of the action sequence, parameter configuration suggestions for each meta-action / meta-skill (such as "part gripping pressure threshold recommended 15N-20N", "mobile robot navigation speed recommended 0.8m / s"), and potential risk point prompts (such as "insufficient shelf lighting may affect visual positioning accuracy"). The report is then pushed to the task verification terminal. The operator simulates the task execution process through the terminal. If a logical flaw is found where "the height of the carrier's bearing surface is not considered when the robotic arm places parts onto the carrier," the operator can directly drag and adjust the action sequence on the verification interface, adding the meta-action "mobile robot adjusts chassis height to be flush with the robotic arm's end effector," and modifying the "robotic arm placement height parameter" to "carrier bearing surface height + 5mm." After successful verification, the operator clicks the "Confirm Effective" button. The system automatically marks the initial solution as "verified" and stores it in the "warehouse handling" scenario directory of the task library, while also recording the verification personnel, verification time, and modification records for subsequent traceability and version iteration.

[0095] This "retrieval priority + manual verification and completion" mechanism not only fully utilizes existing mature solutions and shortens the response time for task planning, but also ensures the logical correctness and practical feasibility of the initial solution through manual verification, avoiding task execution failures due to the limitations of the system's automatic reasoning. Simultaneously, the dynamic update mechanism of the task library allows the system's task planning capabilities to continuously accumulate as real-world application scenarios expand. For example, when a new scenario like "cold chain warehouse parts handling" is added, operator-verified combinations of special meta-skills in low-temperature environments, such as "visual anti-fog positioning" and "clamp low-temperature insulation control," will be automatically included in the task library, providing more accurate references for subsequent similar tasks.

[0096] In one embodiment, generating an initial planning scheme based on the task planning request in step S113 may include:

[0097] S1131: Determine the scenario characteristics, task description, and task execution time of the task to be executed based on the task planning request.

[0098] S1132: Invoke the pre-configured task planning model and generate an initial planning scheme corresponding to the task to be executed based on the scenario characteristics, the task description and the task execution time.

[0099] In this embodiment, when generating an initial planning scheme based on a task planning request, this application can call a pre-configured task planning model to perform task planning.

[0100] The task planning model in this application can adopt a four-level architecture of "scene awareness - task decomposition - skill matching - sequence arrangement" to ensure that the generated initial solution is both in line with the needs of the scene and feasible. First, the scene feature analysis module can deeply extract scene information from the task planning request. For example, in the "cold chain warehouse parts handling" scenario, in addition to basic information such as "warehouse environment" and "the object being handled is frozen parts", it can also identify special constraints such as "ambient temperature -18℃", "humidity 85%" and "shelf surface is prone to frost". The task description analysis module focuses on the core objectives and constraints of the task, such as "handling the frozen vaccine boxes (size 20cm×15cm×10cm, weight 1.2kg) on ​​the second shelf of the frozen shelf in area C to the vaccination station in area D, requiring that the temperature fluctuation of the vaccine boxes during the handling process does not exceed ±1℃". The task execution time analysis module will combine the time requirements in the request (such as "must be completed within 30 minutes, and the current time is 9:15 am, and the vaccination station in area D will be used from 10:00 am") to calculate the time thresholds for each sub-task, such as "the movement time from area C to area D must be ≤10 minutes" and "the total time for parts picking and placing must be ≤5 minutes".

[0101] Next, based on the above analysis results, the task decomposition module breaks down the complex task into multiple sub-tasks with uniform granularity. Taking cold chain vaccine handling as an example, this task can be broken down into 10 sub-tasks: "frozen shelf positioning", "frozen vaccine box identification and positioning", "robotic arm low-temperature adaptability adjustment", "end-effector anti-frost pretreatment", "vaccine box gripping", "mobile robot low-temperature path planning", "vaccine box carrier insulation", "D-zone vaccination table positioning", "vaccine box placement", and "equipment reset". Each sub-task is marked with a priority (e.g., "low-temperature adaptability adjustment" has a higher priority than "vaccine box gripping") and a time window (e.g., "frozen shelf positioning must be completed within 9:15-9:20").

[0102] Subsequently, the skill matching module can traverse the meta-action / meta-skill library to match suitable basic skills for each sub-task. For example, for the sub-task of "frozen vaccine box recognition and positioning," since the recognition accuracy of traditional vision sensors decreases in low-temperature frosty environments, the system will automatically match the "low-temperature anti-fog visual recognition skill" (which integrates heating defrosting module control commands and low-temperature image enhancement algorithms); for the sub-task of "robotic arm low-temperature adaptability adjustment," it matches the "joint low-temperature lubrication compensation skill" (which can adjust the heating power of joint lubricating oil according to the ambient temperature to ensure joint movement accuracy); for the sub-task of "vaccine box carrier insulation," it matches the "carrier constant temperature control skill" (which maintains the temperature at 2-8℃ through the carrier's built-in heating element). At the same time, the skill matching module will verify the compatibility between the skill and the target robot hardware, such as confirming whether the wheeled mobile robot has a hardware interface for carrier constant temperature control. If not, it will automatically replace it with the "mobile carrier with insulated box skill" and simultaneously adjust the placement height parameters of the robotic arm.

[0103] Finally, the sequence orchestration module can orchestrate the matched skills into an initial action sequence based on the "action dependency rule base" and the priority and time window of the subtasks, according to their logical and temporal order. For example, "robotic arm low-temperature adaptability adjustment" needs to be executed before "frozen vaccine box identification and positioning," "end-effector anti-frost pretreatment" needs to be performed synchronously with "frozen vaccine box identification and positioning" (to save time), and "mobile robot low-temperature path planning" needs to be started 3 minutes before "vaccine box grasping" is completed (planning the path in advance to avoid waiting). After orchestration, the system can add parameter suggestions for each skill, such as "robotic arm joint lubricant heating power set to 150W," "end-effector anti-frost pretreatment time set to 2 minutes," and "carrier constant temperature control temperature set to 5℃," and calculate the total execution time of the initial plan (e.g., 28 minutes), verify whether it meets the task execution time requirements, and if it exceeds them, automatically optimize the time window of the subtasks (e.g., compress the duration of "equipment reset" to 1 minute).

[0104] Furthermore, the task planning model incorporates a scenario-adaptive optimization mechanism. For example, in a high-temperature workshop parts handling scenario, if the ambient temperature exceeds 40°C, a pre-emptive action of "starting the high-temperature cooling fan" will be automatically added to the robotic arm joint control skill; in a noisy construction site inspection scenario, a parameter configuration of "anti-noise image filtering" will be added to the drone's visual recognition skill. This mechanism allows the initial planning scheme to flexibly cope with the special constraints of different scenarios without the need for manual adjustment of basic skill parameters.

[0105] In one embodiment, the manual verification of the initial planning scheme in step S113, which serves as the target planning scheme, may include:

[0106] S1133: Visualize the initial planning scheme so that the operator can modify the initial planning scheme in the display interface according to the preset correction method.

[0107] S1134: Use the revised initial planning scheme as the target planning scheme.

[0108] In this embodiment, the visualization of the initial planning scheme can be achieved by combining "3D scene simulation + action sequence flow diagram". The operator can intuitively view the scheme details through the interactive interface of the task verification terminal. The 3D scene simulation module can construct a 1:1 virtual scene based on the environmental parameters in the task request (such as the layout of the low-temperature shelves in the cold chain warehouse, the physical dimensions of the mobile robot, and the joint structure of the robotic arm) to simulate the entire process of task execution: for example, the motion trajectory of the robotic arm adjusting joint lubrication compensation in a -18℃ environment, the dynamic effect of the end effector defrosting pretreatment, and the temperature change curve of the constant temperature chamber of the mobile robot carrier. At the same time, different colors can be used to mark the priority of sub-tasks in the scene (red represents high priority, yellow represents medium priority, and green represents low priority), and the execution windows of each sub-task are displayed synchronously on the timeline. If there is a time conflict in the scheduling of a sub-task (such as the time windows of "frozen vaccine box recognition" and "robotic arm low-temperature adjustment" overlapping), the system will automatically pop up a red warning box in the scene to indicate the type of conflict and the task links that may be affected.

[0109] The action sequence diagram displays the logical dependencies between subtasks and meta-skills in a node-connection format. Each node is labeled with the subtask name, the matching skill type, the estimated execution time, and hardware call information (e.g., the "vehicle temperature control skill" calls the vehicle heating element interface of the wheeled mobile robot). Operators can adjust the execution order of subtasks by dragging nodes or click on nodes to expand the skill parameter configuration panel and directly modify parameter thresholds—for example, if the "end-point gripper defrosting pretreatment time of 2 minutes" is insufficient to handle thick frost on the shelf surface, the time can be adjusted to 3 minutes in the panel, and the simulation results of the surface state after gripper defrosting in the 3D scene can be viewed in real time to verify the effect of the modification. If the operator needs to add a subtask (e.g., to add the "battery low temperature detection before mobile robot departure" step), they can right-click to add a node in the blank area of ​​the sequence diagram. The system will automatically match the corresponding "battery low temperature detection skill" and recalculate the overall task execution time to ensure that it does not exceed the preset time window.

[0110] After the operator completes all corrections, clicking the "Verify Simulation" button in the lower right corner of the interface allows the system to perform a full-process virtual execution based on the corrected scheme, recording the execution data for each step (such as the actual motion accuracy of the robotic arm joints and the temperature fluctuation value of the vaccine box). If no abnormalities occur during the simulation (such as temperature fluctuation ≤ ±1℃, and all sub-tasks being completed within the time window), the system will display a "Verification Passed" message, mark the corrected scheme as "Verified," and store it in the task library. If an abnormality occurs during the simulation, such as "temperature exceeds the threshold when placing the vaccine box," the system will locate the corresponding sub-task node (such as "vehicle temperature control") and highlight the temperature change curve of the vehicle in the 3D scene to assist the operator in further adjusting the insulation parameters (such as lowering the vehicle temperature from 5℃ to 4℃) until the simulation verification is successful.

[0111] This visual correction mechanism not only lowers the technical threshold for manual verification, but also avoids potential risks in actual implementation through virtual simulation, ensuring the reliability and accuracy of the target planning scheme.

[0112] In one embodiment, such as Figure 2 As shown, Figure 2 This is a schematic diagram illustrating the process of determining standardized task instructions provided in an embodiment of this application; S120 involves choreographing actions based on the target planning scheme and a predefined meta-action / meta-skill library decoupled from robot hardware to form standardized task instructions independent of the robot, which may include:

[0113] S121: Obtain the descriptions of the subtasks corresponding to each subtask included in the target planning scheme.

[0114] S122: For each subtask, determine whether there is a composite action sequence in the composite action library that matches the subtask description of that subtask. If it exists, execute S123; otherwise, execute S124.

[0115] S123: Generate standardized task instructions that are independent of the robot based on the compound action sequence.

[0116] S124: Based on the subtask description of this subtask and a predefined meta-action / meta-skill library that is decoupled from the robot hardware, perform motion orchestration to form standardized task instructions that are independent of the robot.

[0117] In this embodiment, the construction of the composite action library is based on the core principle of "scenario-based reuse." All stored composite action sequences have been verified in actual scenarios and are decoupled from the robot hardware model. For example, the "component action of grasping parts in a low-temperature environment" for the "cold chain warehouse parts handling" scenario encompasses a fixed arrangement logic for four meta-actions: "low-temperature lubrication compensation of robotic arm joints → defrosting preprocessing of end-effectors → visual anti-fog positioning → adaptive pressure grasping." The parameter thresholds for each meta-action are set to a range adaptable to different hardware (e.g., joint lubrication compensation power is adjustable from 50W to 200W). When the system receives the sub-task description of "frozen vaccine box grasping" in the target planning scheme, it can first extract the core features of the sub-task ("low-temperature environment," "vaccine box grasping," "anti-frost requirement") using a natural language matching algorithm, and then traverse the tag index of the composite action library (e.g., "low-temperature grasping," "fragile item grasping," "anti-frost") to quickly locate the matching "component action of grasping parts in a low-temperature environment." At this point, the system does not need to re-decompose the sub-tasks and matching meta-actions. It can directly call the composite action sequence and automatically adjust the threshold of the meta-actions based on the specific parameters in the target planning scheme (such as the weight of the vaccine box being 1.2kg). For example, the grasping pressure threshold can be adjusted from the default 10N-15N to 12N-18N to generate standardized task instructions.

[0118] If a subtask in the target planning scheme is a special requirement under a new scenario (such as "grabbing a chip with anti-static packaging in a -25℃ cryogenic storage"), and there is no matching composite action sequence in the composite action library, the system will initiate the "dynamic skill orchestration" process: First, extract key constraints based on the subtask description ("cryo-25℃", "anti-static packaging", "chip grabbing"); second, select suitable basic meta-actions from the meta-action / meta-skill library, such as "robotic arm cryogenic lubrication compensation skill" (supports -30℃ environment), "end-effector anti-static coating activation skill", "visual recognition anti-static". The system employs several techniques: "Packaging Chip Skills" (integrating anti-static packaging feature recognition algorithms) and "Micro-force Grip Control Skills" (pressure threshold 5N-10N). Next, based on preset "Action Dependency Rules" (such as "Anti-static coating activation must be completed before gripper closure" and "Cryogenic lubrication compensation must be performed before robotic arm movement"), these meta-actions are logically arranged into a temporary action sequence. Finally, the system automatically adds hardware-independent parameter interfaces to each meta-action (such as "Lubrication compensation power: {{Cryogenic compensation power}}" and "Grip pressure: {{Chip weight matching pressure}}"), forming standardized task instructions. These instructions do not contain any specific robot hardware address or model information; they only interact with the subsequent hardware driver layer through a general parameter interface, ensuring that the instructions are adaptable to heterogeneous robots of different brands and models.

[0119] Through a dual-path mechanism of "compound action reuse + dynamic skill choreography", the system not only ensures the efficiency of action choreography, but also achieves complete decoupling between standardized task instructions and robot hardware, providing a unified instruction interaction basis for the collaborative execution of heterogeneous robots.

[0120] In one embodiment, such as Figure 3 As shown, Figure 3 This is a schematic diagram illustrating the process of forming standardized task instructions independent of the robot, as provided in an embodiment of this application. In step S124, based on the subtask description and a predefined meta-action / meta-skill library decoupled from the robot hardware, motion orchestration is performed to form standardized task instructions independent of the robot. This may include:

[0121] S1241: Based on the subtask description of the subtask and the basic descriptions of each meta-action / meta-skill in the predefined meta-action / meta-skill library that is decoupled from the robot hardware, select the list of meta-actions / meta-skills corresponding to the subtask description and the order of meta-actions / meta-skills from the meta-action / meta-skill library.

[0122] S1242: Determine the security check action corresponding to each meta-action / meta-skill in the meta-action / meta-skill list.

[0123] S1243: Based on the safety check actions corresponding to each meta-action / meta-skill, and the sequence of each meta-action / meta-skill, composite action choreography is performed to form standardized task instructions that are independent of the robot.

[0124] In this embodiment, each basic skill in the meta-action / meta-skill library is associated with a corresponding safety check action template. For example, the safety check actions associated with the "robotic arm cryogenic lubrication compensation skill" include "joint lubricating oil temperature sensor calibration," "lubrication system pressure detection," and "heating module insulation test." The safety check actions associated with the "end-effector antistatic coating activation skill" include "antistatic coating resistance value detection" and "fistor grounding status verification." After the system filters out the list of meta-actions corresponding to the sub-task, it automatically calls the safety check action matching rules to bind a pre-operational safety check step to each meta-action. For example, for the meta-action "robotic arm cryogenic lubrication compensation," "joint lubricating oil temperature sensor calibration" can be set as a mandatory safety check action before its execution to ensure the accuracy of sensor data and avoid insufficient lubrication compensation due to temperature detection deviation.

[0125] Taking the subtask of "grabbing chips with antistatic packaging in a -25℃ ultra-low temperature warehouse" as an example, the selected list of meta-actions is "robotic arm ultra-low temperature lubrication compensation → end-effector antistatic coating activation → visual recognition of antistatic packaged chips → micro-force gripping control". The system will match safety check actions for each meta-action: "joint lubricating oil temperature sensor calibration" and "lubrication system pressure detection" will be added before "robotic arm ultra-low temperature lubrication compensation"; "antistatic coating resistance value detection" and "gripper grounding status verification" will be added before "end-effector antistatic coating activation"; "low temperature camera lens defrosting status check" will be added before "visual recognition of antistatic packaged chips"; and "end-effector force sensor zero-point calibration" will be added before "micro-force gripping control". Subsequently, based on the rule that "safety checks must be completed before the corresponding meta-actions are executed," the system inserts the safety checks into the corresponding positions in the meta-action sequence, forming a composite action chain that includes safety steps: "Joint lubricant temperature sensor calibration → Lubrication system pressure detection → Robotic arm ultra-low temperature lubrication compensation → Antistatic coating resistance value detection → Fixture grounding status verification → End-effector antistatic coating activation → Low temperature camera lens defrosting status check → Visual recognition antistatic packaging chip → End-effector force sensor zero-point calibration → Micro-force gripping control."

[0126] Simultaneously, the system can add judgment logic and anomaly handling branches to each safety inspection action. For example, in the "joint lubricant temperature sensor calibration" step, if the detected sensor error exceeds ±0.5℃, the "automatic sensor calibration program" is triggered; if the error still exceeds the standard after calibration, an "abnormal prompt of sensor failure" is generated, and the action programming is paused, awaiting manual intervention. These safety inspection actions and anomaly handling logic are encapsulated into standardized instruction modules, which, together with the meta-action instructions, constitute the final standardized task instructions, ensuring the safety and stability of the instructions during execution.

[0127] In one embodiment, S1243 involves arranging composite actions based on the safety check actions corresponding to each meta-action / meta-skill and the sequence of these actions / meta-skills to form standardized task instructions independent of the robot. This may include:

[0128] S12431: Obtain real-time environmental information of the target robot to be controlled.

[0129] S12432: Based on the real-time environmental information, the safety check actions corresponding to each meta-action / meta-skill, and the sequence of each meta-action / meta-skill, composite action choreography is performed to form standardized task instructions independent of the robot.

[0130] The standardized task instructions include sequential execution instructions and cyclic execution instructions, and the condition triggering mechanism of the standardized task instructions includes cyclic exit conditions and action triggering conditions.

[0131] In this embodiment, the acquisition of real-time environmental information can be achieved through the multimodal perception module in the heterogeneous robot system. This module integrates devices such as temperature sensors, humidity sensors, lidar, and vision cameras, and can collect environmental parameters in the task scenario in real time (such as the current warehouse temperature, the relative distance between the shelf and the robot, and the flatness of the ground).

[0132] Taking the subtask of "grabbing a chip with anti-static packaging in a -25℃ ultra-low temperature warehouse" as an example, the environmental information fed back by the multimodal perception module in real time during the action orchestration stage includes, but is not limited to: the real-time warehouse temperature is -26℃ (lower than the preset -25℃ threshold), there is a 3cm ice bump on the edge of the shelf, and there is a 0.5cm frost layer on the ground. Based on this real-time environmental information, the system can dynamically adjust the parameters of safety inspection actions and meta-actions: for the safety inspection action of "joint lubricating oil temperature sensor calibration", because the real-time temperature is even lower, the system automatically tightens the allowable error range of calibration from ±0.5℃ to ±0.3℃ to ensure the accuracy of lubrication compensation; for the meta-action of "robotic arm ultra-low temperature lubrication compensation", according to the difference between the real-time temperature and the preset value, the lubrication compensation power is increased from the default 150W to 180W to ensure the flexible movement of the joint at lower temperatures. Meanwhile, based on information about the frost layer on the ground, the system temporarily adds a safety check action of "mobile robot chassis anti-slip detection" to the subsequent stage of the "micro-force gripping control" meta-action to prevent the robot from slipping during movement.

[0133] Furthermore, the sequential execution instructions in the standardized task instructions of this application strictly follow the logical order of the compound action chain. For example, "joint lubricant temperature sensor calibration → lubrication system pressure detection → robotic arm ultra-low temperature lubrication compensation" must be executed sequentially. If the previous action is not completed or fails the safety verification, the next action cannot be started. The cyclic execution instructions are applicable to links that need to be executed repeatedly. For example, in the "visual recognition antistatic packaging chip" meta-action, if the first visual recognition fails (due to temporary lens frost), the system will trigger the cyclic execution instructions to repeatedly execute the combined action of "low temperature camera lens defrosting status check → visual recognition antistatic packaging chip". The maximum number of cycles is 3, and the cycle exit condition is "visual recognition successful" or "the number of cycles reaches the upper limit". The action triggering condition is associated with real-time environmental information. For example, when the result of "end-effector antistatic coating resistance value detection" shows that the resistance value is higher than 10^9Ω, the action instruction of "antistatic coating reactivation" is triggered to ensure that the antistatic effect of the coating meets the requirements.

[0134] By integrating real-time environmental information into the choreography process of composite actions, standardized task instructions can better adapt to dynamically changing scenarios, further improving the adaptability and safety of instructions, and providing reliable instruction support for heterogeneous robots to perform tasks in complex environments.

[0135] In one embodiment, the method may further include:

[0136] S150: During the process of the target robot performing the task, the execution status of the target robot is monitored, and when an abnormality is detected in the execution status, the execution status of the target robot is intervened according to a preset abnormality handling strategy.

[0137] In this embodiment, during the process of the target robot performing a task, the system can also monitor the execution status of the target robot. The monitoring of the execution status is achieved through the real-time data acquisition module of the heterogeneous robot system. This module can synchronously acquire the operating parameters of each hardware unit of the robot (such as the joint angle of the robotic arm, the pressure of the end gripper, the temperature of the carrier, the speed of the moving chassis, etc.) as well as the progress data of task execution (such as the number of sub-tasks completed, the remaining execution time, etc.).

[0138] Anomalies can be categorized into three main types: hardware failure, environmental interference, and task deviation. Hardware failures include joint jamming of the robotic arm and loss of sensor data. Environmental interference includes sudden temperature fluctuations and obstacle intrusion. Task deviations include grasping position deviation and action execution timeout. The preset anomaly handling strategy is divided into three response levels based on the anomaly severity: Level 1 response addresses minor anomalies (e.g., visual recognition success rate drops to 90% but the task can still be completed), where the system automatically adjusts parameters (e.g., increasing the feature matching threshold for visual recognition) and continues task execution. Level 2 response addresses moderate anomalies (e.g., insufficient lubrication of the robotic arm joints leading to reduced movement speed), where the system pauses the current subtask, initiates a local correction process (e.g., re-executes lubrication compensation actions), and resumes the task after correction. Level 3 response addresses severe anomalies (e.g., a malfunctioning end-effector force sensor prevents detection of grasping pressure), where the system immediately stops all actions, issues an audible and visual alarm, and pushes anomaly details (including the faulty hardware ID and real-time data curves) to the operator for manual investigation and repair.

[0139] Taking the "ultra-low temperature storage chip grasping" task as an example, if the system detects a sudden drop in end-effector pressure to 0N (a serious hardware failure) during execution, it first locates the faulty unit as the end-effector force sensor through the data acquisition module. Then, it triggers a three-level response mechanism: immediately stopping the robotic arm's grasping action, locking the mobile chassis, highlighting the force sensor's fault status on the monitoring interface, and generating an anomaly report containing "fault time, sensor model, and historical data trend" to be sent to the operator's terminal. Upon receiving the alarm, the operator can view the sensor's real-time voltage data using remote diagnostic tools, confirming the fault is caused by a broken sensor cable due to low temperature. The operator then arranges for maintenance personnel to replace the cable. After the fault is repaired, the system restarts the task from the "end-effector force sensor zero-point calibration" stage before the anomaly occurred using the "breakpoint resume execution" function, eliminating the need to execute the entire complex action chain from scratch, significantly improving the fault tolerance and efficiency of task execution.

[0140] In one embodiment, the intervention in the execution state of the target robot according to a preset exception handling strategy in S150 may include:

[0141] S151: Determine the strategy type of the preset exception handling strategy, wherein the strategy type includes local strategy and global strategy.

[0142] S152: If the strategy type is a local strategy, then the execution state of the target robot is intervened according to the local strategy.

[0143] S153: If the strategy type is a global strategy, then the execution state of the target robot is intervened according to the global strategy.

[0144] In this embodiment, the preset anomaly handling strategy can include local and global strategies. Local strategies focus on anomalies in specific sub-steps or single hardware units during task execution, with intervention limited to the action chain segments associated with the anomaly, avoiding excessive impact on the overall task flow. For example, during the "robotic arm cryogenic lubrication compensation" meta-action, if the lubrication system pressure is detected to be below a preset threshold (e.g., the normal pressure range is 0.8MPa-1.2MPa, and the current pressure is 0.6MPa), the system determines it as a local anomaly and triggers the local strategy: suspend the "robotic arm cryogenic lubrication compensation" action, start the "lubrication system pressure replenishment procedure" (injecting a measured amount of lubricating oil into the lubrication pipeline), and re-detect the pressure after replenishment. If the pressure returns to the normal range, the subsequent "antistatic coating activation" meta-action continues; if the pressure still does not meet the standard after replenishment, it is upgraded to a moderate anomaly, triggering a secondary response mechanism. The core of the local strategy is "minimizing intervention," ensuring the continuity of the overall task by specifically correcting the anomaly.

[0145] The global strategy applies to severe anomalies affecting the entire task flow, such as multi-hardware unit failures or breaches of core safety constraints, with intervention covering all ongoing actions. For example, in a cryogenic storage task, if real-time environmental monitoring detects a sudden rise in storage temperature to -10℃ (exceeding the task requirement of -25℃±2℃), and temperature sensors continue to report a warming trend, the system determines this as a global anomaly and triggers the global strategy: immediately halting all robot actions (including robotic arm grasping and chassis movement), initiating emergency insulation procedures (such as closing storage doors and activating the backup refrigeration system), and simultaneously sending an emergency alarm for "abnormal ambient temperature" to the operator. Once the ambient temperature returns to the permissible range, the system re-executes all safety checks (such as robotic arm lubrication compensation and antistatic coating detection), and resumes task execution only after confirming there are no anomalies. The core of the global strategy is "safety first," pausing the entire task to prevent safety incidents, and re-verifying the safety of all aspects after the anomaly is resolved to ensure the reliability of subsequent task execution.

[0146] Furthermore, the system can dynamically switch strategy types based on the real-time development of anomalies. For example, if the initial detection indicates that "the resistance value of the antistatic coating on the end clamp is slightly higher than 10^9 Ω" (a minor local anomaly), the system triggers a local strategy (reactivating the coating). If the resistance value continues to rise to 10^11 Ω after reactivation (exceeding the safety threshold), the anomaly escalates to a global safety risk, and the system automatically switches from a local strategy to a global strategy, stopping all actions and initiating in-depth troubleshooting (such as checking coating wear, power grounding status, etc.). This dynamic strategy switching mechanism ensures the accuracy of anomaly handling and improves the system's adaptability to complex anomaly scenarios.

[0147] In one embodiment, intervening in the execution state of the target robot according to the local strategy in S152 may include:

[0148] S1521: Determine the target sub-task executed when the execution state of the target robot is detected to be abnormal, and the target composite action corresponding to the target sub-task.

[0149] S1522: Control the target robot to re-execute the target composite action, skip the target composite action and execute the next composite action, or stop executing the target composite action according to the local strategy, and return the execution failure result of the target subtask.

[0150] In this embodiment, the local strategy provides four specific intervention methods for the target sub-task and its corresponding target composite action to adapt to different abnormal scenarios. The specific abnormal handling strategies include the following:

[0151] A. Stop execution and return the result indicating that the subtask failed;

[0152] B. Skip this action and continue to the next action;

[0153] C. Try re-executing the action until a certain number of times is reached. If it still fails, then execute strategy A.

[0154] D. Restore to the state before the compound action was executed, and re-execute the compound action.

[0155] Taking the "Visual Recognition of Antistatic Packaging Chips" target subtask within the "Ultra-Low Temperature Storage Chip Grabbing" task as an example, its corresponding target composite action is "Low Temperature Camera Lens Defrosting Status Check → Visual Recognition of Antistatic Packaging Chips". If the first visual recognition fails during the execution of this composite action (due to brief frost on the lens but not complete obstruction), the system determines it as a minor local anomaly and adopts the intervention method of "re-executing the target composite action": controlling the low temperature camera to restart the defrosting program (extending the defrosting time by 2 seconds), and then re-performing the visual recognition. If the second recognition is successful, the subsequent "Micro-Force Grabbing Control" meta-action continues to be executed; if recognition still fails after two consecutive re-executions, the system switches to "stop executing the target composite action and return the execution failure result", while recording the reason for the failure (such as "stubborn frost on the lens leads to a feature point matching rate of less than 60%)".

[0156] If an anomaly occurs during the execution of the target composite action, such as "the visual recognition result is stable but the chip position deviates from the preset coordinates by 0.2cm (within the safe allowable range)," the system determines it as a tolerable local deviation and adopts the intervention method of "skipping the target composite action and executing the next composite action": directly enter the "micro-force grasping control" meta-action, and automatically adjust the grasping coordinates to the actual coordinates of visual recognition, without having to re-execute the visual recognition step, thus saving task time.

[0157] For the target composite action of "activating the anti-static coating of the end clamp", if the coating activation voltage is detected to be slightly lower than the standard value but still meets the anti-static requirements (resistance value between 10^8Ω and 10^9Ω) during execution, the system also adopts the intervention method of "skipping the target composite action" and directly executes the subsequent "visual recognition" meta-action, while marking "coating activation voltage is slightly low and requires subsequent maintenance" in the task log; if the coating activation fails (resistance value is higher than 10^10Ω), the composite action is stopped and the execution failure result is returned, triggering the secondary response mechanism.

[0158] By clearly defining the correspondence between target sub-tasks and target composite actions, local strategies can accurately locate abnormal links, select appropriate intervention methods based on the severity and scope of the abnormality, and minimize interference with the overall task process while ensuring task safety, thereby improving the task execution efficiency and flexibility of heterogeneous robots in complex scenarios.

[0159] In one embodiment, intervening in the execution state of the target robot according to the global strategy in S153 may include:

[0160] S1531: Control the target robot to re-execute the task, re-execute the currently abnormal sub-task, or skip the currently abnormal sub-task and continue to execute the next sub-task, and monitor the execution status of the target robot.

[0161] S1532: If the execution state of the target robot continues to be abnormal, control the target robot to capture an image of the current scene, and generate an abnormality handling action sequence based on the current scene image, the set of safety meta-actions corresponding to the current scene, and the complete information of the current task. After the target robot executes the abnormality handling action sequence, it continues to execute subsequent sub-tasks.

[0162] S1533: If an anomaly is detected when the target robot is executing the anomaly handling action sequence, then request manual intervention.

[0163] In this embodiment, the global strategy intervention process revolves around the overall safety and continuity of task execution, and uses a layered processing mechanism to deal with global anomalies of different severity. Taking the "chip sorting and transfer" task in an ultra-low temperature storage environment as an example, if the system detects that the LiDAR of the mobile robot chassis suddenly loses more than 50% of the environmental feature points (which is a global perception anomaly), the intervention step S1531 is executed first: control all heterogeneous robots (including robotic arms and mobile chassis) to pause their current actions, and try to restart the LiDAR feature point matching algorithm. If the feature points recover to more than 80% after re-execution, the transfer task continues; if the feature points are still less than 50% after re-execution, the anomaly handling action sequence generation stage of S1532 is entered. At this point, the system calls the preset set of safety actions for the current scene (such as "emergency braking of the mobile chassis → self-diagnosis of LiDAR hardware → activation of backup vision sensor → reconstruction of environmental map"), and generates an action sequence by combining the complete information of the task (such as the current position of the chip and the coordinates of the target sorting table): First, the mobile chassis is controlled to maintain a braking state to avoid collisions due to loss of perception; then the hardware self-diagnosis program of the LiDAR is started to check for loose cables or module failures; if the self-diagnosis shows no hardware abnormalities, the backup depth camera is activated, and environmental features are completed using depth images and historical map data to reconstruct the local map; after the map reconstruction is completed and the feature point matching rate recovers to more than 70%, the subsequent sub-tasks of chip transfer are executed (such as the robotic arm placing the chip on the sorting table).

[0164] If a new anomaly occurs during the execution of the anomaly handling sequence, such as the backup depth camera failing to start due to low temperature (a hardware-level global fault), the system triggers a manual intervention request (S1533): immediately pushes a composite anomaly report containing "LDA feature point loss + backup camera failure" to the operator terminal, along with the robot's current position coordinates, ambient temperature curve, and hardware status log, and activates an audible and visual alarm. The operator can view the robot's real-time video feed through the remote control interface. After confirming the fault scenario, the operator arranges for technicians to bring the backup camera to the site for replacement. Once the hardware replacement is complete and passes the system self-check, the operator can remotely send a "resume task execution" command. The system then resumes the transfer task from the "chip grabbing completed" stage before the anomaly occurred, without needing to re-execute the completed sub-tasks.

[0165] Furthermore, the global strategy intervention also supports the dynamic adjustment of the anomaly handling action sequence. For example, in the "chip sorting and transfer" task, if the initially generated action sequence reaches the "environmental map reconstruction" stage and the warehouse temperature is detected to have further dropped to -30℃ (exceeding the lower limit of the robot's operating temperature), the system can automatically adjust the action sequence: temporarily add the "robot core component temperature protection" meta-action before "environmental map reconstruction," controlling the heating module to preheat the camera, radar, and other sensing units (raising the temperature to -25℃), and continue map reconstruction after the temperature reaches the standard, ensuring the safety and effectiveness of the anomaly handling process.

[0166] This dynamic adjustment mechanism further enhances the adaptability of the global strategy to extremely complex scenarios, providing more comprehensive safety guarantees for the task execution of heterogeneous robots.

[0167] The general control system provided in the embodiments of this application is described below. The general control system of the collection system described below and the task planning and execution method of the heterogeneous robot described above can be referred to in correspondence.

[0168] In one embodiment, such as Figure 4 As shown, Figure 4 This is a schematic diagram of a general control system provided in an embodiment of this application. This application also provides a general control system that communicates with different types of robots via an adapter. The system may include a task planning module 210, a motion orchestration module 220, an instruction generation module 230, and a task execution module 240, specifically including the following:

[0169] The task planning module 210 is used to obtain a target planning scheme in response to a task planning request. The target planning scheme is obtained after manual verification.

[0170] The motion choreography module 220 is used to choreograph motions based on the target planning scheme and a predefined meta-motion / meta-skill library that is decoupled from the robot hardware, forming standardized task instructions that are independent of the robot.

[0171] The instruction generation module 230 is used to convert the standardized task instructions into the underlying control instructions of the target robot according to the type of the target robot to be controlled, through the corresponding adapter.

[0172] The task execution module 240 is used to control the target robot to perform tasks according to the underlying control instructions.

[0173] In the above embodiments, the general control system communicates with different types of robots through adapters, enabling unified management of different types of robots. Specifically, this application introduces a meta-action / meta-skill library and standardized task instructions decoupled from the robot hardware, achieving "abstraction" of task planning and execution logic for robot hardware types. This avoids the repetitive work of developing separate control code for different robots, significantly reducing the development cost and maintenance difficulty of heterogeneous robot systems. Simultaneously, the target planning scheme of this application has been manually verified to ensure the rationality of the task logic, and the adapter's "hardware-instruction" conversion mechanism allows the same set of standardized instructions to be adapted to different types of target robots such as wheeled mobile robots, robotic arms, and drones, greatly improving the system's compatibility with heterogeneous hardware and the flexibility of task execution.

[0174] In one embodiment, this application also provides a computer-readable storage medium storing computer-readable instructions that, when executed by one or more processors, cause the one or more processors to perform the steps of the task planning and execution method for heterogeneous robots as described in any of the above embodiments.

[0175] In one embodiment, this application also provides a computer device, including: one or more processors, and memory.

[0176] The memory stores computer-readable instructions, which, when executed by the one or more processors, perform the steps of the task planning and execution method for heterogeneous robots as described in any of the above embodiments.

[0177] Indicatively, such as Figure 5 As shown, Figure 5 This is a schematic diagram of the internal structure of a computer device 300 provided in an embodiment of this application. The computer device 300 can be provided as a server. (Refer to...) Figure 5The computer device 300 includes a processing component 302, which further includes one or more processors, and memory resources represented by memory 301 for storing instructions, such as application programs, that can be executed by the processing component 302. The application programs stored in memory 301 may include one or more modules, each corresponding to a set of instructions. Furthermore, the processing component 302 is configured to execute instructions to perform the task planning and execution methods for heterogeneous robots described in any of the above embodiments.

[0178] The computer device 300 may also include a power supply component 303 configured to perform power management of the computer device 300, a wired or wireless network interface 304 configured to connect the computer device 300 to a network, and an input / output (I / O) interface 305. The computer device 300 may operate on an operating system stored in memory 301, such as Windows Server™, Mac OS X™, Unix™, Linux™, Free BSD™, or similar.

[0179] Those skilled in the art will understand that Figure 5 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0180] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0181] The various embodiments in this specification are described in a progressive manner. Each embodiment focuses on the differences from other embodiments. The various embodiments can be combined as needed, and the same or similar parts can be referred to each other.

[0182] The above description of the disclosed embodiments enables those skilled in the art to make or use this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for task planning and execution of heterogeneous robots, characterized in that, The method is applied to a general-purpose control system, which communicates with different types of robots via an adapter. The method includes: In response to a task planning request, a target planning scheme is obtained, which is obtained after manual verification; Based on the target planning scheme and a predefined meta-action / meta-skill library that is decoupled from robot hardware, motion choreography is performed to form standardized task instructions that are independent of the robot. Based on the type of the target robot to be controlled, the standardized task instructions are converted into the underlying control instructions of the target robot through the corresponding adapter; The target robot is controlled to perform tasks according to the underlying control instructions; During the execution of the target robot's task, the execution status of the target robot is monitored, and when an abnormality in the execution status is detected, the strategy type of the preset abnormality handling strategy is determined. The strategy type includes local strategy and global strategy. If the strategy type is a local strategy, then the execution state of the target robot is intervened according to the local strategy; If the strategy type is a global strategy, then the target robot is controlled to re-execute the task, re-execute the currently abnormal sub-task, or skip the currently abnormal sub-task and continue to execute the next sub-task, and the execution status of the target robot is monitored. If the execution state of the target robot continues to be abnormal, the target robot is controlled to capture an image of the current scene, and an abnormality handling action sequence is generated based on the current scene image, the set of safety meta-actions corresponding to the current scene, and the complete information of the current task. After the target robot executes the abnormality handling action sequence, it continues to execute subsequent sub-tasks. If an anomaly is detected in the target robot while it is executing the anomaly handling action sequence, manual intervention is requested.

2. The method according to claim 1, characterized in that, The process of obtaining the target planning scheme includes: The task library is used to search for a task planning scheme that matches the task planning request. The task library stores task planning schemes that have been planned for different scenarios and verified by humans. If it exists, then the task planning scheme shall be used as the target planning scheme; If it does not exist, an initial planning scheme is generated based on the task planning request, and the initial planning scheme is manually verified and then used as the target planning scheme.

3. The method according to claim 2, characterized in that, The process of generating an initial planning scheme based on the task planning request includes: Based on the task planning request, determine the scenario characteristics, task description, and task execution time of the task to be executed; The pre-configured task planning model is invoked, and an initial planning scheme corresponding to the task to be executed is generated based on the scenario characteristics, the task description, and the task execution time.

4. The method according to claim 2, characterized in that, The step of manually verifying the initial planning scheme and then using it as the target planning scheme includes: The initial planning scheme is visualized so that the operator can modify the initial planning scheme in the display interface according to the preset correction method; The revised initial planning scheme will be used as the target planning scheme.

5. The method according to claim 1, characterized in that, The process of choreographing actions based on the target planning scheme and a predefined meta-action / meta-skill library, decoupled from robot hardware, to form standardized task instructions independent of the robot includes: Obtain the subtask descriptions corresponding to each subtask included in the target planning scheme; For each subtask, determine whether there is a composite action sequence in the composite action library that matches the subtask description of that subtask. The composite action library stores composite actions that have been manually verified and arranged for different subtask descriptions and a predefined meta-action / meta-skill library that is decoupled from robot hardware. If present, then a standardized task instruction independent of the robot is formed based on the composite action sequence; If it does not exist, then motion orchestration is performed based on the subtask description of the subtask and a predefined meta-motion / meta-skill library that is decoupled from the robot hardware, forming standardized task instructions that are independent of the robot.

6. The method according to claim 5, characterized in that, The process of choreographing actions based on the subtask description and a predefined, decoupled meta-action / meta-skill library from the robot hardware forms standardized task instructions independent of the robot, including: Based on the subtask description of the subtask and the basic descriptions of each meta-action / meta-skill in the predefined meta-action / meta-skill library that is decoupled from the robot hardware, the list of meta-actions / meta-skills corresponding to the subtask description and the order of meta-actions / meta-skills are selected from the meta-action / meta-skill library. Determine the security check action corresponding to each meta-action / meta-skill in the meta-action / meta-skill list; Based on the safety check actions corresponding to each meta-action / meta-skill, and the sequence of each meta-action / meta-skill, composite actions are choreographed to form standardized task instructions independent of the robot.

7. The method according to claim 6, characterized in that, The process of choreographing composite actions based on the safety check actions corresponding to each meta-action / meta-skill, and the sequence of these actions / meta-skills, to form standardized task instructions independent of the robot, includes: Obtain real-time environmental information of the target robot to be controlled; Based on the real-time environmental information, the safety check actions corresponding to each meta-action / meta-skill, and the sequence of each meta-action / meta-skill, composite actions are arranged to form standardized task instructions that are independent of the robot. The standardized task instructions include sequential execution instructions and cyclic execution instructions, and the condition triggering mechanism of the standardized task instructions includes cyclic exit conditions and action triggering conditions.

8. The method according to claim 1, characterized in that, The intervention in the execution state of the target robot according to the local strategy includes: Determine the target sub-task to be executed when the execution state of the target robot is detected to be abnormal, and the target composite action corresponding to the target sub-task; According to the local strategy, the target robot can be controlled to re-execute the target composite action, skip the target composite action and execute the next composite action, or stop executing the target composite action and return the execution failure result of the target subtask.

9. A general-purpose control system, characterized in that, The general control system communicates with different types of robots via adapters, and the system includes: The task planning module is used to obtain a target planning scheme in response to a task planning request. The target planning scheme is obtained after manual verification. The motion choreography module is used to choreograph motions based on the target planning scheme and a predefined meta-motion / meta-skill library that is decoupled from the robot hardware, forming standardized task instructions that are independent of the robot. The instruction generation module is used to convert the standardized task instructions into the underlying control instructions of the target robot according to the type of the target robot to be controlled, through the corresponding adapter; The task execution module is used to control the target robot to perform tasks according to the underlying control instructions; During the execution of the target robot's task, the execution status of the target robot is monitored, and when an abnormality in the execution status is detected, the strategy type of the preset abnormality handling strategy is determined. The strategy type includes local strategy and global strategy. If the strategy type is a local strategy, then the execution state of the target robot is intervened according to the local strategy; If the strategy type is a global strategy, then the target robot is controlled to re-execute the task, re-execute the currently abnormal sub-task, or skip the currently abnormal sub-task and continue to execute the next sub-task, and the execution status of the target robot is monitored. If the execution state of the target robot continues to be abnormal, the target robot is controlled to capture an image of the current scene, and an abnormality handling action sequence is generated based on the current scene image, the set of safety meta-actions corresponding to the current scene, and the complete information of the current task. After the target robot executes the abnormality handling action sequence, it continues to execute subsequent sub-tasks. If an anomaly is detected in the target robot while it is executing the anomaly handling action sequence, manual intervention is requested.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-readable instructions that, when executed by one or more processors, cause the one or more processors to perform the steps of the task planning and execution method for heterogeneous robots as described in any one of claims 1 to 8.

11. A computer device, characterized in that, include: One or more processors, and memory; The memory stores computer-readable instructions, which, when executed by the one or more processors, perform the steps of the task planning and execution method for heterogeneous robots as described in any one of claims 1 to 8.

Images