Adaptive cooperative control method and system based on time domain decoupling and application thereof

Abstract

This invention proposes an adaptive cooperative control method based on time-domain decoupling and its application. Its core idea is to creatively transform a complex, strongly coupled physical problem in the spatial domain into a more manageable time-domain logical scheduling problem. This method uses an independent external sensing system to evaluate the stability of the mobile platform in real time and dynamically adjusts the time progress of tasks executed by the operating device based on this stability index: when the platform is stable, authorized tasks are executed normally; when the platform becomes unstable due to disturbances, tasks are actively paused to provide a pure stability recovery window for the platform, and tasks resume after stabilization. As a model-independent top-level controller, this invention can be compatible with any "black box" hardware combination by sending high-level abstract instructions, exhibiting excellent robustness, versatility, and engineering practicality, significantly reducing the development cost and technical threshold of mobile operating robots.

Description

Technical Field

[0001] This invention belongs to the field of robotics and automation control technology, specifically involving collaborative control technology for dynamic and unstable mobile platforms (such as drones, unmanned vehicles, etc.) and operating equipment (such as robotic arms, work tools, etc.). In particular, it addresses the top-level control method and system for achieving highly robust collaborative operation in scenarios where the underlying controller is a closed "black box" without dynamic model support. Background Technology

[0002] In the field of robotics and automation control, integrating robotic arms and other manipulators onto mobile platforms such as drones to perform complex tasks is a current research hotspot. However, achieving precise and stable coordination between mobile platforms and manipulators, especially in dynamically changing real-world environments, presents significant technical challenges.

[0003] Currently, the industry mainly employs the following technological approaches:

[0004] The first approach is a control method based on precise system dynamics models. This method attempts to establish a complete mathematical model encompassing the interactions of the mobile platform, robotic arm, and even the environment, and then designs feedforward compensation or complex nonlinear controllers based on this model. However, this approach has inherent drawbacks: firstly, the modeling process is extremely complex and costly, and it is difficult to cover all dynamic effects, such as outdoor wind disturbances and load changes; secondly, any mismatch between the model and reality can lead to a sharp decline in control performance or even failure, making it robust and difficult to deploy on a large scale commercially.

[0005] The second method is the static step-by-step operation mode. To avoid the dynamic coupling problem between platform movement and robotic arm operation, this method adopts a rigid process of "moving into position and hovering completely before performing the operation." However, in real-world environments with continuous disturbances, "perfect absolute stillness" is almost impossible to achieve. Even slight drifts and vibrations of the platform can be transmitted to the robotic arm's end effector, leading to operation failure. At the same time, this mode also greatly limits work efficiency.

[0006] More importantly, the aforementioned theoretical methods generally face a fundamental "double black box" integration dilemma in engineering practice. The underlying firmware and core control algorithms of mainstream drone flight controllers (such as DJI) and commercial robotic arm controllers are typically closed, not providing users with access to internal states or parameter modification interfaces. This barrier prevents the deployment of most advanced control algorithms that require access to internal states or modification of underlying control laws (such as the first model method), severely limiting the engineering application of the technology.

[0007] Therefore, there is a huge gap between existing technological approaches and the real market's demand for "reliable results, low cost, and easy deployment". Summary of the Invention

[0008] This invention provides an adaptive cooperative control method based on time-domain decoupling and its application, addressing the problems of existing technologies, such as reliance on complex dynamic models that are difficult to maintain robustness in real-world environments, the use of inefficient static operating modes, and the difficulty in achieving reliable engineering deployment due to the "double black box" barrier of commercial hardware.

[0009] The core technology of this invention is to creatively transform the complex spatial domain strongly coupled dynamics problem into a logical control problem that dynamically schedules the time process of upper-layer tasks (i.e., normal execution, active pause, or abort and rollback) based on the real-time stability of the platform perceived by the outside world.

[0010] In a first aspect, the present invention provides an adaptive cooperative control method based on time-domain decoupling, applied to a composite robot system including a mobile platform and an operating device, comprising the following steps:

[0011] a) Acquire real-time status data of the mobile platform through an external sensing system that is independent of the mobile platform and the internal controller of the operating device;

[0012] b) Based on real-time status data, determine a platform stability error index to characterize the stability of the mobile platform;

[0013] c) Based on the platform stability error index, dynamically adjust the time progress of the tasks executed by the operating equipment. Dynamic adjustment includes:

[0014] When the platform stability error index is lower than the preset stability threshold, the task of operating the device is authorized or continues.

[0015] When the platform stability error index is higher than the stability threshold, the task of operating the device is suspended to provide a stable recovery time window for the mobile platform.

[0016] d) Sending high-level motion commands to the respective low-level controllers of the mobile platform and the operating device to drive the composite robot system, wherein the method operates without relying on the dynamic model of the composite robot system.

[0017] Furthermore, dynamic regulation also includes:

[0018] When the platform stability error index is higher than the preset instability threshold, the current task of the operating device is terminated and it is returned to the preset safe state, where the instability threshold is greater than the stability threshold.

[0019] Furthermore, the platform stability error index is determined based on the spatial deviation between the real-time pose of the mobile platform and the target pose.

[0020] Furthermore, higher-level motion commands include velocity commands sent to the mobile platform and target pose or relative displacement commands sent to the operating device.

[0021] Furthermore, the external sensing system is a visual sensing system that acquires real-time status data by processing image data.

[0022] Secondly, the present invention provides an adaptive cooperative control device based on time-domain decoupling, applied to a composite robot system including a mobile platform and an operating device, comprising:

[0023] a) Status acquisition module, used to acquire real-time status data of the mobile platform through an external sensing system independent of the mobile platform and the internal controller of the operating device;

[0024] b) Error determination module, used to determine a platform stability error index to characterize the stability of the mobile platform based on real-time status data;

[0025] c) The task arbitration module is used to generate control decisions for the tasks performed by the operating equipment based on the platform stability error index. The control decisions include:

[0026] When the platform stability error index is lower than the preset stability threshold, an authorization signal for task execution or continuation is generated.

[0027] When the platform stability error index exceeds the stability threshold, a command signal to pause the task is generated.

[0028] d) The instruction distribution module is used to send high-level motion instructions to the underlying controllers of the mobile platform and the operating device, and to control the timing of sending instructions to the operating device according to the control decision of the task arbitration module. The system does not need to load the dynamic model of the composite robot system during operation.

[0029] Furthermore, the task arbitration module is also used for:

[0030] When the platform stability error index is higher than the preset instability threshold, an instruction signal is generated to terminate the current task and return to the preset safe state, wherein the instability threshold is greater than the stability threshold.

[0031] Furthermore, the platform stability error index is determined based on the spatial deviation between the real-time pose of the mobile platform and the target pose; the high-level motion commands issued by the command distribution module include velocity commands sent to the mobile platform and target pose or relative displacement commands sent to the operating device; the external sensing system is a visual sensing system, and the state acquisition module obtains real-time state data by processing image data.

[0032] Thirdly, the present invention provides an electronic device including a memory and a processor, wherein the memory stores a computer program and the processor is configured to run the computer program to execute the above-described adaptive cooperative control method based on time-domain decoupling.

[0033] Fourthly, the present invention provides a readable storage medium storing a computer program, the computer program including program code for controlling a process to execute the process, the process including the adaptive cooperative control method based on time-domain decoupling described above.

[0034] The main contributions and innovations of this invention are as follows:

[0035] 1. Superior versatility and engineering practicality: This invention adopts a "dual black box" compatible and "model-free" design, eliminating the need to access or modify the underlying firmware and control algorithms of the mobile platform and operating device. This allows for rapid deployment on hardware combinations from any manufacturer in a "plug-and-play" manner, fundamentally solving the "dual black box" integration dilemma of existing technologies, greatly reducing the technical threshold and shortening the development cycle.

[0036] 2. Strong Robustness and Environmental Adaptability: The invention's unique control paradigm establishes a "pause-resume-reset" mechanism by dynamically adjusting task timelines based on platform stability. When external disturbances cause platform instability, the system proactively pauses the task and resumes only after the platform stabilizes, rather than directly causing task failure. This strategy, which sacrifices temporary efficiency for reliable task completion, results in an extremely high success rate in real, complex, and variable environments.

[0037] 3. Advanced Control Simplified from Complexity: This invention cleverly employs a top-level strategy to decompose a complex nonlinear, strongly coupled system control problem into two relatively independent and simple sub-problems: "platform self-stability" and "quasi-static equipment operation." It utilizes simple feedback control logic to achieve advanced adaptive and synergistic effects, serving as a paradigm for solving complex problems with simple methods.

[0038] 4. Disruptive cost structure advantage: Since it eliminates the need for expensive and time-consuming system dynamics modeling, identification and debugging processes, and is compatible with low-cost consumer-grade hardware platforms, this invention fundamentally reduces the hardware and R&D costs of mobile operating robots, paving the way for the commercialization and popularization of the technology.

[0039] Details of one or more embodiments of the present invention are set forth in the following drawings and description, so that other features, objects and advantages of the invention will be more readily understood. Attached Figure Description

[0040] The accompanying drawings, which are included to provide a further understanding of the invention and form part of this invention, illustrate exemplary embodiments of the invention and are used to explain the invention, but do not constitute an undue limitation of the invention. In the drawings:

[0041] Figure 1 This is an architecture diagram of an adaptive cooperative control system based on time-domain decoupling according to an embodiment of the present invention;

[0042] Figure 2 This is a flowchart of an adaptive cooperative control method based on time-domain decoupling according to an embodiment of the present invention;

[0043] Figure 3 This is a structural diagram of a collaborative control system according to an embodiment of the present invention;

[0044] Figure 4 yes Figure 3 Another perspective view;

[0045] Figure 5 This is a schematic diagram of an outdoor test according to an embodiment of the present invention;

[0046] Figure 6 This is a real-time image based on an embodiment of the present invention (with added translation and annotation);

[0047] Figure 7 This is a schematic diagram of the hardware structure of an electronic device according to an embodiment of the present invention.

[0048] In the diagram, 1 is the mobile platform; 2 is the operating equipment; 3 is the external sensing system; 4 is the onboard computer; 5 is the visual tag; and 6 is the golf ball. Detailed Implementation

[0049] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numerals in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with one or more embodiments of this specification. Rather, they are merely examples of apparatuses and methods consistent with some aspects of one or more embodiments of this specification as detailed in the appended claims.

[0050] It should be noted that the steps of the corresponding methods are not necessarily performed in the order shown and described in this specification in other embodiments. In some other embodiments, the methods may include more or fewer steps than described in this specification. Furthermore, a single step described in this specification may be broken down into multiple steps in other embodiments; and multiple steps described in this specification may be combined into a single step in other embodiments.

[0051] This invention proposes a novel adaptive cooperative control method and system based on time-domain decoupling. To better illustrate this invention, the following application scenario of "an arm-mounted drone automatically picking up golf balls from the ground in a windy outdoor environment" will be used as a specific embodiment of this invention.

[0052] Example 1: Outdoor Golf Ball Autonomous Grabbing Unmanned Aerial Vehicle System

[0053] I. Overall System Composition

[0054] like Figure 1 As shown, the collaborative control system of this embodiment is materialized as a composite robot system consisting of a hardware platform and a software architecture (the various hardware and software parameters provided in this embodiment are preferred implementations and not the only solutions. Those skilled in the art can select other hardware or software processing to achieve the same function or better performance based on the actual situation and current technological development, but none of them can be regarded as departing from the protection scope of this invention).

[0055] 1. The hardware platform is shown in Table 1 below:

[0056] Table 1

[0057]

[0058] The composite robot in this embodiment consists of a mobile platform 1 and an operating device 2, such as Figure 3 and Figure 4 As shown:

[0059] Mobile Platform 1: A quadcopter drone whose flight controller uses Zero One Technology X6 (flight controller firmware is Ardupilot). In this embodiment, this flight controller is considered a "black box" controller, and the present invention does not require any modification to its internal parameters.

[0060] Operating device 2: A MicroSnow Electronics RoArm-M2-Pro four-degree-of-freedom robotic arm, mounted under the drone's fuselage. This robotic arm is also driven by its own "black box" controller.

[0061] External sensing system 3: An Orbbec Gemini 335 3D RGBD camera, fixed to the bottom of the body, whose field of view can simultaneously cover the end of the robotic arm and the grasping target below. This camera is a prerequisite for realizing the control method of this invention and constitutes the physical basis of the "state acquisition module".

[0062] Airborne computer 4: An AMD8845HS industrial computer, which serves as the computing core of the control system of this invention and runs all upper-level software algorithms.

[0063] 2. The software architecture is shown in Table 2 below:

[0064] Table 2

[0065]

[0066] The control system in this embodiment is built on the Ubuntu 20.04 operating system and is based on the Robot Operating System (ROSNoetic). The entire system consists of multiple functionally independent ROS nodes (i.e., software modules) that work together to achieve the system functions of this invention.

[0067] apm_node.py (Mobile Platform Interface Module): This node is responsible for bidirectional communication between the ROS system and the UAV flight controller. It converts high-level speed commands (geometry_msgs / Twist type) issued from the upper layer into MAVLink commands executable by the flight controller, enabling interaction with the mobile platform's "black box" controller. Interface descriptions are shown in Table 3 below:

[0068] Table 3

[0069]

[0070] The ROS packages depend on rospy, td_msgs, sensor_msgs, geometry_msgs, and rosgraph_msgs; the Python libraries depend on dronekit, pymavlink, psutil, threading, and argparse.

[0071] arm_node.py (Operating Device Interface Module): This node is responsible for bidirectional communication between the ROS system and the robotic arm controller. It sends high-level pose commands (JSON format strings) from the upper layer to the robotic arm controller via serial port, enabling interaction with the "black box" controller of the operating device. Interface descriptions are shown in Table 4 below:

[0072] Table 4

[0073]

[0074] The ROS package depends on rospy, tf, tf2_ros, std_msgs, geometry_msgs, and sensor_msgs; the Python library depends on pyserial, json, and threading.

[0075] `vision_fusion_node.py` (State Acquisition Module): This node corresponds to the state acquisition module. It receives data from the RGBD camera, uses the YOLOv8 model to identify the grasping target (golf ball 6), and uses the April Tag visual label 5 to identify the robotic arm gripper. By fusing depth information, it calculates the 3D position of the target and gripper in the camera coordinate system. This node broadcasts this position information, which is entirely based on external observations, through the ROS TF coordinate transformation system, providing upper-level decision-making with reliable information independent of the internal states of the UAV and the robotic arm. Based on ground observation Data. The interface description is shown in Table 5 below:

[0076] Table 5

[0077]

[0078] The ROS package depends on rospy, tf2_ros, cv_bridge, sensor_msgs, and geometry_msgs; the Python library depends on numpy, opencv-python, pyapriltags, and axengine.

[0079] uav_control.py (core control module): This node is the core of this invention, integrating the functions of the error determination module, task arbitration module, and instruction distribution module. It subscribes to the TF information published by vision_fusion_node.py and, based on the core "time domain decoupling" logic of this invention, coordinates the actions of the drone and the robotic arm. Interface descriptions are shown in Table 6 below:

[0080] Table 6

[0081]

[0082] The ROS package includes dependencies such as rospy, tf2_ros, tf, std_msgs, sensor_msgs, and geometry_msgs; the Python library includes dependencies such as json, threading, math, and collections.deque.

[0083] II. Specific Control Methods and Procedures

[0084] like Figure 2 As shown below, the entire process of executing a single capture task in this embodiment will be described in detail to illustrate the adaptive collaborative control method of the present invention.

[0085] Step 1: Obtain the status and determine the error index

[0086] After the task begins, the error determination module in the `uav_control.py` node continuously listens for TF transformation information published by `vision_fusion_node.py` (the state acquisition module, the "core perception module" of this system; it does not rely on any internal state of the UAV flight controller or robotic arm controller, but directly processes image data from external RGBD cameras, calculates the 3D positions of the grasped target and robotic arm gripper using YOLOv8 and AprilTag technologies, and finally broadcasts this high-precision pose information externally through the ROSTF coordinate transformation system). It calculates the 3D spatial distance and orientation deviation between the UAV base (`base_link`) and the grasped target (`ball_closest`). This deviation is quantified as a platform stability error index `E`, directly reflecting the stability of the UAV's current position relative to the ideal grasping position.

[0087] Step 2: Dynamically adjust the task timeline

[0088] The task arbitration module within the uav_control.py node is the concrete executor of the "time domain decoupling" concept. Internally, it implements a dual-threshold state machine based on the platform stability error index E to achieve dynamic control of the robotic arm's task time progression.

[0089] 1. Stable state (STABLE) -> Authorize task execution:

[0090] When the error index E remains below a first preset threshold (e.g., horizontal error less than 2 cm, yaw angle less than 3 degrees), the task arbitration module determines that the UAV platform is in a high-precision stable hovering state, i.e., the STABLE state. At this time, it generates a task execution authorization signal, allowing the robotic arm state machine to enter the TRACK_AND_GRASP (tracking and grasping) state and begin moving towards the target. In other words, "when the platform stability error index is below a preset stability threshold, the task of the operating device is authorized or continues."

[0091] 2. Semi-stable state (SEMI_STABLE) -> Pause task:

[0092] During the movement of the robotic arm, if external factors such as wind disturbance cause the error index E to increase and exceed the stability threshold, but remain below the second, larger instability threshold, the task arbitration module determines that the platform has entered the SEMI_STABLE (semi-stable) state. At this point, it immediately generates a task pause command signal, forcing the robotic arm state machine to switch to the PAUSED_AWAITING_STABILITY (pause waiting for stabilization) state. In this state, the robotic arm immediately stops all movement and "freezes" in place, thus providing the drone platform with a pure, undisturbed window for stable recovery.

[0093] This precisely achieves the goal of "pausing the operation of the device when the platform stability error index exceeds the stability threshold." If the drone subsequently successfully suppresses the disturbance and brings the error index E back below the stability threshold, the task arbitration module will issue an authorization signal again, and the robotic arm will resume execution of the task from the point of interruption.

[0094] 3. Instability (POSITIONING) -> Task abort:

[0095] If the disturbance is too large, causing the error index E to exceed the preset instability threshold, the task arbitration module will determine that the platform has entered the POSITIONING (instability or positioning) state. At this time, it will generate a command signal to terminate the current grasping attempt, force the robotic arm state machine to enter the GRASP_FAILED_RETRY (grasping failure retry) logic, and command the robotic arm to safely return to the initial observation position, waiting for the next platform stabilization authorization.

[0096] Step 3: Send high-level commands

[0097] The instruction dispatch module within the uav_control.py node is responsible for translating decisions into hardware-executable commands.

[0098] For drones, it issues high-level speed commands to apm_node.py by publishing ROS messages of type geometry_msgs / Twist to the / apm_drone / target_cmd_vel topic.

[0099] For the robotic arm, it issues high-level relative displacement commands to arm_node.py by publishing JSON-formatted strings to topics such as / arm / move_relative.

[0100] Throughout the process, uav_control.py does not involve any dynamic models such as the drone's mass, thrust coefficient, or the forward and inverse kinematics of the robotic arm. It treats the flight controller and robotic arm controller as "black boxes" capable of autonomously executing high-level commands.

[0101] In summary, this embodiment, through the combination of the aforementioned hardware and software systems and specific control processes, fully realizes the technical solution of the present invention. It transforms a complex physical coupling problem into a simple logical scheduling problem, achieving a highly robust collaborative grasping task in a dynamic environment without relying on a dynamic model and while being fully compatible with "black box" hardware, thus fully verifying the innovation and practicality of the present invention.

[0102] Example 2

[0103] Based on the same concept, the application objective of this embodiment is:

[0104] like Figure 5 As shown, in an outdoor environment with random wind disturbances (wind speed ≤ 5 m / s), a quadcopter drone equipped with a 4-DOF robotic arm autonomously identifies a golf ball 6 (42.67 mm in diameter) on the ground and completes the entire collaborative operation of "positioning-grabbing-retrieval". In this embodiment, both the drone flight controller and the robotic arm controller are closed "black boxes" (without access to modify the underlying firmware and control laws). Through the "time domain decoupling" top-level control logic of this invention, highly robust collaboration without the need for dynamic models or underlying debugging is achieved, with a mission success rate of ≥ 90%.

[0105] Specifically, the adaptive cooperative control system of the present invention consists of a top-level control module (i.e., the airborne computer in Embodiment 1), a status acquisition module, an error determination module, and a task arbitration module. The hardware selection, software configuration, and functional implementation of each module are as follows:

[0106] 1. Top-level control module

[0107] Hardware carrier: An AMD 8845HS industrial computer (same as in Example 1) is used, integrated in the middle of the UAV fuselage to provide the computing and control core for the system;

[0108] Software framework: Consistent with the implementation example;

[0109] Interface abstraction unit:

[0110] This embodiment combines apm_node.py (mobile platform interface module) and arm_node.py (operation device interface module) from Embodiment 1, unifying them into an interface abstraction unit. This unit configures the UAV flight controller (ZeroOne Technology X6, firmware Ardupilot) and the robotic arm controller (Wildetech RoArm-M2-Pro) as "black box actuators" through a standardized communication protocol. The specific interface implementation is as follows:

[0111] For UAV flight control: Issue advanced speed commands (topic name / apm_drone / target_cmd_vel, message type geometry_msgs / Twist), which include x / y / z axis speeds (unit: m / s) and yaw rate (unit: rad / s). The flight control receives the commands via the MAVLink protocol (UDP / Serial communication) and autonomously completes the underlying attitude control.

[0112] For the robotic arm controller: issue advanced target pose commands (topic name / arm / move_relative, message type std_msgs / String, content is a JSON format string, such as {"x":50,"y":0,"z":-30,"speed":50}, where x / y / z are relative displacements (unit: mm), and speed is the motion speed (unit: mm / s)). The robotic arm controller receives the commands through the / dev / ttyUSB0 serial port and autonomously calculates the joint motion.

[0113] 2. Status Acquisition Module

[0114] Hardware components: The system uses an Orbbec Gemini 335 3D RGBD camera (mounted under the drone fuselage with the lens facing the ground, covering both the robotic arm gripper and the ground target area), paired with a Core Cocoon Technology AI acceleration module (used to accelerate target detection and pose calculation).

[0115] Functionality implementation:

[0116] This module outputs the state error E of the mobile platform (UAV) through a three-step process of "3D vision acquisition - data fusion - error quantization". The specific steps are as follows:

[0117] 3D visual acquisition: RGBD camera outputs color images in real time (resolution 1280×720, frame rate...) 6 0fps) and depth images (ranging range 0.1- 2 0m, accuracy ± 1.5 %);

[0118] Data fusion: such as Figure 6 As shown, the vision processing unit integrates the YOLOv8 deep learning object detection model to identify golf balls from color images and output their 2D bounding boxes; simultaneously, it uses the pyapriltags library to identify multiple April Tag visual labels 5 (total size 84mm × 80mm) attached to the robotic arm gripper, and calculates the gripper's 2D pose; the data fusion unit combines the center coordinates of the bounding boxes with the depth values ​​of the corresponding positions in the depth image, converting them into three-dimensional coordinates in the camera coordinate system (the ball's coordinates P_ball = (X... b ,Y b Z b ), the center coordinates of the gripper are P_gripper=(X g ,Y g Z g ));

[0119] Error quantization: such as Figure 6As shown, the coordinate transformation unit converts the coordinates in the camera coordinate system to the coordinates in the UAV base coordinate system (base_link) using a ROSTF tree, and calculates the three-dimensional pose deviation (i.e., state error E) of the UAV base relative to the golf ball—E is the Euclidean distance from the origin of the UAV base to the center of the golf ball, and the calculation formula is:

[0120] E=sqrt[(X b -X base )²+(Y b -Y base ) 2 +(Z b -Z base ) 2 ]

[0121] Among them, X base ,Y base Z base E represents the coordinates of the origin of the UAV base coordinate system, with E in meters (m).

[0122] 3. Error Determination Module

[0123] This module is integrated into the uav_control.py node of the top-level control module. It establishes a negative correlation mapping relationship between the state error E and the task time elapsed rate V_task by discretizing the threshold. The specific mapping rules are as follows (the threshold is determined based on the UAV hovering accuracy (±0.1m) and the robotic arm operating range (±100mm), as shown in Table 7 below:

[0124] Table 7

[0125]

[0126] 4. Task Arbitration Module

[0127] This module is also integrated into the uav_control.py node and consists of a drone state machine and a robotic arm state machine. Synchronous scheduling is achieved through the shared state variable drone_localization_status (with values ​​of STABLE / SEMI_STABLE / POSITIONING / SEARCHING). (The two state machines run in parallel and are synchronized through the shared variable drone_localization_status, thereby enabling complex collaborative grasping tasks.) The specific state logic is shown in Tables 8 and 9 below:

[0128] Table 8. Unmanned Aerial Vehicle (UAV) State Machine

[0129]

[0130] Table 9. Robotic Arm State Machine

[0131]

[0132] The drone state machine's responsibility is to schedule the drone platform, moving it to and maintaining it in the optimal operating position of the target, while continuously evaluating the stability of its positioning. The evaluation results are then broadcast as the `drone_localization_status` state variable, providing decision-making support for the collaborating partner (robotic arm state machine). The robotic arm (grasping) state machine's responsibility is to autonomously execute the complete physical grasping task sequence. To ensure the success rate and safety of the operation, it synchronizes its key actions with the platform stability status (`drone_localization_status`) broadcast by the drone state machine.

[0133] III. Steps for Implementing the Adaptive Cooperative Control Method

[0134] The following section, using the application scenario of this embodiment, breaks down the four core steps of the adaptive cooperative control method in detail, while incorporating "pause-resume" logic and "safe motion envelope" settings:

[0135] Step 1: Construct the top-level control system and abstract interface

[0136] The drone flight controller (Ardupilot) and the robotic arm controller (RoArm-M2-Pro) are treated as "black box actuators," and communication is established through the interface abstraction unit of the top-level control module:

[0137] On the UAV side: Start the apm_node.py node, which subscribes to the / apm_drone / target_cmd_vel topic published by the top-level control module, converts the geometry_msgs / Twist command into MAVLink protocol commands and sends them to the flight controller; at the same time, it collects GPS, attitude and other data from the flight controller and feeds them back to the top-level control module through the / apm_drone / current_local_location topic.

[0138] On the robotic arm side: Start the arm_node.py node, which subscribes to the / arm / move_relative topic published by the top-level control module and sends JSON format commands to the robotic arm controller via serial port; at the same time, it collects the joint angle and end effector position data of the robotic arm and feeds them back to the top-level control module through the / arm / end_effector_position topic.

[0139] Step 2: Deploy the status acquisition module and quantify the status error E

[0140] Start the `vision_fusion_node.py` node, which subscribes to the ` / camera / color / image_raw` (color image) and ` / camera / depth / image_raw` (depth image) topics published by the RGBD camera, and quantizes E according to the following process:

[0141] Run the YOLOv8 model on a color image to detect a golf ball and output its center coordinates (u,v) in the image.

[0142] Extract the depth value d (in meters) at position (u, v) from the depth image. Combine this with camera intrinsic parameters (e.g., focal length fx=600px, fy=600px, principal point x0=640px, y0=360px), and calculate the coordinates (Xc, Yc, Zc) of the sphere in the camera coordinate system using the perspective projection formula: Xc=(u-x0)*d / fx, Yc=(v-y0)*d / fy, Zc=d;

[0143] By using ROSTF transformation (camera coordinate system → UAV base coordinate system), (Xc,Yc,Zc) is transformed into (Xb,Yb,Zb), and E=sqrt[(Xb)] is calculated. 2 +(Yb) 2 +(Zb-1.2) 2 (Where Zb-1.2 is the z-axis deviation between the UAV base (height 1.2m) and the sphere, and sqrt is the square root.)

[0144] Step 3: Establish a negative correlation mapping relationship between E and V_task

[0145] In the uav_control.py node, preset the discretization threshold (stable interval E≤0.1m, semi-stable interval 0.1m<E≤0.3m, unstable interval E>0.3m), and define the mapping function V_task=f(E):

[0146] When E ∈ the stable interval, f(E) = 1;

[0147] When E ∈ the semi-stable interval, f(E) = 0;

[0148] When E ∈ the unstable interval, f(E) = -1.

[0149] Here is a code example:

[0150] # Define the stability gradation threshold

[0151] STABLE_THRESHOLD = 0.04# An error less than 4cm is considered "stable".

[0152] UNSTABLE_THRESHOLD = 0.08 # An error greater than 8cm is considered "instability".

[0153] def discretize_platform_status(platform_error_metric: float) ->str:

[0154] """

[0155] The continuous platform error values ​​are transformed into discrete logic states.

[0156] Args:

[0157] platform_error_metric: A floating-point number that combines various errors; the smaller the value, the more stable the system.

[0158] Returns:

[0159] A state string describing the current level of stability.

[0160] """

[0161] if platform_error_metric <STABLE_THRESHOLD:

[0162] # The error is extremely small, and it is within the core stable region.

[0163] return "STABLE"

[0164] elif platform_error_metric>UNSTABLE_THRESHOLD:

[0165] # The error is too large and has drifted out of the acceptable range.

[0166] return "UNSTABLE"

[0167] else:

[0168] # The error falls between these two values, within the "semi-stable" buffer zone that requires caution.

[0169] return "SEMI_STABLE"

[0170] # --- Usage Examples ---

[0171] # current_error = calculate_distance_to_target() # Assume this function returns 0.06

[0172] # system_status = discretize_platform_status(current_error)

[0173] # print(f"The current continuous error is {current_error:.2f}m, and the discrete status is: {system_status}")

[0174] # Output: Current continuous error is 0.06m, discrete state is: SEMI_STABLE

[0175] This code demonstrates a standalone decision function that takes continuous physical error values ​​and outputs a discrete logical state that characterizes the stability of the system.

[0176] Step 4: Dynamically schedule the working sequence of the mobile platform and operating equipment

[0177] (1) Initial stage: positioning and preparation

[0178] Drone state machine: After the mission is started, the drone takes off to an altitude of 1.5m. The vision module does not detect the ball, the shared state is set to SEARCHING, and the drone cruises along a spiral path.

[0179] Robotic arm state machine: Upon receiving the SEARCHING state, it enters the MOVING_TO_OBSERVATION state, moves to the pre-grasping posture, and waits.

[0180] (2) Stable phase: Authorized fetching (V_task=1)

[0181] When the vision module detects a ball (E=0.08m), the drone state machine switches to STABLE, and the shared state is updated to STABLE;

[0182] The robotic arm's state machine detects the STABLE state and switches from MOVING_TO_OBSERVATION to TRACK_AND_GRASP: it issues commands via / arm / move_relative (such as {"x":0,"y":0,"z":-20,"speed":20}) to control the gripper to descend slowly; at the same time, it adjusts the x / y axis displacement in real time based on visual feedback to ensure that the gripper is aligned with the ball; when the gripper contacts the ball and detects a grasping force ≥5N, it confirms that the grasping is successful.

[0183] (3) Disturbance phase: Pause and recovery (V_task=0)

[0184] If a sudden gust of wind occurs outdoors, causing E to increase to 0.2m (semi-stable range), the drone's state machine switches to SEMI_STABLE, and the P controller is activated to adjust the pose (e.g., x-axis velocity 0.16m / s, y-axis velocity 0).

[0185] The robotic arm state machine detects the SEMI_STABLE state, enters the PAUSED_AWAITING_STABILITY state, and immediately stops issuing / arm / move_relative commands, maintaining the current position of the gripper;

[0186] When the drone adjusts to bring E back to 0.09m (the stable range), the shared state switches back to STABLE, and the robotic arm continues to issue / arm / move_relative commands from the paused position (such as {"x":0,"y":0,"z":-10,"speed":20}) to complete the remaining grasping actions.

[0187] Here is sample code:

[0188] class ArmTaskController:

[0189] def __init__(self):

[0190] self.is_paused = False

[0191] def update(self, platform_status: str):

[0192] "Based on the platform's status, decide whether to execute or pause your own tasks."

[0193] # --- The core logic of "time injection" ---

[0194] if platform_status == "SEMI_STABLE" and not self.is_paused:

[0195] print("Platform semi-stable detected, 'pause time' injected (V_task = 0)")

[0196] self.pause_arm_movement()

[0197] self.is_paused = True

[0198] # --- Restore Logic ---

[0199] elif platform_status == "STABLE" and self.is_paused:

[0200] print("Platform has stabilized, task time has returned to normal (V_task = 1)")

[0201] self.resume_arm_movement()

[0202] self.is_paused = False

[0203] def pause_arm_movement(self):

[0204] # This is an abstract call to the underlying hardware.

[0205] print("[ACTION]>> Send zero-speed command, robotic arm stops moving.")

[0206] def resume_arm_movement(self):

[0207] # Resume previous exercise

[0208] print("[ACTION]>>Resume the robotic arm's previous tracking task.")

[0209] # --- Usage Examples ---

[0210] # arm_controller = ArmTaskController()

[0211] # platform_status = "STABLE" # Initial state

[0212] # arm_controller.update(platform_status) # Nothing happened

[0213] # platform_status = "SEMI_STABLE" # Platform begins to drift

[0214] # arm_controller.update(platform_status)

[0215] # Output: Platform semi-stable detected, 'pause time' injected (V_task = 0)

[0216] # Output: [ACTION]>> Send zero-speed command, the robotic arm stops moving.

[0217] This code snippet demonstrates how the upper-level controller manages the ebb and flow of task time based on discrete states.

[0218] (4) Instability phase: rollback and retry (V_task=-1)

[0219] If the gusts intensify further, causing E to increase to 0.35m (the instability range), the drone's state machine will switch to POSITIONING and initiate coarse adjustment (x-axis speed 0.3m / s).

[0220] The robotic arm state machine detects the POSITIONING state and enters the GRASP_FAILED_RETRY state: it opens the gripper and issues a rollback command ({"x":0,"y":0,"z":50,"speed":50}) via / arm / move_relative, returning to the MOVING_TO_OBSERVATION pose;

[0221] When the drone adjusts to reduce E to 0.07m (STABLE state), the robotic arm re-enters TRACK_AND_GRASP state and retryes the grasping.

[0222] Here is sample code:

[0223] # --- Task macro-state definition ---

[0224] # 'WAITING_FOR_AUTHORIZATION': Waiting for the platform to stabilize before obtaining operation authorization.

[0225] # 'ACTION_AUTHORIZED': Authorized, fine-grained operation in progress.

[0226] # 'RETREATING_TO_SAFE_STATE': Authorization revoked, returning to a safe location

[0227] current_task_state = 'WAITING_FOR_AUTHORIZATION'

[0228] def macro_time_manager(platform_status: str):

[0229] "Based on the platform's status, task processes are scheduled at a macro level."

[0230] global current_task_state

[0231] # --- The logic of "remain calm first, then delegate authority" ---

[0232] if current_task_state == 'WAITING_FOR_AUTHORIZATION':

[0233] if platform_status == "STABLE":

[0234] print("Platform is stable and meets authorization requirements.")

[0235] current_task_state = 'ACTION_AUTHORIZED'

[0236] print("[STATE CHANGE]>>Task status changed to: ACTION_AUTHORIZED")

[0237] # arm.start_grasping() # Execute the critical action

[0238] else:

[0239] print("Waiting for the platform to stabilize...")

[0240] # --- The logic of "revoking authorization if instability occurs" ---

[0241] elif current_task_state == 'ACTION_AUTHORIZED':

[0242] if platform_status == "UNSTABLE":

[0243] print("Platform instability, operation authorization revoked immediately!")

[0244] current_task_state = 'RETREATING_TO_SAFE_STATE'

[0245] print("[STATE CHANGE]>>Task status changed to: RETREATING_TO_SAFE_STATE")

[0246] # arm.emergency_retreat() # Force rollback

[0247] # --- Usage Examples ---

[0248] # macro_time_manager("SEMI_STABLE") # Output: Waiting for platform to stabilize...

[0249] # macro_time_manager("STABLE")# Output: Platform stable... Status changed to ACTION_AUTHORIZED

[0250] # macro_time_manager("UNSTABLE")# Output: Platform instability...state changed to RETREATING_TO_SAFE_STATE

[0251] This code uses a simplified state machine to demonstrate the macro-level time scheduling logic of "authorization" and "revocation of authorization".

[0252] (5) Closing phase: Task completed

[0253] After successfully grabbing the ball, the robotic arm's state machine switches to MOVING_TO_POSTGRASP and issues a retraction command ({"x":0,"y":0,"z":50,"speed":30}) to retract the ball to the underside of the robot body;

[0254] The drone's state machine maintains the STABLE state, and after the robotic arm reaches its position, it controls the drone to return to home.

[0255] Additional steps: Preset safety sports cable management

[0256] To ensure that the robotic arm's movements match the drone's control capabilities, the "safe motion envelope" for the robotic arm is preset in the arm_node.py node:

[0257] Maximum speed: x / y / z axes ≤ 50 mm / s;

[0258] Maximum acceleration: ≤20mm / s² on x / y / z axes 2 ;

[0259] Static load verification: The maximum lift of the drone is ≥18kg, the weight of the robotic arm is 900g (including the gripper), and the static load accounts for ≤5%, which meets the drone's load-bearing capacity requirements.

[0260] IV. Verification of the Effects of the Examples

[0261] This embodiment conducted 50 grabbing tests in an outdoor wind speed environment of 0-5m / s, and the results are as follows:

[0262] Mission success rate: 74% (3 recovery attempts failed and the target was lost; 5 attempts failed due to strong gusts of wind during the final retrieval; 5 missions timed out).

[0263] Average crawling time: 45s (including the entire process of searching, locating, crawling, and retrieving).

[0264] Robustness performance: When E fluctuates in the range of 0.1-0.3m, the robotic arm can resume the task 94% of the time through the "pause-resume" logic, without failure caused by minor disturbances.

[0265] The results show that this embodiment fully realizes the technical solution of the present invention, and can solve the problem of coordinated control between the mobile platform and the operating device in a dynamic environment by means of time-domain decoupling logic under the conditions of "dual black box" hardware architecture and no dynamic model.

[0266] Example 3

[0267] This embodiment also provides an electronic device, see reference. Figure 2 It includes a memory 404 and a processor 402, wherein the memory 404 stores a computer program and the processor 402 is configured to run the computer program to perform the steps in any of the above method embodiments.

[0268] Specifically, the processor 402 may include a central processing unit (CPU), or an application-specific integrated circuit (ASIC), or one or more integrated circuits that can be configured to implement embodiments of the present invention.

[0269] Memory 404 may include a mass storage device for data or instructions. For example, and not limitingly, memory 404 may include a hard disk drive (HDD), a floppy disk drive, a solid-state drive (SSD), flash memory, an optical disk drive, a magneto-optical disk drive, magnetic tape, or a Universal Serial Bus (USB) drive, or a combination of two or more of these. Where appropriate, memory 404 may include removable or non-removable (or fixed) media. Where appropriate, memory 404 may be internal or external to a data processing device. In a particular embodiment, memory 404 is non-volatile memory. In a particular embodiment, memory 404 includes read-only memory (ROM) and random access memory (RAM). Where appropriate, the ROM may be a mask-programmed ROM, a programmable read-only memory (PROM), an erasable read-only memory (EPROM), an electrically erasable read-only memory (EEPROM), an electrically alterable read-only memory (EAROM), or flash memory, or a combination of two or more of these. Where appropriate, the RAM can be Static Random-Access Memory (SRAM) or Dynamic Random-Access Memory (DRAM). DRAM can be Fast Page Mode Dynamic Random-Access Memory (FPMDRAM), Extended Data Out Dynamic Random-Access Memory (EDODRAM), Synchronous Dynamic Random-Access Memory (SDRAM), etc.

[0270] The memory 404 can be used to store or cache various data files that need to be processed and / or communicated, as well as possible computer program instructions executed by the processor 402.

[0271] The processor 402 reads and executes computer program instructions stored in the memory 404 to implement any of the adaptive cooperative control methods based on time-domain decoupling in the above embodiments.

[0272] Optionally, the electronic device may further include a transmission device 406 and an input / output device 408, wherein the transmission device 406 is connected to the processor 402, and the input / output device 408 is connected to the processor 402.

[0273] The transmission device 406 can be used to receive or send data via a network. Specific examples of the network described above may include wired or wireless networks provided by the communication provider of the electronic device. In one example, the transmission device includes a Network Interface Controller (NIC), which can connect to other network devices via a base station to communicate with the Internet. In another example, the transmission device 406 may be a Radio Frequency (RF) module used for wireless communication with the Internet.

[0274] Input / output device 408 is used to input or output information.

[0275] Example 4

[0276] This embodiment also provides a readable storage medium storing a computer program, the computer program including program code for controlling a process to execute the process, the process including the adaptive cooperative control method based on time-domain decoupling according to Embodiment 1.

[0277] It should be noted that the specific examples in this embodiment can refer to the examples described in the above embodiments and optional implementations, and will not be repeated here.

[0278] Generally, various embodiments can be implemented in hardware or dedicated circuitry, software, logic, or any combination thereof. Some aspects of the invention can be implemented in hardware, while others can be implemented by firmware or software executed by a controller, microprocessor, or other computing device, but the invention is not limited thereto. Although various aspects of the invention may be shown and described as block diagrams, flowcharts, or using some other graphical representation, it should be understood that, by way of non-limiting example, these blocks, apparatuses, systems, techniques, or methods described herein can be implemented in hardware, software, firmware, dedicated circuitry or logic, general-purpose hardware or controllers or other computing devices, or some combination thereof.

[0279] Embodiments of the present invention can be implemented by computer software, which may be executable by a data processor of a mobile device, such as a processor entity, or by hardware, or by a combination of software and hardware. Computer software or programs (also referred to as program products) including software routines, applets, and / or macros can be stored in any device-readable data storage medium, and they include program instructions for performing specific tasks. The computer program product may include one or more computer-executable components configured to perform the embodiments when the program is run. The one or more computer-executable components may be at least one piece of software code or a portion thereof. Additionally, it should be noted in this respect that, as Figure 1 Any box in the logical flow can represent a program step, or interconnected logic circuits, boxes and functions, or a combination of program steps and logic circuits, boxes and functions. Software can be stored on physical media such as memory chips or blocks of storage implemented within a processor, magnetic media such as hard disks or floppy disks, and optical media such as DVDs and their data variants, CDs, etc. The physical medium is a non-transient medium.

[0280] Those skilled in the art should understand that the technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments have been described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0281] The above embodiments are merely illustrative of several implementations of the present invention, and their descriptions are relatively specific and detailed, but they should not be construed as limiting the scope of the present invention. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these all fall within the protection scope of the present invention. Therefore, the protection scope of the present invention should be determined by the appended claims.

Claims

1. An adaptive cooperative control method based on time-domain decoupling, applied to a composite robot system including a mobile platform and an operating device, characterized in that, Includes the following steps: a) Acquire real-time status data of the mobile platform through an external sensing system independent of the internal controller of the mobile platform and the operating device; b) Based on the real-time status data, determine a platform stability error index to characterize the stability of the mobile platform; c) Based on the platform stability error index, dynamically adjust the time process of the task executed by the operating device, wherein the dynamic adjustment includes: When the platform stability error index is lower than the preset stability threshold, the task of the operating device is authorized or continues to be executed. When the platform stability error index is higher than the stability threshold, the task of the operating device is suspended to provide a stable recovery time window for the mobile platform. When the platform stability error index is higher than the preset instability threshold, the current task of the operating device is terminated and it is returned to the preset safe state, wherein the instability threshold is greater than the stability threshold. d) Sending high-level motion commands to the respective underlying controllers of the mobile platform and the operating device to drive the composite robot system, wherein the method operates without relying on the dynamic model of the composite robot system.

2. The adaptive cooperative control method as described in claim 1, characterized in that, The platform stability error index is determined based on the spatial deviation between the real-time pose of the mobile platform and the target pose.

3. The adaptive cooperative control method as described in claim 1, characterized in that, The higher-level motion commands include speed commands sent to the mobile platform and target pose or relative displacement commands sent to the operating device.

4. The adaptive cooperative control method as described in claim 1, characterized in that, The external sensing system is a visual sensing system that acquires the real-time status data by processing image data.

5. An adaptive cooperative control system based on time-domain decoupling, applied to a composite robot system including a mobile platform and operating devices, characterized in that, include: a) Status acquisition module, used to acquire real-time status data of the mobile platform through an external sensing system independent of the mobile platform and the internal controller of the operating device; b) An error determination module, used to determine a platform stability error index to characterize the stability of the mobile platform based on the real-time status data; c) A task arbitration module, used to generate control decisions for the tasks performed by the operating device based on the platform stability error index, the control decisions including: When the platform stability error index is lower than the preset stability threshold, an authorization signal for task execution or continuation is generated; When the platform stability error index is higher than the stability threshold, a command signal to pause the task is generated; When the platform stability error index is higher than the preset instability threshold, an instruction signal is generated to terminate the current task and return to the preset safe state, wherein the instability threshold is greater than the stability threshold. d) An instruction distribution module is used to send high-level motion instructions to the underlying controllers of the mobile platform and the operating device, and to control the timing of sending instructions to the operating device according to the control decision of the task arbitration module. The system does not need to load the dynamic model of the composite robot system during operation.

6. The adaptive cooperative control system according to claim 5, characterized in that, The platform stability error index is determined based on the spatial deviation between the real-time pose of the mobile platform and the target pose; the high-level motion commands issued by the command distribution module include speed commands sent to the mobile platform and target pose or relative displacement commands sent to the operating device; the external sensing system is a visual sensing system, and the state acquisition module obtains the real-time state data by processing image data.

7. An electronic device comprising a memory and a processor, characterized in that, The memory stores a computer program, and the processor is configured to run the computer program to execute the adaptive cooperative control method based on time-domain decoupling as described in any one of claims 1 to 4.

8. A readable storage medium, characterized in that, The readable storage medium stores a computer program, the computer program including program code for controlling a process to execute the process, the process including the adaptive cooperative control method based on time-domain decoupling according to any one of claims 1 to 4.

Images