Heuristic aerial robotic arm grasping control method and system

Abstract

This application relates to the field of unmanned aerial vehicle (UAV) technology, and particularly to a heuristic aerial robotic arm grasping control method and system. The heuristic aerial robotic arm grasping control method estimates the inertial parameters of the target object before grasping; controls the aerial robotic arm to grasp the target object, updating the inertial parameters; and dynamically adjusts the control gain of the aerial robotic arm. Estimating the inertial parameters of the target object improves accuracy and real-time performance, quickly completing the inertial parameter estimation and solving the estimation lag problem. The inertial sensing gain scheduling control strategy improves the stability and accuracy of aerial operations. The heuristic aerial robotic arm grasping control system includes a sensor module; an inertial parameter estimation module; an inertial parameter updating module; and an adaptive control module.

Description

Technical Field

[0001] This application relates to the field of unmanned aerial vehicle (UAV) technology, and in particular to a heuristic aerial robotic arm grasping control method and system. Background Technology

[0002] Aerial robotic arms combine the flight capabilities of drones with the dexterity of robotic arms. Drones can hover and move in complex environments inaccessible to traditional vehicles, while robotic arms allow for precise manipulation of objects, such as picking up and placing them. This fusion enables aerial robotic arms to perform tasks in challenging environments, such as transporting goods between tall buildings, cliffs, treetops, and under bridges. However, despite the numerous advantages of aerial robotic arms, achieving precise, agile, and robust flight control remains a challenge.

[0003] The main challenge lies in understanding the system's inertial parameters in real time and accurately, which are crucial for stable control. Mass determines the system's translational dynamics, the center of mass is essential for system stability, and the moment of inertia is crucial for attitude control. Although the parameters of the aerial robotic arm itself are usually known, these parameters change during operation due to load variations and adjustments in the arm's shape, introducing additional torque and control complexity. Furthermore, fluctuations in the ratio of load mass to the total system mass can further affect flight stability and even increase the risk of accidents. Therefore, real-time estimation of these parameters is critical. In existing technologies, inertial parameter estimation mainly relies on traditional methods such as recursive least squares or Kalman filtering. These methods often require tens of seconds of excitation trajectory to converge to reasonable parameter estimates, causing the system to be unable to adapt to load changes in a short time. In addition, existing technologies only begin inertial parameter estimation after the target object is grasped, forcing the flight control system to adjust under unknown inertial parameters, leading to instability in the initial grasping phase, affecting flight trajectory accuracy, and increasing system uncertainty. Meanwhile, traditional control methods, such as gain scheduling, usually rely on predefined operating points or static gain mapping, and cannot adjust control parameters in real time according to the movement and load changes of the robotic arm, resulting in insufficient flight stability in complex environments. Summary of the Invention

[0004] This application aims to at least partially address one of the aforementioned technical problems in the prior art. To this end, embodiments of this application provide a heuristic aerial robotic arm grasping control method and system, which predicts the object's inertial parameters before grasping the target and adjusts the controller gain in real time through an inertial sensing gain scheduling method to improve precise control capabilities.

[0005] A heuristic aerial robotic arm grasping control method includes the following steps:

[0006] Based on visual perception data and a multimodal reasoning model, the inertial parameters of the target object are estimated before grasping. These inertial parameters include mass, center of mass position, and moment of inertia.

[0007] Control the aerial robotic arm to grasp the target object, and update the inertial parameters based on sensor data after grasping;

[0008] The control gain of the aerial robotic arm is dynamically adjusted based on the updated inertial parameters to achieve flight attitude control of the aerial robotic arm.

[0009] In an optional or preferred embodiment, the step of estimating the inertial parameters of the target object includes:

[0010] The visual segmentation model is used to identify and segment target objects in the visual image;

[0011] The spatial pose and volume information of the target object are obtained using a pose estimation network.

[0012] A multimodal inference model is used to predict the volume scaling factor, moment of inertia scaling factor, and density based on the shape and material characteristics of the target object;

[0013] The mass, moment of inertia, and center of mass of the target object are calculated based on the volume scaling factor, moment of inertia scaling factor, and density.

[0014] In an optional or preferred embodiment, the visual segmentation model is the Grounded SAM model, the pose estimation network is IST-Net, and the multimodal inference model is GPT-4.

[0015] In an optional or preferred embodiment, the step of updating the inertial parameters based on sensor data includes:

[0016] The mass of the target object is estimated using a nonlinear perturbation observer;

[0017] The moment of inertia of the target object is recalculated based on the mass estimated by the nonlinear perturbation observer;

[0018] The position of the new centroid is obtained by weighting by mass.

[0019] In an optional or preferred embodiment, the step of estimating the mass of the target object using a nonlinear perturbation observer includes:

[0020] The total thrust of the aerial robotic arm is calculated using the thrust-speed curve and motor speed data.

[0021] The mass estimate of the target object is updated based on acceleration, attitude, and total thrust data.

[0022] In an optional or preferred embodiment, the step of dynamically adjusting the control gain of the aerial robotic arm based on the updated inertial parameters includes:

[0023] Establish the open-loop transfer function of the angular velocity loop for the aerial robotic arm;

[0024] Based on the ratio of the total moment of inertia to the initial moment of inertia of the aerial robotic arm, the adjustment value of the control gain matrix is ​​calculated;

[0025] The adjustment value is applied to the gain parameter of the angular velocity loop PID controller to achieve gain scheduling control.

[0026] In an optional or preferred embodiment, the aerial robotic arm includes a quadcopter flight base and a Delta robotic arm. The control of the Delta robotic arm is based on inverse kinematics, which converts the desired position of the end effector into servo angles and angular velocities. Proportional control combined with velocity feedforward compensation is used to achieve precise trajectory tracking. The flight base control adopts a differential flatness method combined with cascaded feedback control. P+PID control is used in the position loop and PD+PID control is used in the attitude loop.

[0027] A heuristic aerial robotic arm grasping control system, used to execute the heuristic aerial robotic arm grasping control method described in any one of the above claims, comprising:

[0028] The sensor module is used to acquire visual images and depth information of the target object;

[0029] The inertial parameter prediction module is used to predict the inertial parameters of the target object based on visual data and a multimodal reasoning model before grasping.

[0030] An inertial parameter update module is used to update the inertial parameters based on sensor data after the target object is grasped.

[0031] An adaptive control module is used to dynamically adjust the control gain of the aerial robotic arm based on the updated inertial parameters, thereby achieving flight attitude control of the aerial robotic arm.

[0032] Based on the above technical solutions, the embodiments of this application have at least the following beneficial effects: This application uses visual perception data and a multimodal inference model to predict the inertial parameters of the target object before grasping, enabling real-time estimation of the target object's inertial parameters. This significantly improves the accuracy and real-time performance of inertial parameter estimation. This pre-sensing method allows for rapid inertial parameter estimation, effectively solving the estimation lag problem in traditional methods. Secondly, this invention employs an inertial perception gain scheduling control strategy, which dynamically adjusts the control gain when the object's mass and moment of inertia change, thereby significantly improving the stability and accuracy of the airborne manipulator when performing grasping tasks. Attached Figure Description

[0033] The present application will be further described below with reference to the accompanying drawings and embodiments;

[0034] Figure 1 This is a flowchart of the heuristic aerial robotic arm grasping control method provided in the embodiments of this application;

[0035] Figure 2 This is a flowchart of the target object inertial parameter prediction provided in an embodiment of this application;

[0036] Figure 3 This is a flowchart of the target object inertial parameter update process provided in an embodiment of this application;

[0037] Figure 4 This is an adaptive control flowchart of the heuristic aerial robotic arm grasping control method according to an embodiment of this application;

[0038] Figure 5 This is a schematic diagram of an experimental verification of an aerial robotic arm grasping a target object according to an embodiment of this application.

[0039] Figure 6 This is a schematic diagram comparing the inertial parameter estimation results with the actual results of an embodiment of this application. Detailed Implementation

[0040] To enable those skilled in the art to better understand the technical solutions in this application, the technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments in this application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of this application.

[0041] The embodiments of this application will be described in further detail below with reference to the accompanying drawings and examples. The following examples are used to illustrate this application, but should not be used to limit the scope of this application.

[0042] In the description of the embodiments of this application, it should be noted that the terms "center," "longitudinal," "lateral," "upper," "lower," "front," "rear," "left," "right," "vertical," "horizontal," "top," "bottom," "inner," and "outer," etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are only for the convenience of describing the embodiments of this application and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on the embodiments of this application. In addition, the terms "first," "second," and "third" are used for descriptive purposes only and should not be construed as indicating or implying relative importance.

[0043] In the description of the embodiments of this application, it should be noted that, unless otherwise explicitly specified and limited, the terms "connected" and "linked" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium. Those skilled in the art can understand the specific meaning of the above terms in the embodiments of this application based on the specific circumstances.

[0044] In the embodiments of this application, unless otherwise expressly specified and limited, "above" or "below" the second feature can mean that the first feature is in direct contact with the second feature, or that the first feature is in indirect contact with the second feature through an intermediate medium. Furthermore, "above," "on top of," and "over" the second feature can mean that the first feature is directly above or diagonally above the second feature, or simply indicates that the first feature is at a higher horizontal level than the second feature. "Below," "below," and "under" the second feature can mean that the first feature is directly below or diagonally below the second feature, or simply indicates that the first feature is at a lower horizontal level than the second feature.

[0045] Aerial robotic arms combine the flight capabilities of drones with the dexterity of robotic arms. Drones can hover and move in complex environments inaccessible to traditional vehicles, while robotic arms allow for precise manipulation of objects, such as picking up and placing them. This fusion enables aerial robotic arms to perform tasks in challenging environments, such as transporting goods between tall buildings, cliffs, treetops, and under bridges. However, despite the numerous advantages of aerial robotic arms, achieving precise, agile, and robust flight control remains a challenge.

[0046] The main challenge lies in understanding the system's inertial parameters in real time and accurately, which are crucial for stable control. Mass determines the system's translational dynamics, the center of mass is essential for system stability, and the moment of inertia is crucial for attitude control. Although the parameters of the aerial robotic arm itself are usually known, these parameters change during operation due to load variations and adjustments in the arm's shape, introducing additional torque and control complexity. Furthermore, fluctuations in the ratio of load mass to the total system mass can further affect flight stability and even increase the risk of accidents. Therefore, real-time estimation of these parameters is critical. In existing technologies, inertial parameter estimation mainly relies on traditional methods such as recursive least squares or Kalman filtering. These methods often require tens of seconds of excitation trajectory to converge to reasonable parameter estimates, causing the system to be unable to adapt to load changes in a short time. In addition, existing technologies only begin inertial parameter estimation after the target object is grasped, forcing the flight control system to adjust under unknown inertial parameters, leading to instability in the initial grasping phase, affecting flight trajectory accuracy, and increasing system uncertainty. Meanwhile, traditional control methods, such as gain scheduling, usually rely on predefined operating points or static gain mapping, and cannot adjust control parameters in real time according to the movement and load changes of the robotic arm, resulting in insufficient flight stability in complex environments.

[0047] This application provides a heuristic aerial robotic arm grasping control method. Based on visual perception data and a multimodal inference model, this method estimates the inertial parameters of the target object before grasping, controls the aerial robotic arm to grasp the target object, and updates the inertial parameters based on sensor data after grasping. Finally, it dynamically adjusts the control gain of the aerial robotic arm according to the updated inertial parameters to achieve flight attitude control. In this way, this application solves the technical problems of slow inertial parameter estimation, delayed control response, and insufficient flight stability in existing technologies, improving the stability and accuracy of the aerial robotic arm when performing grasping tasks.

[0048] In this application, inertial parameters refer to the physical characteristics describing the response of an object or system to external forces and torques during motion, including mass, center of mass position, and moment of inertia, and are the foundation for precise control and motion planning. Traditional inertial parameter estimation methods typically begin after the object is grasped and require a long estimation time, causing the system to be unable to adapt to load changes in a short period of time. This application, however, significantly shortens the estimation time and improves the system's response speed and stability by predicting inertial parameters before grasping.

[0049] The specific implementation method of the heuristic aerial robotic arm grasping control method of this application is described in detail below:

[0050] First, based on visual perception data and a multimodal reasoning model, the inertial parameters of the target object are estimated before grasping. These inertial parameters include mass, center of mass position, and moment of inertia. Specifically, this step includes:

[0051] Reference Figure 1 , Figure 2 This invention employs a visual segmentation model to identify and segment target objects in visual images. In a preferred embodiment, the visual segmentation model is a Grounded SAM (Segmentation and Attention Model) model. Grounded SAM is a deep learning-based visual model designed to detect and segment target objects by processing semantic information in images. This model can identify target regions in an image and perform accurate segmentation by analyzing text descriptions and geometric features of objects. For example, when the system receives the instruction "grab a Coke can," the Grounded SAM model can accurately identify and segment the Coke can region in the image based on the input image and text information.

[0052] The spatial pose and volume information of the target object are obtained using a pose estimation network. In a preferred embodiment of this application, the pose estimation network is IST-Net. IST-Net is a deep learning network model for estimating object pose, capable of estimating the 9D pose of an object in real time, including the object's rotation (attitude) and displacement (translation) information. This model can predict the position and orientation of the target object by inputting image data, without requiring complete 3D scene reconstruction, thus providing the necessary geometric information for estimating the object's inertial parameters. IST-Net outputs the bounding box dimensions and volume estimates of the object, which will be used for subsequent inertial parameter calculations.

[0053] During visual segmentation, the Grounded SAM model receives image data from a depth camera or an RGB-D camera, combines it with input text data (e.g., "cup," "box," etc.) to locate and segment the target object, and outputs a 2D mask of the object. During pose estimation, the IST-Net network combines RGB images and depth information to calculate the 9D pose (including rotation and translation) and volume of the target object.

[0054] Next, a multimodal inference model is used to predict the volume scaling factor, moment of inertia scaling factor, and density based on the shape and material characteristics of the target object. In a preferred embodiment of this application, the multimodal inference model is GPT-4. GPT-4 is a deep learning-based natural language processing model capable of generating and understanding text. GPT-4 possesses powerful multimodal capabilities, enabling it to process text and image inputs. In this application, GPT-4 is used to predict and adjust physical parameters such as the moment of inertia and volume of an object, especially for inferring about objects with complex shapes. For example, when the system identifies the target object as a "cola can," GPT-4 can infer reasonable volume scaling factors and moment of inertia scaling factors based on its understanding of the can's shape, material, and contents. Assuming the object has uniform density, GPT-4 can directly estimate the object's density value.

[0055] Using the density information, moment of inertia scaling factor, and volume scaling factor provided by GPT-4, combined with the object's volume estimation results, the mass and moment of inertia of the target object are estimated. The specific calculation formulas are as follows:

[0056]

[0057] in, It is the estimated mass of the target object. It is the estimated moment of inertia of the target object. This is an estimated volume of the object. l, w, and h represent the length, width, and height of the object's circumscribed cuboid, respectively.

[0058] The position of the target object's centroid is calculated by combining a pose estimation network with a multimodal inference model.

[0059] Specifically, the centroid location of the target object can be obtained by combining the bounding box size estimated by IST-Net with the density value estimated by GPT-4.

[0060] By following the steps described above, we can obtain estimates of the target object's mass, center of mass position, and moment of inertia. Figure 6 The figure shows a comparison between the estimated inertial parameters of this application and the actual values.

[0061] Reference Figure 3 The system controls an aerial robotic arm to grasp the target object and updates the inertial parameters based on sensor data after grasping.

[0062] Since the initial mass estimate of an object may be inaccurate, this application utilizes a nonlinear perturbation observer to estimate the object's mass in real time. After the object is grasped, the system can be considered to be subjected to a constant external force; therefore, the nonlinear perturbation observer is well-suited for constant force estimation. The specific implementation method is as follows:

[0063] The total thrust of the aerial robotic arm is calculated using the thrust-speed curve and motor speed data. The total thrust is calculated using a pre-calibrated thrust-speed curve and real-time motor speed data obtained through a bidirectional DShot ESC.

[0064] Assuming the drone maintains a horizontal attitude during the grasping process, the updated mass m o Calculated using the following formula:

[0065]

[0066] in, The linear acceleration of the flight platform in the world coordinate system is represented by m. a This represents the mass under no-load conditions, and this value remains constant. (m) o It is the mass of the target object. The rate of change of the target object's mass is represented by g, which changes dynamically based on sensor data. g represents the acceleration due to gravity, and e3 = [0,0,1]. T Represents a unit vector. The rotation matrix from the body coordinate system to the world coordinate system is represented by , and T represents the total thrust of the UAV. Parameter c controls the convergence speed. Once the object's mass is updated, the corresponding moment of inertia is recalculated to reflect changes in mass distribution. The updated moment of inertia J o Calculate using the following formula:

[0067]

[0068] J o It is the updated moment of inertia, reflecting the dynamically updated mass m. o . It is the estimated mass of the target object. This is the estimated moment of inertia of the target object. Then, the position of the new center of mass is obtained through mass weighting, and the moment of inertia at the new center of mass is calculated using the parallel axis theorem.

[0069] In this application, "based on sensor data" refers to real-time data collected using various sensors mounted on the aerial robotic arm. These sensors include, but are not limited to, inertial measurement units (IMUs), force / torque sensors, motor encoders, and current sensors. Specifically, the IMU provides acceleration and angular velocity data, the force / torque sensor measures the contact force and torque between the robotic arm's end effector and the object, the motor encoder records the rotational speed of the thruster, and the current sensor measures the motor's power consumption. This sensor data collectively forms the basis for updating the inertial parameters.

[0070] Reference Figure 4The control gain of the aerial robotic arm is dynamically adjusted based on the updated inertial parameters to achieve flight attitude control of the aerial robotic arm. Specifically, this step includes:

[0071] An open-loop transfer function for the angular velocity loop of an aerial robotic arm is established. The angular velocity loop employs a traditional PID controller, where the desired angular velocity is the input and the desired torque is the target output. Unlike standard angular velocity controllers, the method in this application compensates for changes in the moment of inertia caused by the robotic arm's motion within the angular velocity loop. Specifically, the total moment of inertia of the aerial robotic arm is determined by its initial inertia when it is not grasping an object, the moment of inertia of the target object, and the servo angle of the robotic arm. Therefore, once the moment of inertia of the target object is estimated and updated, gain scheduling can be directly performed within the angular velocity loop based on the current servo angle.

[0072] The open-loop transfer function of the angular velocity loop is:

[0073]

[0074] The adjustment of the control gain is closely related to the change in moment of inertia. Therefore, based on the change in moment of inertia, the real-time estimated value J of the moment of inertia is used. t To adjust the control gain matrix Implement gain scheduling. K m τ represents the steady-state gain, which is the ratio between the output motor speed and the input current of the motor system in steady state. m Let represent the average time constant of the motor system, which indicates the system's response speed to changes in input. Let s represent the complex frequency variable in the Laplace transform. These represent the proportional, integral, and derivative gains of the UAV angular rate controller, respectively.

[0075] In this process, the initial gain matrix It is a 3×3 identity matrix, but as the object's mass and moment of inertia change, the gain matrix will be adjusted based on the real-time estimated value of the moment of inertia J. t Make adjustments. Specifically, adjust the control gain according to the following method:

[0076]

[0077] The adjusted value is applied to the gain parameter of the angular velocity loop PID controller to achieve gain-scheduled control. By dynamically adjusting the gain parameter of the PID controller, control errors caused by changes in moment of inertia can be compensated, ensuring the accuracy and stability of flight control.

[0078] In this application, the aerial robotic arm includes a quadcopter flight base and a Delta robotic arm, employing a distributed control architecture with independent control strategies designed for the Delta robotic arm and the quadcopter flight base respectively. The control of the Delta robotic arm is based on inverse kinematics, converting the desired position of the end effector into servo angles and angular velocities, and using proportional control combined with velocity feedforward compensation to achieve precise trajectory tracking.

[0079] The flight base control adopts the differential flatness method, combined with cascaded feedback control. P+PID control is used in the position loop and PD+PID control is used in the attitude loop. The desired attitude and thrust are calculated through the acceleration-yaw angle-attitude conversion module.

[0080] This application also provides a heuristic aerial robotic arm grasping control system, including a sensor module, an inertial parameter prediction module, an inertial parameter update module, and an adaptive control module.

[0081] The sensor module acquires visual images and depth information of the target object, i.e., depth image data. The inertial parameter prediction module predicts the inertial parameters of the target object based on visual data and a multimodal inference model before grasping. The inertial parameter update module updates the inertial parameters based on sensor data after grasping the target object. The adaptive control module dynamically adjusts the control gain of the aerial robotic arm according to the updated inertial parameters, achieving flight attitude control of the aerial robotic arm.

[0082] The sensor module includes an RGB-D camera, an IMU (Integrated Measurement Unit), force / torque sensors, motor speed sensors, and current sensors. The RGB-D camera acquires color images and depth information of the target object, providing data input for visual segmentation and pose estimation. The IMU measures the acceleration, angular velocity, and attitude of the flight system, providing motion state data for the nonlinear disturbance observer. The force / torque sensor, mounted at the end of the robotic arm, measures the contact force and torque during the grasping process. The motor speed sensor measures the rotational speed of each thruster for thrust calculation. The current sensor measures the real-time current of the motors to aid in determining load changes.

[0083] The inertial parameter prediction module comprises a visual segmentation unit, a pose estimation unit, and a multimodal inference unit. The visual segmentation unit uses the Grounded SAM model to accurately segment the target object. The pose estimation unit calculates the object's pose and volume information using the IST-Net network. The multimodal inference unit calls the GPT-4 model to predict physical parameters based on the object's shape and material characteristics. These units work together to perform a preliminary prediction of the target object's inertial parameters before grasping.

[0084] The inertial parameter update module comprises a disturbance observation unit, a center-of-mass calculation unit, and a moment-of-inertia calculation unit. The disturbance observation unit implements a nonlinear disturbance observer, estimating the object's mass based on flight status and thrust data. The center-of-mass calculation unit calculates the new system center-of-mass position based on the mass-weighted average method. The moment-of-inertia calculation unit applies the parallel axis theorem to calculate the system moment of inertia in the new center-of-mass coordinate system. This module enables real-time updates of inertial parameters after capture.

[0085] The adaptive control module comprises a transfer function analysis unit, a gain calculation unit, and a controller update unit. The transfer function analysis unit establishes the open-loop transfer function model of the system. The gain calculation unit calculates the new control gain based on the proportional change in the moment of inertia. The controller update unit applies the calculated gain to the flight control system. This module enables dynamic adjustment of the control gain based on the updated inertial parameters.

[0086] The modules exchange information via a data bus, forming a complete control system. The sensor module provides data input to the inertial parameter prediction module and the inertial parameter update module; the inertial parameter prediction module provides initial parameters to the control system before capture; the inertial parameter update module continuously updates the system parameters after capture; the adaptive control module adjusts the control gain in real time based on the updated parameters to ensure flight stability.

[0087] Figure 5 This is a schematic diagram of the aerial robotic arm of this application grasping an object and flying.

[0088] Inspired by the perception and adaptation capabilities of humans during the grasping process, this application proposes an adaptive inertial estimation framework. This framework enables airborne operators to adjust their control strategies in real time before and during mission execution, thereby solving the problem of lag in inertial parameter estimation in traditional technologies.

[0089] Furthermore, this application also provides an electronic device, including at least one processor and a memory communicatively connected to the at least one processor. The memory stores instructions executable by the at least one processor, which, when executed, enable the at least one processor to perform the aforementioned heuristic aerial robotic arm grasping control method.

[0090] The electronic device can be an embedded controller, including a main controller and an auxiliary controller. The main controller is responsible for executing the flight control algorithm, including inertial parameter prediction, updating, and gain scheduling control. The auxiliary controller is responsible for sensor data acquisition and processing, robotic arm motion control, and other functions. The processor uses a high-performance ARM architecture CPU, which has sufficient computing power to handle complex vision and control algorithms. The memory includes ROM and RAM; ROM stores system firmware and basic algorithms, while RAM is used for runtime data processing and temporary storage.

[0091] Electronic devices can also be represented as various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. Electronic devices can also be represented as various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present application described and / or claimed herein.

[0092] The electronic device may also include a communication interface for communicating with external devices and exchanging data. The devices are interconnected using different buses and can be mounted on a common motherboard or otherwise as needed. The processor can process computer programs executed within the electronic device, including computer programs stored in or on memory to display graphical information of a GUI on external input / output devices (such as display devices coupled to the interface). In other embodiments, multiple processors and / or multiple buses can be used with multiple memories and multiple storage devices, if desired. Similarly, multiple electronic devices can be connected, each providing some of the necessary operations (e.g., as a server array, a group of blade servers, or a multiprocessor system). The bus can be divided into address buses, data buses, control buses, etc.

[0093] Optionally, in a specific implementation, if the memory, processor, and communication interface are integrated on a single chip, then the memory, processor, and communication interface can communicate with each other through an internal interface.

[0094] It should be understood that the aforementioned processor can be a Central Processing Unit (CPU), or other general-purpose processors, Digital Signal Processors (DSPs), Application Specific Integrated Circuits (ASICs), Field-Programmable Gate Arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. General-purpose processors can be microprocessors or any conventional processor. It is worth noting that the processor can be a processor supporting Advanced Reduced Instruction Set Machines (ARM) architecture.

[0095] Optionally, the memory may include a program storage area and a data storage area, wherein the program storage area may store the operating system and application programs required for at least one function; the data storage area may store data created based on the use of the control electronics for the vehicle's projection lights, etc. Furthermore, the memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device.

[0096] In some embodiments, the memory may optionally include memory remotely generated relative to the processor, which can be connected via a network to the control electronics of the vehicle's projection lights. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0097] This application also provides a non-transitory computer-readable storage medium storing computer instructions for causing a computer to execute the above-described heuristic aerial robotic arm grasping control method.

[0098] The storage medium can be a non-volatile storage device such as flash memory, EEPROM, or SD card, and the stored computer instructions include inertial parameter prediction module code, inertial parameter update module code, adaptive control module code, and various auxiliary function codes.

[0099] Computer-readable media include both permanent and non-permanent, removable and non-removable media that can store information by any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other physical classes of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic cassettes, magnetic disk storage or other magnetic storage media, or any other non-transferable media that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include non-transitory computer-readable media, such as modulated data signals and carrier waves.

[0100] Based on the above technical solution, the embodiments of this application have at least the following beneficial effects:

[0101] This invention, based on visual perception data and a multimodal inference model, can estimate the inertial parameters of a target object in real time, significantly improving the accuracy and real-time performance of inertial parameter estimation. Traditional inertial parameter estimation techniques typically require a lengthy excitation process and have low accuracy. This invention, through a pre-sensing method, can complete inertial parameter estimation within 2 seconds, achieving a mass estimation accuracy of 97.12% and a moment of inertia estimation accuracy of 98.16%, effectively solving the estimation lag problem in traditional methods. Secondly, this invention employs an inertial perception gain scheduling control strategy, dynamically adjusting the control gain when the object's mass and moment of inertia change, thereby significantly improving the stability and accuracy of the airborne operator during grasping tasks. Based on the control method of this invention, flight position accuracy is improved by 43% compared to traditional control methods, ensuring the precision of object grasping and the stability of the aircraft. Furthermore, the innovation of this invention lies in its adaptability to changing environments and object characteristics, especially under wind disturbances or complex object shapes, maintaining efficient and stable operation. This advantage makes the application of this invention more flexible in complex environments, greatly improving the adaptability and mission execution efficiency of the airborne operator.

[0102] This application combines biomimetic inspiration to simulate the mechanism of human grasping actions, enabling the aerial manipulator to pre-perceive and adjust its strategy in real time during mission execution, similar to the human's ability to adapt to dynamic changes in the environment and objects when grasping them, thereby improving the robot's grasping ability and flight control stability.

[0103] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., refer to specific features, structures, materials, or characteristics described in connection with that embodiment or example, which are included in at least one embodiment or example of the embodiments of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0104] The embodiments of this application have been described in detail above with reference to the accompanying drawings. However, this application is not limited to the above embodiments. Within the scope of knowledge possessed by those skilled in the art, various changes can be made without departing from the spirit of this application.

Claims

1. A heuristic aerial robotic arm grasping control method, characterized in that, Includes the following steps: Based on visual perception data and a multimodal inference model, the inertial parameters of the target object are estimated before grasping. These inertial parameters include mass, center of mass position, and moment of inertia. The steps for estimating the inertial parameters of the target object include: identifying and segmenting the target object in the visual image using a visual segmentation model; obtaining the spatial pose and volume information of the target object using a pose estimation network; predicting the volume scaling factor, moment of inertia scaling factor, and density based on the shape and material characteristics of the target object using a multimodal inference model; and calculating the mass, moment of inertia, and center of mass position of the target object based on the volume scaling factor, moment of inertia scaling factor, and density. The system controls an aerial robotic arm to grasp the target object and updates the inertial parameters based on sensor data after grasping. The step of updating the inertial parameters based on sensor data includes: estimating the mass of the target object using a nonlinear perturbation observer; recalculating the moment of inertia of the target object based on the mass estimated by the nonlinear perturbation observer; and obtaining the position of the new center of mass through mass weighting. The control gain of the aerial robotic arm is dynamically adjusted based on the updated inertial parameters to achieve flight attitude control of the aerial robotic arm. The step of dynamically adjusting the control gain of the aerial robotic arm based on the updated inertial parameters includes: establishing the open-loop transfer function of the angular velocity loop of the aerial robotic arm; calculating the adjustment value of the control gain matrix based on the ratio of the total moment of inertia of the aerial robotic arm to the initial moment of inertia; and applying the adjustment value to the gain parameter of the angular velocity loop PID controller to achieve gain scheduling control. The aerial robotic arm includes a quadcopter flight base and a Delta robotic arm. The control of the Delta robotic arm is based on inverse kinematics. By converting the desired position of the end effector into servo angle and angular velocity, precise trajectory tracking is achieved by using proportional control combined with velocity feedforward compensation. The control of the flight base adopts the differential flatness method combined with cascaded feedback control. P+PID control is used in the position loop and PD+PID control is used in the attitude loop.

2. The heuristic aerial robotic arm grasping control method according to claim 1, characterized in that, The visual segmentation model is the Grounded SAM model, the pose estimation network is IST-Net, and the multimodal inference model is GPT-4.

3. The heuristic aerial robotic arm grasping control method according to claim 1, characterized in that, The step of estimating the mass of the target object using a nonlinear perturbation observer includes: The total thrust of the aerial robotic arm is calculated using the thrust-speed curve and motor speed data. The mass estimate of the target object is updated based on acceleration, attitude, and total thrust data.

4. A heuristic aerial robotic arm grasping control system, characterized in that, The heuristic aerial robotic arm grasping control method for performing any one of claims 1 to 3 includes: The sensor module is used to acquire visual images and depth information of the target object; The inertial parameter prediction module is used to predict the inertial parameters of the target object based on visual data and a multimodal reasoning model before grasping. An inertial parameter update module is used to update the inertial parameters based on sensor data after the target object is grasped. An adaptive control module is used to dynamically adjust the control gain of the aerial robotic arm based on the updated inertial parameters, thereby achieving flight attitude control of the aerial robotic arm.

Images