Heuristic type aerial mechanical arm grabbing control method and system

Abstract

The invention relates to the technical field of unmanned aerial vehicles, in particular to a heuristic type aerial mechanical arm grabbing control method and system.The heuristic type aerial mechanical arm grabbing control method comprises the steps that inertial parameters of a target object are estimated before grabbing; the air mechanical arm is controlled to grab a target object, and the inertial parameters are updated; and the control gain of the aerial mechanical arm is dynamically adjusted. The inertial parameters of the target object are estimated, the precision and the real-time performance are improved, inertial parameter estimation is rapidly completed, and the problem of estimation lag is solved. The stability and the accuracy of air operation are improved through an inertia perception gain scheduling control strategy. A heuristic type aerial mechanical arm grabbing control system comprises a sensor module. An inertial parameter estimation module; an inertial parameter updating module; and an adaptive control module.

Description

Technical Field

[0001] This application relates to the technical field of unmanned aerial vehicles, and particularly to a heuristic aerial manipulator grasping control method and system. Background Art

[0002] The aerial manipulator combines the flight ability of an unmanned aerial vehicle and the flexibility of a manipulator. The unmanned aerial vehicle can hover and move in complex environments that are difficult for traditional vehicles to reach, while the manipulator allows precise operation of objects, such as picking up and placing objects. This combination enables the aerial manipulator to perform tasks in challenging environments, such as cargo transportation under high-rise buildings, cliffs, tree tops, and bridges. However, despite the many advantages of the aerial manipulator, achieving precise, agile, and robust flight control still poses challenges.

[0003] The main challenge lies in the need to understand the system's inertial parameters in real time and accurately, which are crucial for stable control. Mass determines the translational dynamics of the system, the center of mass is crucial for system stability, and the moment of inertia is crucial for attitude control. Although the parameters of the aerial manipulator itself are usually known, during operation, due to load changes and adjustments to the manipulator's configuration, these parameters will change, introducing additional torques and control complexity. In addition, fluctuations in the ratio of the load mass to the total system mass may further affect flight stability and even increase the risk of accidents. Therefore, it is crucial to estimate these parameters in real time. In the prior art, inertial parameter estimation mainly relies on traditional methods such as recursive least squares or Kalman filtering, which often require excitation trajectories of dozens of seconds to converge to reasonable parameter estimates, resulting in the system being unable to adapt to load changes in a short time. In addition, the prior art starts inertial parameter estimation only after the target object is grasped, causing the flight control system to have to make adjustments under unknown inertial parameters, resulting in instability during the initial grasping phase, affecting flight trajectory accuracy and increasing system uncertainty. At the same time, traditional control methods, such as gain scheduling, usually rely on predefined operating points or static gain mappings and are unable to adjust control parameters in real time according to manipulator motion and load changes, resulting in insufficient flight stability in complex environments. Summary of the Invention

[0004] This application aims to at least partly solve one of the above technical problems in the prior art. To this end, embodiments of this application provide a heuristic aerial manipulator grasping control method and system, which predict the object's inertial parameters before target grasping and adjust the controller gain in real time through an inertial perception gain scheduling method to improve the precise manipulation ability.

[0005] A heuristic aerial manipulator grasping control method includes the following steps:

[0006] Based on visual perception data and a multi-modal reasoning model, estimate the inertial parameters of the target object before grasping, where the inertial parameters include mass, centroid position, and moment of inertia;

[0007] Control the aerial manipulator to grasp the target object, and update the inertial parameters based on sensor data after grasping;

[0008] Dynamically adjust the control gain of the aerial manipulator according to the updated inertial parameters to achieve flight attitude control of the aerial manipulator.

[0009] In an optional or preferred embodiment, the step of estimating the inertial parameters of the target object includes:

[0010] Use a visual segmentation model to identify and segment the target object in the visual image;

[0011] Use a pose estimation network to obtain the spatial pose and volume information of the target object;

[0012] Use a multi-modal reasoning model to predict the volume scaling factor, moment of inertia scaling factor, and density according to the shape and material characteristics of the target object;

[0013] Calculate the mass, moment of inertia, and centroid position of the target object according to the volume scaling factor, moment of inertia scaling factor, and density.

[0014] In an optional or preferred embodiment, the visual segmentation model is the Grounded SAM model, the pose estimation network is the IST-Net, and the multi-modal reasoning model is GPT-4.

[0015] In an optional or preferred embodiment, the step of updating the inertial parameters based on sensor data includes:

[0016] Use a non-linear disturbance observer to estimate the mass of the target object;

[0017] Recalculate the moment of inertia of the target object based on the mass estimated by the non-linear disturbance observer;

[0018] Obtain the position of the new centroid by mass weighting.

[0019] In an optional or preferred embodiment, the step of using a non-linear disturbance observer to estimate the mass of the target object includes:

[0020] Calculate the total thrust of the aerial manipulator through the thrust-rotation speed curve and motor rotation speed data;

[0021] Update the mass estimation value of the target object according to the acceleration, attitude, and total thrust data.

[0022] In an optional or preferred embodiment, the step of dynamically adjusting the control gain of the aerial manipulator according to the updated inertial parameters includes:

[0023] Establish an open-loop transfer function of the angular velocity loop of the aerial manipulator;

[0024] Calculate the adjustment value of the control gain matrix according to the proportional relationship between the total inertia moment and the initial inertia moment of the aerial manipulator;

[0025] Apply the adjustment value to the gain parameters of the angular velocity loop PID controller to achieve gain scheduling control.

[0026] In an optional or preferred embodiment, the aerial manipulator includes a quadrotor flight base and a Delta manipulator. The control of the Delta manipulator is based on inverse kinematics solution. By converting the desired position of the end effector into servo angles and angular velocities, proportional control combined with velocity feedforward compensation is used to achieve precise trajectory tracking. The flight base control adopts the differential flatness method, combined with cascade feedback control, using P+PID control in the position loop and PD+PID control in the attitude loop.

[0027] A heuristic aerial manipulator grasping control system for performing the heuristic aerial manipulator grasping control method described in any one of the above, includes:

[0028] A sensor module for acquiring visual images and depth information of the target object;

[0029] An inertial parameter estimation module for predicting the inertial parameters of the target object based on visual data and a multimodal inference model before grasping;

[0030] An inertial parameter update module for updating the inertial parameters based on sensor data after grasping the target object;

[0031] An adaptive control module for dynamically adjusting the control gain of the aerial manipulator according to the updated inertial parameters to achieve flight attitude control of the aerial manipulator.

[0032] Based on the above technical solutions, the embodiments of the present application have at least the following beneficial effects: The present application estimates the inertial parameters of the target object before grasping through visual perception data and a multimodal inference model, can estimate the inertial parameters of the target object in real time, significantly improves the accuracy and real-time performance of inertial parameter estimation. Through this pre-perception method, the inertial parameter estimation can be quickly completed, effectively solving the estimation lag problem in traditional methods. Secondly, the present invention adopts an inertial perception gain scheduling control strategy, which can dynamically adjust the control gain when the object mass and inertia moment change, thus significantly improving the stability and accuracy of the aerial manipulator when performing grasping tasks. Description of the Drawings

[0033] The present application will be further described below in conjunction with the accompanying drawings and embodiments;

[0034] Figure 1 is the overall flowchart of the heuristic aerial manipulator grasping control method provided by the embodiments of the present application;

[0035] Figure 2 is the flowchart of the target object inertial parameter estimation provided by the embodiments of the present application;

[0036] Figure 3 is the flowchart of the target object inertial parameter update provided by the embodiments of the present application;

[0037] Figure 4 is the adaptive control flowchart of the heuristic aerial manipulator grasping control method provided by the embodiments of the present application;

[0038] Figure 5 is the test verification schematic diagram of the aerial manipulator grasping the target object provided by the embodiments of the present application;

[0039] Figure 6 is the comparison schematic diagram of the inertial parameter estimation result and the true result provided by the embodiments of the present application. Specific embodiments

[0040] In order to enable those skilled in the art to better understand the technical solutions in the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present application.

[0041] The embodiments of the present application will be further described in detail below in conjunction with the accompanying drawings and embodiments. The following embodiments are used to illustrate the present application, but cannot be used to limit the scope of the present application.

[0042] In the description of the embodiments of the present application, it should be noted that the orientation or positional relationship indicated by the terms "center", "longitudinal", "transverse", "upper", "lower", "front", "rear", "left", "right", "vertical", "horizontal", "top", "bottom", "inner", "outer", etc. is based on the orientation or positional relationship shown in the accompanying drawings. It is only for the convenience of describing the embodiments of the present application and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation to the embodiments of the present application. In addition, the terms "first", "second", and "third" are only used for descriptive purposes and cannot be understood as indicating or implying relative importance.

[0043] In the description of the embodiments of the present application, it should be noted that unless otherwise clearly specified or limited, the terms "connected" and "coupled" should be understood in a broad sense. For example, it may be a fixed connection, a detachable connection, or an integral connection; it may be a mechanical connection or an electrical connection; it may be directly connected or indirectly connected through an intermediate medium. For those of ordinary skill in the art, the specific meanings of the above terms in the embodiments of the present application can be understood according to specific situations.

[0044] In the embodiments of the present application, unless otherwise clearly specified or limited, the first feature being "on" or "under" the second feature may mean that the first and second features are in direct contact, or the first and second features are indirectly in contact through an intermediate medium. Moreover, the first feature being "above", "over" and "on top of" the second feature may mean that the first feature is directly above or obliquely above the second feature, or merely indicates that the first feature has a higher horizontal height than the second feature. The first feature being "under", "beneath" and "underneath" the second feature may mean that the first feature is directly below or obliquely below the second feature, or merely indicates that the first feature has a lower horizontal height than the second feature.

[0045] The aerial manipulator combines the flight ability of an unmanned aerial vehicle (UAV) and the flexibility of a manipulator. The UAV can hover and move in complex environments that are difficult for traditional vehicles to reach, while the manipulator allows for precise operation of objects, such as picking up and placing objects. This combination enables the aerial manipulator to perform tasks in challenging environments, such as cargo transportation under high-rise buildings, cliffs, tree tops, and bridges. However, despite the many advantages of the aerial manipulator, achieving precise, agile, and robust flight control still poses challenges.

[0046] The main challenge lies in the need to understand the inertial parameters of the system in real-time and accurately, which are crucial for stable control. Mass determines the translational dynamics of the system, the center of mass is vital for system stability, and the moment of inertia is crucial for attitude control. Although the parameters of the aerial manipulator itself are usually known, during operation, due to load changes and adjustments in the manipulator's configuration, these parameters change, introducing additional torques and control complexity. Additionally, fluctuations in the ratio of the load mass to the total system mass can further affect flight stability and even increase the risk of accidents. Therefore, it is crucial to estimate these parameters in real-time. In the prior art, inertial parameter estimation mainly relies on traditional methods such as recursive least squares or Kalman filtering, which often require excitation trajectories of dozens of seconds to converge to reasonable parameter estimates, resulting in the system being unable to adapt to load changes in a short time. Moreover, in the prior art, inertial parameter estimation only starts after the target object is grasped, forcing the flight control system to adjust under unknown inertial parameters, leading to instability during the initial grasping phase, affecting flight trajectory accuracy and increasing system uncertainty. At the same time, traditional control methods such as gain scheduling usually rely on predefined operating points or static gain mappings and cannot adjust control parameters in real-time according to manipulator motion and load changes, resulting in insufficient flight stability in complex environments.

[0047] This application provides a heuristic aerial manipulator grasping control method. Based on visual perception data and a multi-modal reasoning model, this method pre-estimates the inertial parameters of the target object before grasping, controls the aerial manipulator to grasp the target object, updates the inertial parameters based on sensor data after grasping, and finally dynamically adjusts the control gain of the aerial manipulator according to the updated inertial parameters to achieve flight attitude control of the aerial manipulator. In this way, this application solves the technical problems in the prior art such as slow inertial parameter estimation, lagging control response, and insufficient flight stability, and improves the stability and accuracy of the aerial manipulator when performing grasping tasks.

[0048] In this application, inertial parameters refer to the physical properties that describe the reaction of an object or system to external forces and external torques during motion, including mass, center of mass position, and moment of inertia, which are the basis for precise control and motion planning. Traditional inertial parameter estimation methods usually start only after the object is grasped and require a long estimation time, resulting in the system being unable to adapt to load changes in a short time. This application greatly shortens the estimation time, improves the system response speed and stability by pre-estimating inertial parameters before grasping.

[0049] The following details the specific implementation of the heuristic aerial manipulator grasping control method of this application:

[0050] First, based on visual perception data and a multimodal reasoning model, estimate the inertial parameters of the target object before grasping. The inertial parameters include mass, center of mass position, and moment of inertia. Specifically, this step includes:

[0051] Refer to Figure 1 , Figure 2 , use a visual segmentation model to identify and segment the target object in the visual image. In a preferred embodiment of the present application, the visual segmentation model is the Grounded SAM (Segmentation and Attention Model) model. Grounded SAM is a deep learning-based visual model designed to detect and segment target objects by processing semantic information in images. The model can identify the target area in the image and perform precise segmentation by analyzing the text description and geometric features of the object in the image. For example, when the system receives an instruction to "grasp a cola can", the Grounded SAM model can accurately identify and segment the cola can area in the input image based on the input image and text information.

[0052] Use a pose estimation network to obtain the spatial pose and volume information of the target object. In a preferred embodiment of the present application, the pose estimation network is the IST-Net. IST-Net is a deep learning network model for estimating the pose of an object, which can estimate the 9D pose of an object in real time, including the rotation (orientation) and displacement (translation) information of the object. By inputting image data, the model can predict the position and direction of the target object without complete 3D scene reconstruction, thereby providing the necessary geometric information for estimating the inertial parameters of the object. IST-Net outputs the bounding box size and volume estimate of the object, and this information will be used for subsequent inertial parameter calculations.

[0053] During the visual segmentation process, the Grounded SAM model receives image data from a depth camera or an RGB-D camera, combines the input text data (such as "cup", "box", etc.) to locate and segment the target object, and outputs a 2D mask of the object. During the pose estimation process, the IST-Net network combines the RGB image and depth information to calculate the 9D pose (including rotation, translation) and volume of the target object.

[0054] Next, a multimodal reasoning model is used to predict the volume scaling factor, moment of inertia scaling factor, and density based on the shape and material characteristics of the target object. In a preferred embodiment of the present application, the multimodal reasoning model is GPT-4. GPT-4 is a natural language processing model based on deep learning that can generate and understand text. GPT-4 has powerful multimodal capabilities and can process text and image inputs. In the present application, GPT-4 is used to predict and adjust physical parameters such as the moment of inertia and volume of an object, especially for objects with complex shapes. For example, when the system identifies the target object as a "cola can", GPT-4 can infer reasonable volume scaling factors and moment of inertia scaling factors based on its understanding of the shape, material, and contents of the cola can. Assuming the object has a uniform density, GPT-4 can directly estimate the density value of the object.

[0055] Using the density information, moment of inertia scaling factor, and volume scaling factor provided by GPT-4, combined with the estimated volume result of the object, estimate the mass and moment of inertia of the target object. The specific calculation formulas are as follows:

[0056]

[0057] Where, is the estimated mass of the target object, is the estimated moment of inertia of the target object, is the estimated volume value of the object. l, w, and h respectively represent the length, width, and height of the circumscribed cuboid of the object.

[0058] The centroid position of the target object is calculated by combining the pose estimation network and the multimodal reasoning model.

[0059] Specifically, the size of the bounding box of the target object estimated by IST-Net, combined with the density value estimated by GPT-4, can obtain the centroid position of the target object.

[0060] Through the above steps, the estimated values of the mass, centroid position, and moment of inertia of the target object can be obtained. Figure 6 The comparison between the estimated result of the inertia parameter of the present application and the true value is shown.

[0061] Refer to Figure 3 , control the aerial manipulator to grasp the target object, and update the inertia parameter based on the sensor data after grasping.

[0062] Since the initial mass estimate value of the object may be inaccurate, the present application uses a nonlinear disturbance observer to estimate the mass of the object in real time. After grasping the object, the system can be regarded as being affected by a constant external force, so the nonlinear disturbance observer is very suitable for constant force estimation. The specific implementation method is as follows:

[0063] The total thrust of the aerial manipulator is calculated through the thrust - rotational speed curve and the motor rotational speed data. The total thrust is calculated through a pre - calibrated thrust - rotational speed curve and the real - time motor rotational speed data obtained through a bidirectional DShot electronic speed controller.

[0064] Assume that the UAV maintains a horizontal attitude during grasping, and the updated mass m o is calculated by the following formula:

[0065]

[0066] where, represents the linear acceleration of the flight base in the world coordinate system, m a represents the mass in the no - load state, and this value remains unchanged, m o is the mass of the target object, represents the change rate of the mass of the target object, which changes dynamically according to the sensor data, g represents the acceleration due to gravity, e3 = [0, 0, 1] T represents the unit vector, represents the rotation matrix from the body coordinate system to the world coordinate system, T represents the total thrust of the UAV. The parameter c controls the convergence speed. Once the mass of the object is updated, the corresponding moment of inertia is also recalculated to reflect the change in mass distribution. The updated moment of inertia J o is calculated according to the following formula:

[0067]

[0068] where J o is the updated moment of inertia, reflecting the dynamically updated mass m o . is the estimated mass of the target object, is the estimated moment of inertia of the target object. Subsequently, the position of the new centroid is obtained by mass weighting, and the moment of inertia at the new centroid is calculated by the parallel - axis theorem.

[0069] In this application, "based on sensor data" means using the real - time data collected by various sensors installed on the aerial manipulator. These sensors include, but are not limited to, inertial measurement units (IMUs), force / torque sensors, motor encoders, current sensors, etc. Specifically, the IMU provides acceleration and angular velocity data, the force / torque sensor measures the contact force and torque between the end - effector of the manipulator and the object, the motor encoder records the rotational speed of the thruster, and the current sensor measures the power consumption of the motor. These sensor data together form the basis for updating the inertial parameters.

[0070] Refer to Figure 4, dynamically adjust the control gain of the aerial manipulator according to the updated inertial parameters to achieve the flight attitude control of the aerial manipulator. Specifically, this step includes:

[0071] Establish the open-loop transfer function of the angular velocity loop of the aerial manipulator. The angular velocity loop uses a traditional PID controller, where the desired angular velocity is used as the input and the target output is the desired torque. Different from the standard angular velocity controller, the method of this application compensates for the change in moment of inertia caused by the movement of the manipulator in the angular velocity loop. Specifically, the total moment of inertia of the aerial manipulator is determined by its initial inertia when it does not grasp an object, the moment of inertia of the target object, and the servo angle of the manipulator. Therefore, when the moment of inertia of the target object is estimated and updated, we can directly perform gain scheduling in the angular velocity loop according to the current servo angle.

[0072] The open-loop transfer function of the angular velocity loop is:

[0073]

[0074] The adjustment of the control gain is closely related to the change in moment of inertia. Therefore, based on the change in moment of inertia, use the real-time estimated value J of the moment of inertia t to adjust the control gain matrix to achieve gain scheduling. K m represents the steady-state gain, which represents the ratio between the output motor speed and the input current of the motor system in the steady state. τ m represents the average time constant of the motor system, which represents the response speed of the system to input changes. s represents the complex frequency variable in the Laplace transform. respectively represent the proportional, integral, and derivative gains of the UAV angular rate controller.

[0075] In this process, initially the gain matrix is a 3×3 identity matrix, but as the object mass changes and the moment of inertia changes, the gain matrix will be adjusted according to the real-time estimated value J of the moment of inertia t Specifically, adjust the control gain according to the following method:

[0076]

[0077] Apply the adjusted value to the gain parameters of the angular velocity loop PID controller to achieve gain scheduling control. By dynamically adjusting the gain parameters of the PID controller, the control error caused by the change in moment of inertia can be compensated to ensure the accuracy and stability of flight control.

[0078] In this application, the aerial manipulator includes a quadrotor flight base and a Delta manipulator, adopting a decentralized control architecture, and designing independent control strategies for the Delta manipulator and the quadrotor flight base respectively. The control of the Delta manipulator is based on inverse kinematics solution. By converting the desired position of the end effector into servo angles and angular velocities, proportional control combined with velocity feedforward compensation is used to achieve precise trajectory tracking.

[0079] The control of the flight base adopts the differential flatness method, combined with cascaded feedback control. P+PID control is used in the position loop, and PD+PID control is used in the attitude loop. The desired attitude and thrust are calculated through the acceleration-yaw angle-attitude conversion module.

[0080] This application also provides a heuristic aerial manipulator grasping control system, including a sensor module, an inertial parameter estimation module, an inertial parameter update module, and an adaptive control module.

[0081] The sensor module is used to obtain the visual image and depth information of the target object, that is, depth image data. The inertial parameter estimation module is used to predict the inertial parameters of the target object based on visual data and a multi-modal inference model before grasping. The inertial parameter update module is used to update the inertial parameters based on sensor data after grasping the target object. The adaptive control module is used to dynamically adjust the control gain of the aerial manipulator according to the updated inertial parameters to achieve the flight attitude control of the aerial manipulator.

[0082] The sensor module includes an RGB-D camera, an IMU, a force / torque sensor, a motor speed sensor, a current sensor, etc. The RGB-D camera collects the color image and depth information of the target object, providing data input for visual segmentation and pose estimation. The IMU measures the acceleration, angular velocity, and attitude of the flight system, providing motion state data for the nonlinear disturbance observer. The force / torque sensor is installed at the end of the manipulator to measure the contact force and torque during grasping. The motor speed sensor measures the speed of each thruster for thrust calculation. The current sensor measures the real-time current of the motor to assist in judging the load change.

[0083] The inertial parameter estimation module includes a visual segmentation unit, a pose estimation unit, and a multi-modal inference unit. The visual segmentation unit achieves accurate segmentation of the target object based on the Grounded SAM model. The pose estimation unit calculates the pose and volume information of the object based on the IST-Net network. The multi-modal inference unit calls the GPT-4 model to predict the physical parameters according to the shape and material characteristics of the object. These units work together to preliminarily estimate the inertial parameters of the target object before grasping.

[0084] The inertial parameter update module includes a disturbance observation unit, a centroid calculation unit, and a moment of inertia calculation unit. The disturbance observation unit implements a non-linear disturbance observer to estimate the object mass based on the flight state and thrust data. The centroid calculation unit calculates the new system centroid position based on the mass weighted average method. The moment of inertia calculation unit applies the parallel axis theorem to calculate the system moment of inertia in the new centroid coordinate system. This module realizes the real-time update of inertial parameters after grasping.

[0085] The adaptive control module includes a transfer function analysis unit, a gain calculation unit, and a controller update unit. The transfer function analysis unit establishes the open-loop transfer function model of the system. The gain calculation unit calculates the new control gain according to the change ratio of the moment of inertia. The controller update unit applies the calculated gain to the flight control system. This module realizes the function of dynamically adjusting the control gain according to the updated inertial parameters.

[0086] Information is exchanged between each module through a data bus to form a complete control system. The sensor module provides data input for the inertial parameter estimation module and the inertial parameter update module; the inertial parameter estimation module provides initial parameters for the control system before grasping; the inertial parameter update module continuously updates the system parameters after grasping; the adaptive control module adjusts the control gain in real time according to the updated parameters to ensure flight stability.

[0087] Figure 5 This is a schematic diagram of the flight of an aerial manipulator grasping an object in this application.

[0088] Inspired by the perception and adaptation ability in the human grasping process, this application proposes an adaptive inertial estimation framework. It enables the aerial manipulator to adjust the control strategy in real time before and during the execution of tasks, thus solving the problem of lag in inertial parameter estimation in traditional technologies.

[0089] In addition, this application also provides an electronic device, including at least one processor and a memory communicatively connected to the at least one processor. The memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the above-mentioned heuristic aerial manipulator grasping control method.

[0090] This electronic device can be an embedded controller, including a main controller and an auxiliary controller. The main controller is responsible for the execution of flight control algorithms, including inertial parameter estimation, update, and gain scheduling control. The auxiliary controller is responsible for functions such as sensor data acquisition and processing, and manipulator motion control. The processor uses a high-performance ARM architecture CPU, with sufficient computing power to process complex vision algorithms and control algorithms. The memory includes ROM and RAM, where ROM stores system firmware and basic algorithms, and RAM is used for runtime data processing and temporary storage.

[0091] The electronic device can also be represented in various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present application described and / or claimed herein.

[0092] The electronic device may further include a communication interface for communicating with external devices and performing data interaction and transmission. Each device is interconnected using different buses and can be mounted on a common motherboard or otherwise as needed. The processor can process computer programs executed within the electronic device, including computer programs stored in or on the memory to display graphical information of a GUI on an external input / output device (such as a display device coupled to the interface). In other embodiments, if necessary, multiple processors and / or multiple buses can be used together with multiple memories and multiple memories. Similarly, multiple electronic devices can be connected, with each device providing some necessary operations (such as an array of servers, a set of blade servers, or a multiprocessing system). The bus can be divided into an address bus, a data bus, a control bus, etc.

[0093] Optionally, in a specific implementation, if the memory, processor, and communication interface are integrated on a single chip, the memory, processor, and communication interface can communicate with each other through an internal interface.

[0094] It should be understood that the above processor can be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor, etc. It is worth noting that the processor can be a processor that supports the advanced reduced instruction set machines (ARM) architecture.

[0095] Optionally, the memory may include a program storage area and a data storage area. The program storage area may store an operating system and application programs required for at least one function. The data storage area may store data created according to the use of the control electronic device of the vehicle projection lamp, etc. In addition, the memory may include a high-speed random access memory and may also include a non-transitory memory, such as at least one magnetic disk storage device, a flash memory device, or other non-transitory solid-state storage devices.

[0096] In some embodiments, the memory may optionally include memories generated remotely with respect to the processor, and these remote memories may be connected to the control electronic device of the vehicle projection lamp through a network. Examples of the above network include but are not limited to the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0097] This application also provides a non-transitory computer-readable storage medium storing computer instructions, and the computer instructions are used to cause a computer to execute the above-mentioned heuristic aerial manipulator grasping control method.

[0098] The storage medium may be a non-volatile storage device such as a flash memory, an EEPROM, an SD card, etc. The stored computer instructions include the code of the inertial parameter estimation module, the code of the inertial parameter update module, the code of the adaptive control module, and various auxiliary function codes.

[0099] Computer-readable media includes both permanent and non-permanent, removable and non-removable media and can store information by any method or technology. The information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage media, or any other non-transmission media that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transitory media such as modulated data signals and carrier waves.

[0100] Based on the above technical solutions, the embodiments of this application at least have the following beneficial effects:

[0101] Based on visual perception data and a multi-modal reasoning model, the present invention can estimate the inertial parameters of a target object in real time, significantly improving the accuracy and real-time performance of inertial parameter estimation. In traditional technologies, the estimation of inertial parameters usually requires a long excitation process and has low estimation accuracy. However, through a pre-perception method, the present invention can complete the estimation of inertial parameters within 2 seconds, with a mass estimation accuracy of 97.12% and an inertia moment estimation accuracy of 98.16%, effectively solving the problem of estimation lag in traditional methods. Secondly, the present invention adopts an inertial perception gain scheduling control strategy, which can dynamically adjust the control gain when the mass and inertia moment of the object change, thus significantly improving the stability and accuracy of the aerial manipulator when performing a grasping task. Based on the control method of the present invention, the flight position accuracy is improved by 43% compared with the traditional control method, ensuring the accuracy of object grasping and the stability of the aircraft. In addition, the innovation of the present invention lies in its ability to adapt to changing environments and object characteristics. Especially in the case of wind disturbances or complex object shapes, the system can still maintain efficient and stable operation. This advantage makes the present invention more flexible in applications in complex environments, greatly enhancing the adaptability and task execution efficiency of the aerial manipulator.

[0102] Through the combination of bionic inspiration, the present application simulates the mechanism of human grasping actions, enabling the aerial manipulator to perform pre-perception and real-time strategy adjustment during task execution, similar to the human's ability to adapt to dynamic changes in the environment and objects when grasping an object, thereby enhancing the grasping ability and flight control stability of the robot.

[0103] In the description of this specification, the descriptions referring to terms such as "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. mean that the specific features, structures, materials, or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the embodiments of the present application. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described can be combined in a suitable manner in any one or more embodiments or examples. In addition, without conflict, those skilled in the art can combine and combine the different embodiments or examples described in this specification and the features of different embodiments or examples.

[0104] The above has described the embodiments of the present application in detail with reference to the accompanying drawings. However, the present application is not limited to the above embodiments. Within the scope of knowledge possessed by those of ordinary skill in the art in the said technical field, various changes can be made without departing from the purpose of the present application.

Claims

1. A heuristic aerial manipulator grasping control method, characterized in that: The following steps are involved: Based on visual perception data and a multimodal reasoning model, the inertial parameters of the target object are estimated before grasping, and the inertial parameters include mass, center of mass position and moment of inertia; Controlling the aerial manipulator to grasp the target object, and updating the inertial parameters based on sensor data after grasping; The control gain of the aerial manipulator is dynamically adjusted according to the updated inertial parameters to achieve flight attitude control of the aerial manipulator.

2. The heuristic aerial manipulator grasping control method according to claim 1 is characterized in that: The step of estimating the inertial parameters of the target object comprises: Using a visual segmentation model to identify and segment the target object in the visual image; Using a pose estimation network to obtain the spatial pose and volume information of the target object; predicting a volume scaling factor, a moment of inertia scaling factor, and a density based on a shape and material characteristics of the target object using a multimodal reasoning model; The mass, moment of inertia and center of mass position of the target object are calculated according to the volume scaling factor, moment of inertia scaling factor and density.

3. The heuristic aerial manipulator grasping control method according to claim 2 is characterized in that: The visual segmentation model is the Grounded SAM model, the pose estimation network is IST-Net, and the multimodal reasoning model is GPT-4.

4. The heuristic aerial manipulator grasping control method according to claim 1, characterized in that: The step of updating the inertial parameters based on the sensor data comprises: estimating the mass of the target object using a nonlinear disturbance observer; recalculating the moment of inertia of the target object based on the mass estimated by the nonlinear disturbance observer; The position of the new center of mass is obtained by mass weighting.

5. The heuristic aerial manipulator grasping control method according to claim 4 is characterized in that: The step of estimating the mass of the target object using a nonlinear disturbance observer comprises: Calculate the total thrust of the aerial manipulator through the thrust-speed curve and motor speed data; The mass estimate of the target object is updated based on the acceleration, attitude, and total thrust data.

6. The heuristic aerial manipulator grasping control method according to claim 1, characterized in that: The step of dynamically adjusting the control gain of the aerial manipulator according to the updated inertia parameters comprises: Establish the open-loop transfer function of the angular velocity loop of the aerial manipulator; According to the proportional relationship between the total moment of inertia and the initial moment of inertia of the aerial manipulator, the adjustment value of the control gain matrix is ​​calculated; The adjustment value is applied to the gain parameter of the angular velocity loop PID controller to implement gain scheduling control.

7. The heuristic aerial manipulator grasping control method according to claim 1, characterized in that: The aerial manipulator includes a four-rotor flight base and a Delta manipulator. The control of the Delta manipulator is based on inverse kinematics solution. By converting the desired position of the end effector into a servo angle and angular velocity, proportional control combined with speed feedforward compensation is used to achieve accurate trajectory tracking. The flight base control adopts a differential flatness method combined with cascade feedback control, and P+PID control is used in the position loop, and PD+PID control is used in the attitude loop.

8. A heuristic aerial manipulator grasping control system, characterized in that: A heuristic aerial manipulator grasping control method for executing any one of claims 1 to 7, comprising: A sensor module for acquiring visual images and depth information of a target object; The inertial parameter estimation module is used to predict the inertial parameters of the target object based on visual data and multimodal reasoning models before grasping; An inertial parameter updating module, used for updating the inertial parameters based on sensor data after grasping the target object; The adaptive control module is used to dynamically adjust the control gain of the aerial manipulator according to the updated inertial parameters to achieve flight attitude control of the aerial manipulator.

Images