Air control and learning method taking end effector as center

Abstract

The invention discloses an air control and learning method taking an end effector as a center, which comprises the following steps: constructing a reference coordinate system, and carrying out system dynamics modeling according to the reference coordinate system, including unmanned aerial vehicle dynamics modeling, mechanical arm kinematics modeling, motor dynamics modeling and model prediction control strategy MPC design. The L1 is used for self-adaptively compensating external disturbance and unknown disturbance suffered by the unmanned aerial vehicle and the mechanical arm motor in the movement process, and accurate movement control is achieved; and finally, constructing a teleoperation system based on vision and pose fusion and developing an action block strategy network ACT based on a Transform structure to realize strategy learning. The purposes of decoupling the task and the platform, improving the task execution efficiency control precision and robustness, effectively coping with system modeling errors and external disturbance, learning air control skills from human demonstration, having good task generalization ability and being capable of adapting to diversified air operation task requirements are achieved.

Description

Technical Field

[0001] The present invention relates to the technical field of aerial control of unmanned aerial vehicles (UAVs), and in particular to an aerial control and learning method centered on an end effector. Background Art

[0002] In recent years, unmanned aerial manipulators (UAMs) have garnered widespread attention from both industry and academia, with widespread application in performing complex aerial tasks such as tower light bulb replacement, aircraft inspection, and bridge maintenance. These tasks are complex and typically comprise multiple basic motion units, such as interaction, grasping, insertion, and rotation. However, most current aerial control systems are designed for specific tasks, with hardware design tightly coupled to the control framework, lacking cross-task and cross-platform versatility. End-effector-centric approaches have been widely adopted in robotic control, enabling platform- and task-independent policy development by decoupling high-level policy from low-level control. However, in the field of aerial control, the dynamics of the UAM's floating base and the coupling between the manipulator and the UAM present significant challenges in their implementation. Therefore, a universal and high-precision teleoperation and policy learning method for UAM aerial control is urgently needed. This paper uses a full-body model predictive controller (MPC) as the central interface, where the end-effector receives high-level policy instructions. The DH method is a Denavit–Hartenberg-based method. Summary of the Invention

[0003] The purpose of the present invention is to solve the problem in the prior art that most of the current air control systems are designed for specific tasks, the hardware design is tightly coupled with the control framework, and there is a lack of cross-task and cross-platform versatility.

[0004] In order to solve the above problems, the present invention provides an air control and learning method centered on an end effector, comprising: S1: Constructing reference coordinate system including: inertial world coordinate system , UAV body coordinate system , Robot arm base coordinate system , end effector coordinate system ; S2: System dynamics modeling includes: UAV dynamics modeling, robotic arm kinematics modeling, and motor dynamics modeling; S3: Design the model predictive control strategy MPC, construct the error function, and set the system constraints; System constraints include: aerial manipulator dynamics constraints, initial state constraints, operating state constraints and actuator output constraints; Operational state constraints include: self-collision avoidance constraints, environmental collision avoidance constraints, and safe operation constraints; S4: Using L1 adaptive compensation for external disturbances in the motion of the drone and robotic arm motors With unknown disturbance , precise motion control; S5: Build a teleoperation system based on vision and posture fusion; S6: Construct an action block strategy network ACT based on the Transformer structure.

[0005] In a preferred embodiment, step S1 of constructing a reference coordinate system includes: UAV body coordinate system The origin is located at the center of mass of the drone, and the coordinate axis Pointing to the front, right and top of the drone respectively; the coordinate system of the robotic arm base The origin is located at the connection between the robotic arm base and the drone, and the directions of the coordinate axes are the same as the drone body coordinate system; the end effector coordinate system The origin is located at the end of the robotic arm, and the coordinates are , 、 They point to the roll axis direction and horizontal direction of the fourth joint of the robotic arm respectively. = * ; Inertial world coordinate system of The direction of the axis is opposite to the direction of gravity. 、 Axis direction and same.

[0006] In a preferred embodiment, step S2 of system dynamics modeling includes: The UAV dynamics model uses the Newton-Euler equation to define the generalized position , generalized velocity , where p is the inertial world coordinate system Next, the coordinates of the center of mass of the UAV, is the UAV attitude rotation matrix, are the linear velocity and angular velocity of the body respectively, and the dynamic form is: in, is the inertia matrix, are the Coriolis force and centrifugal force terms, is the gravity term, Indicates adding a coordinate system subscript to the rotation matrix. is the normalized system control input, with a value range of [-1, 1], is an external disturbance, diag represents a diagonal matrix function generated according to a vector, For drone quality, is a 3*3 unit matrix, R is a rotation matrix, =[0, 0, -1]; The kinematic modeling of the robotic arm uses the DH method to define the joint angles , joint axial displacement , connecting rod length Angle between front and rear coordinate axes Parameter group , Represents a real vector space with a dimension of 4, i represents the i-th joint from the base of the robot to the end, then the coordinate system of the end of the robot is Relative to the robot base coordinate system Space transformation Expressed as: in, Represents the spatial transformation matrix of two adjacent joint coordinate systems, Indicates the current joint angle of the motor; Motor dynamics model, due to the delay in the dynamic response of the servo motor, the modeled motor response is a first-order lag system, the formula is: in, is the motor target joint angle, is the motor motion delay coefficient, is the unknown disturbance to the motor; S3: Design the model predictive control strategy MPC. In the MPC framework, the system state x and control u input are defined as: Construct the error function, the formula is: in, Indicates the current end position of the unmanned aerial robotic arm. Indicates the end position of the target unmanned aerial manipulator, 、 、 、 represents the error function, represents the matrix transpose, Indicates the current reference posture of the end of the robotic arm. is the reference posture of the target robotic arm end, Indicates the current posture of the unmanned aerial manipulator end. It is the vee operator, which converts the antisymmetric matrix Converted to a vector, is the generalized reference velocity of the end of the manipulator, is the reference angle of the robot arm motor, is the system reference control signal, is the current robot arm joint angle observation value, and the objective function is: in, is the optimal control sequence, argmin represents the optimization operator, represents the prediction window length, is the trajectory of the robotic arm, is the reference trajectory, is the reference trajectory for the prediction window length, and are quadratic functions of the tracking error, n represents the nth vector in the sequence; the system constraints are as follows: Corresponding to the dynamic constraints of the aerial manipulator , initial state constraints , operating state constraints and actuator output constraints , the continuous form of the aerial manipulator dynamics is given by the system dynamics modeling in step S2 above, which is expressed as , Represents the current state estimate of the system; the operating state constraint is expressed as the system trajectory in the reachable state set It includes: self-collision avoidance constraints: ensuring that the robotic arm does not collide with the drone's main structure during movement; environmental collision avoidance constraints: preventing the drone from contacting external obstacles; safe operation constraints: including speed limits and joint angle limits; The actuator output constraints impose actuator restrictions on the aerial manipulator, where and The lower limit and upper limit of the control input are defined respectively, the lower limit is [0, 0, 0, 0], and the upper limit is ; S4: Build L1 adaptive compensation for external disturbances in the motion of drone and manipulator motors With unknown disturbance , to achieve precise motion control; S4-1: The L1 compensation law of the drone is: in, represents the estimated generalized velocity, represents the L1 adaptive disturbance compensation term of the body, represents the projection gain matrix for velocity compensation, represents the exponential form of the projection matrix, represents the 6*6 identity matrix, represents the body differential disturbance compensation, Represents a low-pass filtering operation; S4-2: The compensation law of the robot arm is: in, represents the estimated angular velocity, is the motor L1 adaptive disturbance compensation term, is the perturbation compensation projection matrix, is the exponential term of the projection matrix, is the 4*4 identity matrix, is the motor differential disturbance compensation; S5: Build a teleoperation system based on vision and posture fusion; Using the RGB camera installed on the end effector of the robotic arm and the RGB camera installed on the base of the UAV, dual-view image information is captured in real time. The feature extraction network ResNet-18 is used to encode and fuse the dual-view image information to obtain image features. The obtained image features and real-time posture data are used to render the end effector target trajectory of the user's instructions in a three-dimensional virtual environment; with the end effector as the central interface, the spatial motion instructions of the robotic arm input by the user through the remote control device are converted into a command sequence of drone and robotic arm torque that can be executed by the robot through the whole-body model prediction controller. The command sequence is the target unmanned aerial robotic arm end position. and the target unmanned aerial manipulator end posture The instruction sequence; S6: Develop an action block strategy network ACT based on the Transformer structure. The output and observation definitions of the strategy network ACT are as follows: in, Indicates the ACT strategy, are network parameters, Indicates the current moment, is the action block size, 、 The RGB images are from the base camera and the end effector camera, respectively, with a resolution of 640*480. ResNet-18 is used as the backbone network for feature extraction. The RGB images are encoded and input into the Transformer encoder. Indicates the end pose of the target unmanned aerial manipulator.

[0007] The beneficial effects of the present invention are as follows: improved versatility and operational efficiency; decoupling of tasks and platforms is achieved through an end-effector-centric control paradigm, making it applicable to a variety of drone and robotic arm combinations; the proposed robotic arm end coordinate interface is intuitive and easy to use, significantly reducing operational complexity and improving task execution efficiency. Improved control accuracy and robustness. Combining full-body model predictive control with L1 adaptive compensation methods effectively addresses system modeling errors and external disturbances, achieving high-precision trajectory tracking of the end effector and robust control of the system. Supports imitation learning and multi-task adaptation. A Transformer-based imitation learning strategy network is constructed that can learn aerial manipulation skills from human demonstrations, possesses good task generalization capabilities, and can adapt to diverse aerial manipulation mission requirements. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] Figure 1 It is a schematic diagram of the system structure; Figure 2 Schematic diagram of the end-effector trajectory tracking experimental results; Figure 3 Schematic diagram of the end-effector trajectory tracking experimental results; Figure 4 Schematic diagram of the end-effector trajectory tracking experimental results; Figure 5 Schematic diagram of the end-effector trajectory tracking experimental results; Figure 6 Schematic diagram of the end-effector trajectory tracking experimental results; Figure 7 Schematic diagram of the end-effector trajectory tracking experimental results; Figure 8 Schematic diagram of the execution results of the imitation learning task; Figure 9 Schematic diagram of the execution results of the imitation learning task; Figure 10 Schematic diagram of the execution results of the imitation learning task; Figure 11 Schematic diagram of the execution results of the imitation learning task; Figure 12 It is a schematic diagram of the execution results of the flight process; Figure 13 This is a schematic diagram of the valve rotation execution results; Figure 14 It is a schematic diagram for pick-up and place execution; Figure 15 This is a schematic diagram of the result of disassembling the light bulb; Figure 16 This is a schematic diagram of the result of installing the light bulb; Figure 17 This is a schematic diagram of the results of the implementation of the present invention; Figure 18 Schematic diagram of the coordinate change curve of the end effector of the present invention; Figure 19 Schematic diagram of the reference coordinate system of the present invention. DETAILED DESCRIPTION

[0009] Example 1: System modeling and controller design;

[0010] This system features a fully-driven hexacopter drone capable of independently generating forces and torques along three axes. A robotic arm is mounted at the center of the drone. A fitting algorithm identifies inertial parameters and the arm's DH parameters, constructing a dynamic model of the entire system. This model can be used for multi-task simulation and verification.

[0011] Example 2: MPC trajectory controller solution and constraint design, MPC is a whole-body model predictive controller; The optimization solution is performed under the discrete prediction range H = 200 (2s). Trajectory error weight matrix After theoretical calculation, it is adjusted in the experiment to control the input range , Constraints are actual physical restrictions. Constraints include avoiding collisions with the environment, robot arm joint angle limits, and end acceleration limits.

[0012] Example 3: L1 adaptive controller implementation; For drone control input The disturbance compensation term is generated based on the estimated velocity error, and the filter bandwidth is obtained by Similarly, for each robot joint, set The controller smoothes the disturbance estimate through a low-pass filter. L1 adaptive control (L1-AC) is an advanced control method that estimates system uncertainty in real time and dynamically adjusts the control law. It is widely used in fields such as aircraft control.

[0013] Example 4: Remote operation interface and user input design; Use a remote control or joystick to control the end effector's 6-dimensional position, establishing a two-way communication channel between the user and the robot. Camera images are transmitted to the control terminal in real time, and 3D trajectory feedback enhances spatial perception. The system supports switching between autonomous and teleoperation.

[0014] Example 5: Imitation learning strategy training and deployment; Over 100 teleoperation demonstration trajectories were collected, taking as input an RGB image and the end-user pose, and outputting future trajectories. A CVAE architecture was used to encode the trajectory style during training, and a prior mean was used during testing to achieve generalization. Once deployed, the policy network enables efficient transfer to unknown tasks.

[0015] This system can be widely used in various task scenarios, including air writing, pin-in-hole insertion, precision grasping, light bulb replacement, etc., and has good platform scalability and task adaptability.

[0016] Example 6: MPC control algorithm implementation details; The optimal control problem in the end-effector-centric MPC is implemented using ACADOS, with a discrete time step of 25 milliseconds and a fixed prediction horizon of 2.5 seconds, running at a frequency of 100 Hz. The control output is executed in a rolling horizon manner, that is, in each iteration, only the first control input Applied to control signals.

[0017] Example 7: System identification implementation details; Since both L1 adaptive controller and MPC controller require accurate system modeling function To achieve effective control performance, we perform system identification to estimate the model uncertain parameters. We use two types of motion to excite the system: (1) While keeping the drone stationary, only the robot arm motion trajectory is executed to calibrate the DH parameters and joint servo delay parameters These trajectories ensure that the robot arm kinematic parameters and joint motor dynamics can accurately reflect the actual response of the robot arm. (2) Conduct UAV free flight trajectory experiments to identify the aircraft dynamic model.

[0018] Example 8: An end-effector-centric aerial control and learning method, including: The end effector, which receives high-level strategy instructions from the whole-body model predictive controller (MPC), is the central interface; the end effector-centered teleoperation system and imitation learning strategy; S1: Constructing reference coordinate system including: inertial world coordinate system , UAV body coordinate system , Robot arm base coordinate system , end effector coordinate system ; S2: System dynamics modeling includes: UAV dynamics modeling, robotic arm kinematics modeling, and motor dynamics modeling; S3: Design the model predictive control strategy MPC, construct the error function, and set the system constraints; Constraints include: Aerial manipulator dynamic constraints, initial state constraints, operating state constraints and actuator output constraints; Runtime constraints include: Self-collision avoidance constraints, environmental collision avoidance constraints, and safe operation constraints; S4: Using L1 adaptive compensation for external disturbances in the motion of the drone and robotic arm motors With unknown disturbance , precise motion control; S5: Construct a teleoperation system based on vision and posture fusion; S6: Construct an action block strategy network ACT based on Transformer structure, such as Figure 19 As shown, step S1 of constructing the reference coordinate system includes: UAV body coordinate system The origin is located at the center of mass of the drone, and the coordinate axis Pointing to the front, right and top of the drone respectively; the coordinate system of the robotic arm base The origin is located at the connection between the robotic arm base and the drone, and the directions of the coordinate axes are the same as the drone body coordinate system; the end effector coordinate system The origin is located at the end of the robotic arm, and the coordinates are , 、 They point to the roll axis direction and horizontal direction of the fourth joint of the robotic arm respectively. = * ; Inertial world coordinate system of The direction of the axis is opposite to the direction of gravity. 、 Axis direction and same.

[0019] Step S2 system dynamics modeling includes: The UAV dynamics model uses the Newton-Euler equation to define the generalized position , generalized velocity , where p is the inertial world coordinate system Next, the coordinates of the center of mass of the UAV, is the UAV attitude rotation matrix, are the linear velocity and angular velocity of the body respectively, and the dynamic form is: in, is the inertia matrix, are the Coriolis force and centrifugal force terms, is the gravity term, Indicates adding a coordinate system subscript to the rotation matrix. is the normalized system control input, with a value range of [-1, 1], is an external disturbance, diag represents a diagonal matrix function generated according to a vector, For drone quality, is a 3*3 unit matrix, R is a rotation matrix, =[0, 0, -1]; The kinematic modeling of the robotic arm uses the DH method to define the joint angles , joint axial displacement , connecting rod length Angle between front and rear coordinate axes Parameter group , Represents a real vector space with a dimension of 4, i represents the i-th joint from the base of the robot to the end, then the coordinate system of the end of the robot is Relative to the robot base coordinate system Space transformation Expressed as: in, Represents the spatial transformation matrix of two adjacent joint coordinate systems, Indicates the current joint angle of the motor; the motor dynamics model, due to the delay in the dynamic response of the servo motor, the modeled motor response is a first-order lag system, the formula is: in, is the motor target joint angle, is the motor motion delay coefficient, is the unknown disturbance to the motor; S3: Design the model predictive control strategy MPC. In the MPC framework, the system state x and control u input are defined as: Construct the error function, the formula is: in, Indicates the current end position of the unmanned aerial robotic arm. Indicates the end position of the target unmanned aerial manipulator; 、 、 、 represents the error function, represents the matrix transpose, Indicates the current reference posture of the end of the robotic arm; is the reference posture of the target robotic arm end, Indicates the current posture of the unmanned aerial manipulator end. It is the vee operator, which converts the antisymmetric matrix Converted to a vector, is the generalized reference velocity of the end of the manipulator, is the reference angle of the robot arm motor, is the system reference control signal, is the current robot arm joint angle observation value, and the objective function is: in, is the optimal control sequence, argmin represents the optimization operator, represents the prediction window length, is the trajectory of the robotic arm, is the reference trajectory, is the reference trajectory for the prediction window length, and are quadratic functions of tracking error, n represents the nth vector in the sequence; Consider the following system constraints: Corresponding to the dynamic constraints of the aerial manipulator , initial state constraints , operating state constraints and actuator output constraints ; The continuous form of the aerial manipulator dynamics is given by the system dynamics modeling in step S2 above, which is expressed as , Represents the current state estimate of the system; The running state constraint is expressed as the system trajectory in the set of reachable states The constraints include: self-collision avoidance constraints: ensuring that the manipulator does not collide with the drone's main structure during movement; environmental collision avoidance constraints: preventing the drone from contacting external obstacles; safe operation constraints: including speed limits and joint angle limits; actuator output constraints impose actuator restrictions on the aerial manipulator, including and The lower limit and upper limit of the control input are defined respectively, the lower limit is [0, 0, 0, 0], and the upper limit is ; S4: Build L1 adaptive compensation for external disturbances in the motion of drone and manipulator motors With unknown disturbance , to achieve precise motion control; S4-1: The L1 compensation law of the drone is: in, represents the estimated generalized velocity, represents the L1 adaptive disturbance compensation term of the body, represents the projection gain matrix for velocity compensation, represents the exponential form of the projection matrix, represents the 6*6 identity matrix, represents the body differential disturbance compensation, Represents a low-pass filtering operation; S4-2: The compensation law of the robot arm is: in, represents the estimated angular velocity, is the motor L1 adaptive disturbance compensation term, is the perturbation compensation projection matrix, is the exponential term of the projection matrix, is the 4*4 identity matrix, is the motor differential disturbance compensation; S5: Build a teleoperation system based on vision and posture fusion; Using the RGB camera installed on the end effector of the robotic arm and the RGB camera installed on the base of the UAV, dual-view image information is captured in real time. The feature extraction network ResNet-18 is used to encode and fuse the dual-view image information to obtain image features. The obtained image features and real-time posture data are used to render the end effector target trajectory of the user's instructions in a three-dimensional virtual environment; with the end effector as the central interface, the spatial motion instructions of the robotic arm input by the user through the remote control device are converted into a command sequence of drone and robotic arm torque that can be executed by the robot through the whole-body model prediction controller. The command sequence is the target unmanned aerial robotic arm end position. and the target unmanned aerial manipulator end posture The instruction sequence; S6: Develop an action block strategy network ACT based on the Transformer structure. The output and observation definitions of the strategy network ACT are as follows: in, Indicates the ACT strategy, are network parameters, Indicates the current moment, is the action block size, 、 The RGB images are from the base camera and the end effector camera, respectively, with a resolution of 640*480. ResNet-18 is used as the backbone network for feature extraction. The RGB images are encoded and input into the Transformer encoder. Indicates the end pose of the target unmanned aerial manipulator.

[0020] like Figure 2 、 Figure 3 、 Figure 4 As shown in the figure, the experimental results of the end-effector trajectory tracking are shown. In the figure, the black line represents the target value, the blue line represents the experimental value of the present invention, the orange dotted line represents the experimental value of the MPC model, and the green dotted line represents the experimental value of the L1 model. The horizontal axis represents time in seconds and the vertical axis represents distance in meters. like Figure 5 、 Figure 6 、 Figure 7 As shown in the figure, the results of the imitation learning task are shown. The blue line in the figure represents the experimental value and the gray line represents the target value. Figure 8 、 Figure 9 、 Figure 10 、 Figure 11 As shown in the figure, the results of the imitation learning task are shown. Figure 12 As shown, the results of the flight process are displayed; Figure 13 A schematic diagram showing the results of valve rotation execution is shown; Figure 14 To perform display results for pick and place, Figure 15 This is a schematic diagram of the result of disassembling the light bulb. Figure 16 For installation of light bulb results, such as Figure 17 As shown, a schematic diagram of the execution result is shown. In the figure, 1 is a schematic diagram of the drone taking off, 2 is a schematic diagram of the drone flying to the target, 3 is a schematic diagram of the drone inserting work, 4 is a schematic diagram of the drone opening the fixture, 5 is a schematic diagram of the drone returning, and 6 is a schematic diagram of the drone landing; Figure 18 As shown in the figure, a schematic diagram of the end effector coordinate change curve is shown. The horizontal axis represents time, unit: second / S, and the vertical axis represents position, unit: meter / m. Among them: the red line represents the end effector x-axis position curve, the green line represents the end effector y-axis position curve, the purple line represents the end effector z-axis position curve, and the black line represents the fixture position curve.

[0021] This method consists of two parts: end-of-arm control and policy learning. Precise end-of-arm control is the foundation of policy learning. End-of-arm control is defined as a model predictive control (MPC) problem, primarily consisting of the system modeling step (S2) and the MPC solution step (S3). Furthermore, due to modeling errors and unknown disturbances between the model and the real machine, an L1 adaptive control step (S4) is introduced to align the real machine's system responses with the modeled responses.

[0022] The above shows and describes the basic principles, main features and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The above embodiments and descriptions only illustrate the principles of the present invention. Without departing from the spirit and scope of the present invention, the present invention may have various changes and improvements, which fall within the scope of the present invention to be protected. The scope of protection of the present invention is defined by the attached claims and their equivalents.

Claims

1. An end-effector-centered aerial control and learning method, characterized in that: include: S1: Constructing a reference coordinate system includes: Inertial world coordinate system , UAV body coordinate system , Robot arm base coordinate system , end effector coordinate system ; S2: System dynamics modeling includes: UAV dynamics modeling, robotic arm kinematics modeling, motor dynamics model; S3: Design the model predictive control strategy MPC, construct the error function, and set the system constraints; System constraints include: Aerial manipulator dynamic constraints, initial state constraints, operating state constraints and actuator output constraints; Runtime constraints include: Self-collision avoidance constraints, environmental collision avoidance constraints, and safe operation constraints; S4: Using L1 adaptive compensation for external disturbances in the motion of the drone and robotic arm motors With unknown disturbance , achieving precise motion control; S5: Build a teleoperation system based on vision and posture fusion; S6: Construct an action block strategy network ACT based on the Transformer structure.

2. The end-effector-centered aerial control and learning method according to claim 1, characterized in that: Step S1 builds the reference coordinate system including: UAV body coordinate system The origin is located at the center of mass of the drone, and the coordinate axis Pointing to the front, right and top of the drone respectively; the coordinate system of the robotic arm base The origin is located at the connection between the robotic arm base and the drone, and the directions of the coordinate axes are the same as the drone body coordinate system; the end effector coordinate system The origin is located at the end of the robotic arm, and the coordinates are , 、 They point to the roll axis direction and horizontal direction of the fourth joint of the robotic arm respectively. = * ; Inertial world coordinate system of The direction of the axis is opposite to the direction of gravity. 、 Axis direction and same.

3. The end-effector-centered aerial control and learning method according to claim 1, characterized in that: Step S2 system dynamics modeling includes: The UAV dynamics model uses the Newton-Euler equation to define the generalized position , generalized velocity , where p is the inertial world coordinate system Next, the coordinates of the center of mass of the UAV, is the UAV attitude rotation matrix, are the linear velocity and angular velocity of the body respectively, and the dynamic form is: in, is the inertia matrix, are the Coriolis force and centrifugal force terms, is the gravity term, Indicates adding a coordinate system subscript to the rotation matrix. is the normalized system control input, with a value range of [-1, 1], is an external disturbance, diag represents a diagonal matrix function generated according to a vector, For drone quality, is a 3*3 unit matrix, R is a rotation matrix, =[0, 0, -1]; The kinematic modeling of the robotic arm uses the DH method to define the joint angles , joint axial displacement , connecting rod length Angle between front and rear coordinate axes Parameter group , represents a real vector space with a dimension of 4; i represents the i-th joint from the base of the robot to the end, then the coordinate system of the end of the robot is Relative to the robot base coordinate system Space transformation Expressed as: in, Represents the spatial transformation matrix of two adjacent joint coordinate systems, Indicates the current joint angle of the motor; Motor dynamics model, due to the delay in the dynamic response of the servo motor, the modeled motor response is a first-order lag system, the formula is: in, is the motor target joint angle, is the motor motion delay coefficient, is the unknown disturbance to the motor; S3: Design the model predictive control strategy MPC. In the MPC framework, the system state x and control u input are defined as: Construct the error function, the formula is: in, Indicates the current end position of the unmanned aerial robotic arm. Indicates the end position of the target unmanned aerial manipulator, 、 、 、 represents the error function, represents the matrix transpose, Indicates the current reference posture of the end of the robotic arm. is the reference posture of the target robotic arm end, Indicates the current posture of the unmanned aerial manipulator end. It is the vee operator, which converts the antisymmetric matrix Converted to a vector, is the generalized reference velocity of the end of the manipulator, is the reference angle of the robot arm motor, is the system reference control signal, is the current robot arm joint angle observation value, and the objective function is: in, is the optimal control sequence, argmin represents the optimization operator, represents the prediction window length, is the trajectory of the robotic arm, is the reference trajectory, is the reference trajectory for the prediction window length, and are quadratic functions of tracking error, n represents the nth vector in the sequence; Consider the following system constraints: Corresponding to the dynamic constraints of the aerial manipulator , initial state constraints , operating state constraints and actuator output constraints , the continuous form of the aerial manipulator dynamics is given by the system dynamics modeling in step S2 above, which is expressed as , Represents the current state estimate of the system; The running state constraint is expressed as the system trajectory in the set of reachable states Including: Self-collision avoidance constraint: ensures that the robotic arm does not collide with the drone's main structure during movement; Environmental collision avoidance constraints: prevent the drone from coming into contact with external obstacles; Safety operation constraints: including speed limits and joint angle limits; The actuator output constraints impose actuator restrictions on the aerial manipulator, where and The lower limit and upper limit of the control input are defined respectively, the lower limit is [0, 0, 0, 0], and the upper limit is ; S4: Build L1 adaptive compensation for external disturbances in the motion of drone and manipulator motors With unknown disturbance , to achieve precise motion control; S4-1: The L1 compensation law of the drone is: in, represents the estimated generalized velocity, represents the L1 adaptive disturbance compensation term of the body, represents the projection gain matrix for velocity compensation, represents the exponential form of the projection matrix, represents the 6*6 identity matrix, represents the body differential disturbance compensation, Represents a low-pass filtering operation; S4-2: The compensation law of the robot arm is: in, represents the estimated angular velocity, is the motor L1 adaptive disturbance compensation term, is the perturbation compensation projection matrix, is the exponential term of the projection matrix, is the 4*4 identity matrix, is the motor differential disturbance compensation; S5: Build a teleoperation system based on vision and posture fusion; Using the RGB camera installed on the end effector of the robotic arm and the RGB camera installed on the base of the UAV, dual-view image information is captured in real time. The feature extraction network ResNet-18 is used to encode and fuse the dual-view image information to obtain image features. The obtained image features and real-time posture data are used to render the end effector target trajectory of the user's instructions in a three-dimensional virtual environment; with the end effector as the central interface, the spatial motion instructions of the robotic arm input by the user through the remote control device are converted into a command sequence of drone and robotic arm torque that can be executed by the robot through the whole-body model prediction controller. The command sequence is the target unmanned aerial robotic arm end position. and the target unmanned aerial manipulator end posture The instruction sequence; S6: Develop an action block strategy network ACT based on the Transformer structure. The output and observation definitions of the strategy network ACT are as follows: in, Indicates the ACT strategy, are network parameters, Indicates the current moment, is the action block size, 、 The RGB images are from the base camera and the end effector camera, respectively, with a resolution of 640*480. ResNet-18 is used as the backbone network for feature extraction. The RGB images are encoded and input into the Transformer encoder. Indicates the end pose of the target unmanned aerial manipulator.

Images