Unmanned aerial vehicle-gondola whole body aiming control method based on deep reinforcement learning

Abstract

The application relates to the technical field of unmanned equipment motion control, and discloses a kind of unmanned aerial vehicle-pod full body aiming control method based on deep reinforcement learning, state vector is acquired in real time from the high coupling system composed of multi-rotor unmanned aerial vehicle platform, photoelectric pod and task pod, state vector is input into deep reinforcement learning strategy network, and a joint action instruction containing unmanned aerial vehicle platform expected attitude instruction and photoelectric pod and task pod expected angular velocity instruction is output, then desired attitude instruction is converted into rotor thrust constrained by maximum thrust of motor through middle controller, and desired angular velocity instruction is converted into joint torque constrained by maximum torque of joint and applied to the system. Active prediction and collaborative inhibition of coupling interference are realized, dynamic aiming accuracy and response speed are improved, and high robustness is achieved.

Description

Technical Field

[0001] This invention relates to the field of motion control technology for unmanned equipment, and in particular to a whole-body aiming control method for a drone-pod based on deep reinforcement learning. Background Technology

[0002] Currently, multi-rotor unmanned equipment systems are widely used in reconnaissance, surveillance, and special operations. In a typical UAV-pod mission system, the UAV platform carries an electro-optical pod and a mission pod. The platform's attitude stabilization and the pod's aiming and tracking are achieved separately through layered, decoupled traditional control methods (such as cascaded PID control). Specifically, the flight control subsystem is responsible for maintaining the UAV's attitude, while the pod servo subsystem independently executes target aiming commands. The two interact only through a limited information interface. This modular design reduces system development complexity to some extent and is currently the mainstream technical solution in the industry.

[0003] However, the aforementioned traditional control methods have inherent limitations when dealing with high-load, highly dynamic coupling tasks. First, when a heavy pod (such as a mission pod weighing approximately 9.6 kg) moves at large angles or rapidly, the resulting inertial reaction torque is directly transmitted to the UAV platform. Traditional flight control systems can only passively suppress this coupling interference through feedback mechanisms, leading to response delays and making it difficult to maintain the optimal performance of the overall targeting system while suppressing interference. Second, traditional controllers are designed based on the assumption of linear systems, while the thrust saturation of the UAV rotor (such as a maximum thrust of 120 N) and the torque saturation of the pod joints (such as a maximum torque of 30 N·m) exhibit significant nonlinearity. Traditional methods struggle to find the transient optimal torque distribution strategy when approaching physical limits, easily leading to overshoot and oscillations. Furthermore, existing hierarchical control can only achieve local optimization of a single objective (such as flight control pursuing the minimum attitude error and pod pursuing the minimum tracking error), and cannot dynamically and in real-time make global trade-offs between the two contradictory objectives of platform stability and aiming response speed. As a result, the system cannot simultaneously achieve both response speed (which is usually difficult to meet sub-second requirements) and dynamic aiming accuracy (usually less than 2.0 mrad) in highly dynamic missions.

[0004] Therefore, overcoming the coupling interference between the UAV platform and the pod, achieving active coordinated control of platform attitude and payload movement, and taking into account rapid response, high-precision aiming and platform stability are urgent problems to be solved in this field. Summary of the Invention

[0005] The purpose of this invention is to provide a whole-body aiming control method for UAV-pod based on deep reinforcement learning, which solves the problems in the prior art, such as response delay caused by passive suppression of coupling interference between UAV platform and pod, easy overshoot oscillation under nonlinear physical limits, and difficulty in achieving both dynamic aiming accuracy and response speed due to lack of global collaborative optimization.

[0006] To achieve the above objectives, this invention provides a deep reinforcement learning-based whole-body aiming control method for UAV-pod systems, used to control a highly coupled system consisting of a multi-rotor UAV platform, an electro-optical pod, and a mission pod, comprising the following steps: The state vector is obtained in real time from the highly coupled system. The state vector includes the attitude quaternion and angular velocity of the multi-rotor UAV platform, the joint angle and joint angular velocity of the optoelectronic pod and the mission pod, and the target aiming tilt angle error. The state vector is input into a deep reinforcement learning policy network, which outputs a joint action command. The joint action command includes the desired attitude command of the multi-rotor UAV platform and the desired angular velocity command of the optoelectronic pod and the mission pod. The desired attitude command is converted into rotor thrust by the mid-level controller, and the desired angular velocity command is converted into joint torque. The rotor thrust and the joint torque are then applied to the highly coupled system. During the conversion process, the rotor thrust is limited to not exceeding the maximum thrust of the motor, and the joint torque is limited to not exceeding the maximum joint torque.

[0007] The joint action instruction is a seven-dimensional continuous action vector, which is represented as follows: in, For the expected roll angle of the drone, For the desired pitch angle of the drone, For the desired yaw rate of the drone, For the desired pitch rate of the electro-optical pod, For the desired yaw rate of the electro-optical pod, For the desired pitch rate of the mission pod, The desired yaw rate for the mission pod.

[0008] The desired attitude command includes the desired roll angle of the UAV. Desired pitch angle of UAV and the expected yaw rate of the drone It is used to actively adjust the attitude of the multi-rotor UAV platform during the dynamic aiming process of the target, so as to counteract the coupling torque generated by the load movement.

[0009] The state vector specifically includes: the roll angle, pitch angle, yaw angle, roll rate, pitch rate, yaw rate, vertical velocity, forward horizontal velocity, and lateral velocity of the multi-rotor UAV platform; the electro-optical pitch angle, electro-optical yaw angle, electro-optical pitch rate, and electro-optical yaw rate of the electro-optical pod; the mission pitch angle, mission yaw angle, mission pitch rate, and mission yaw rate of the mission pod; the core error term; and the joint action command output at the previous moment. The core error items include UAV roll angle error, UAV pitch angle error, UAV yaw angle error, electro-optical pitch angle error, electro-optical yaw angle error, pitch coaxial alignment error between the electro-optical pod and the mission pod, and yaw coaxial alignment error between the electro-optical pod and the mission pod; the core error items are privileged information.

[0010] The highly coupled system is modeled based on physical inertial parameters and geometric positioning parameters. The physical inertial parameters include the mass, center of gravity, and inertial tensor of the multi-rotor UAV platform and the mass of the mission pod. The geometric positioning parameters include the relative positions of the optoelectronic pod joint anchor points and the mission pod joint anchor points in the coordinate system of the multi-rotor UAV platform.

[0011] The deep reinforcement learning policy network is obtained by training by maximizing a composite reward function R, which is expressed as: R=R 达成 +R 倾角 +R 稳定性 +R 惩罚 Among them, R 达成 Used to encourage rapid response, R 倾角 It is used to penalize aiming errors and achieve a dynamic trade-off between response speed and aiming accuracy.

[0012] The deep reinforcement learning policy network is obtained by training the physical parameters and actuator saturation limits of the highly coupled system with domain randomization perturbation during simulation training.

[0013] The deep reinforcement learning policy network includes an actor network and a critic network, both of which are multilayer perceptrons. The hidden layer structure of the multilayer perceptron is 512×256×128×12, the number of learning rounds is 5, the number of batches of batch data is 4, the PPO pruning factor is 0.2, the discount factor is 0.99, and the learning rate is 0.002.

[0014] This invention discloses a deep reinforcement learning-based whole-body aiming control method for a UAV-pod system. It acquires state vectors in real-time from a highly coupled system consisting of a multi-rotor UAV platform, an electro-optical pod, and a mission pod. These state vectors are input into a deep reinforcement learning policy network, which outputs a joint action command that simultaneously includes the desired attitude command of the UAV platform and the desired angular velocity commands of the electro-optical pod and mission pod. A mid-level controller then converts the desired attitude command into rotor thrust constrained by the maximum thrust of the motors and the desired angular velocity command into joint torque constrained by the maximum joint torque, applying this to the highly coupled system. This method achieves proactive prediction and collaborative suppression of coupling interference between the UAV platform and the two pods, avoiding overshoot and oscillations that are prone to occur under the physical saturation limits of traditional linear controllers. Furthermore, global optimization achieves a dynamic trade-off between fast response and high-precision aiming, enabling the system to simultaneously meet the requirements of aiming accuracy better than 2.0 mrad and sub-second response speed in dynamic aiming tasks. Attached Figure Description

[0015] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below.

[0016] Figure 1 This is a flowchart of the deep reinforcement learning process of the present invention.

[0017] Figure 2 This is a schematic diagram of the deep reinforcement learning policy network of the present invention.

[0018] Figure 3 This is a flowchart of the steps of the UAV-pod whole-body aiming control method based on deep reinforcement learning of the present invention. Detailed Implementation

[0019] The embodiments of the present invention are described in detail below. Examples of the embodiments are shown in the accompanying drawings. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain the present invention, but should not be construed as limiting the present invention.

[0020] Please refer to Figures 1 to 3 This invention provides a deep reinforcement learning-based whole-body aiming control method for a UAV-pod system, used to control a highly coupled system consisting of a multi-rotor UAV platform, an electro-optical pod, and a mission pod. The method includes the following steps: S101: Obtain the state vector from the highly coupled system in real time. The state vector includes the attitude quaternion and angular velocity of the multi-rotor UAV platform, the joint angle and joint angular velocity of the optoelectronic pod and the mission pod, and the target aiming tilt angle error. S102: The state vector is input into a deep reinforcement learning policy network, and the deep reinforcement learning policy network outputs a joint action command, which includes the desired attitude command of the multi-rotor UAV platform and the desired angular velocity command of the optoelectronic pod and the mission pod. S103: The desired attitude command is converted into rotor thrust and the desired angular velocity command is converted into joint torque by the intermediate controller, and the rotor thrust and the joint torque are applied to the highly coupled system; S104: During the conversion process, the rotor thrust is limited to not exceeding the maximum thrust of the motor, and the joint torque is limited to not exceeding the maximum joint torque.

[0021] Specifically, physical model construction and parameter settings: This invention models the multi-rotor unmanned aerial vehicle platform, electro-optical (EO) pod, and mission (WP) pod as a unified rigid body dynamics system. Specifically: (1) Integrated modeling: Modeling is performed based on the precise physical parameters of the provided multi-rotor unmanned aerial vehicle platform UAV, EO pod, and WP pod. The physical parameters include mass, density, center of gravity, and inertial tensor.

[0022] (2) Control abstraction: The thrust system of the multi-rotor UAV platform and the multi-axis joint servo system of the optoelectronic pod and mission pod are abstracted into a unified actuator system. The actuator system is coordinated and controlled by the top-level deep reinforcement learning (DRL) policy network π.

[0023] (3) Actuator limits: Clearly set the maximum rotor thrust T max =120 and the maximum torque τ of the pod joint max =30 N m serves as the hard saturation limit for the intermediate controller.

[0024] Defining a high-dimensional full-body motion space : The DRL policy network π directly outputs seven-degree-of-freedom (7-DoF) cooperative control commands, which cover platform stability and load targeting, thereby achieving full-body control.

[0025] Action Vector: The DRL policy network π outputs a seven-dimensional action vector in each control period t. , represented as: in, For the expected roll angle of the drone, For the desired pitch angle of the drone, The desired yaw rate for the drone; The desired pitch rate of the optoelectronic pod; The desired yaw rate of the electro-optical pod; The desired pitch rate for the mission pod; The desired yaw rate for the mission pod.

[0026] In the seven-dimensional continuous action vector, the first three dimensions are instructions. , , Used for platform attitude control to suppress coupled oscillations; post-fourth dimension , , , Used for payload aiming control and target tracking.

[0027] Obtain the optimal collaborative strategy through DRL training: This invention utilizes the nonlinear learning and global optimization capabilities of deep reinforcement learning to train the policy network π in the IsaacSim simulation environment: The policy network π receives a state vector in each control cycle t. The state vector Includes the roll angle α, pitch angle θ, yaw angle ψ, and roll rate of the multi-rotor unmanned aerial vehicle platform. pitch angular velocity yaw rate vertical speed of the fuselage Horizontal forward velocity Horizontal lateral speed The electro-optical pitch angle of the electro-optical pod Photoelectric yaw angle Photoelectric pitch angular velocity Photoelectric yaw rate The mission pod's pitch angle Mission yaw angle mission pitch angular velocity mission yaw rate The core error term and the combined action command output from the previous time step The core error term includes the UAV roll angle error. UAV pitch angle error UAV yaw angle error Photoelectric pitch angle error Photoelectric yaw angle error pitch coaxial alignment error between the optoelectronic pod and the mission pod Yaw coaxial alignment error between the optoelectronic pod and the mission pod 7-dimensional historical action state The 6-dimensional core error term is privileged information.

[0028] Mid-level instruction execution and coupling suppression: The middle-level controller receives the seven-dimensional continuous action vector. Then, in the high-frequency execution cycle T LLFC For the low-frequency command Linear or spline curve interpolation is performed in the time domain to generate high-frequency smoothing instructions. Real-time constraints and hard enforcement: In the case of Before being assigned to the actuator, the command is checked and constrained in real time to ensure that the command does not exceed the maximum thrust T of the motor. max and the maximum torque τ of the joint max The constrained high-frequency instructions are obtained. .

[0029] The middle-level controller will process the constrained high-frequency commands. The allocation matrix converts the commands into independent thrust commands for the UAV rotor motors and torque commands for the pod servo, which are then sent to the underlying hardware driver unit. The mid-level controller module will... The attitude command is converted into rotor thrust T motors Convert angular velocity commands into joint torque τ gimbal .

[0030] During training, the deep reinforcement learning strategy network learns to: maintain the stability of the hovering reference attitude of the multi-rotor UAV platform, suppress flight attitude deviation, suppress the coupling vibration of the UAV body caused by large-amplitude attitude rotation and swaying of the optoelectronic pod and the mission pod, maintain platform stability, and thus improve the response speed of aiming accuracy.

[0031] Training methods for DRL policy networks: The deep reinforcement learning policy network is obtained by training by maximizing a composite reward function R, which is expressed as: R=R 达成 +R 倾角 +R 稳定性 +R 惩罚 Among them, R 达成 Used to encourage rapid response, R 倾角 Used to penalize aiming errors, R 稳定性 Used to constrain platform oscillations, R 惩罚This is used to limit behavior beyond physical limits. By dynamically balancing stability and response speed in real time, the deep reinforcement learning policy network is driven to learn to achieve a response speed of less than 500 ms throughout the entire process, while ensuring a dynamic aiming accuracy better than 2.0 mrad, using the optimal platform attitude and pod motion coordination sequence.

[0032] During training, domain randomization is employed to randomly perturb the physical parameters (including mass, center of gravity, and inertia tensor) and actuator saturation limits of the highly coupled system, and to randomize actuator delay and noise, thereby enhancing the robustness of the deep reinforcement learning policy network and narrowing the gap between simulation and real-world deployment.

[0033] The deep reinforcement learning policy network comprises an actor network and a critic network, both of which are multilayer perceptrons. The hidden layer structure of the multilayer perceptron is 512×256×128×12, the number of learning epochs is 5, the batch size of the batch data is 4, the PPO pruning factor is 0.2, the discount factor is 0.99, and the learning rate is 0.002.

[0034] In one specific implementation, training and validation are performed in the Isaac Sim high-fidelity simulation environment. The highly coupled system described is built on the PhysX physics engine within the Isaac Sim environment, and the model parameters are rigorously defined using engineering parameters.

[0035] Physical model construction and parameter settings: UAV / Inertial Parameters: Multi-rotor UAV platform mass mUAV≈25.0kg; Mission pod mass mWP≈9.6kg. The model accurately imports the center of mass C and inertial tensor I of each component.

[0036] Geometric positioning parameters: Optoelectronic pod joint anchor point p EO and mission pod joint anchor point p WP Precise relative position calibration in the UAV coordinate system is used to accurately calculate the coupling arm and reaction torque.

[0037] Actuator limits: Explicitly set the maximum rotor thrust T max =120N and the maximum torque τ of the pod joint max =30N m serves as the hard saturation limit for the intermediate controller.

[0038] DRL State Observation and Action Decision-Making: State vector The DRL policy network π receives R in each control cycle t. 31 dimensional state vector Its core observation lies in: the real-time 9-dimensional status of the UAV (roll angle α, pitch angle θ, yaw angle). tumble angular velocity pitch angular velocity yaw rate vertical speed of the fuselage Horizontal forward velocity Horizontal lateral speed ), 4D status of the optoelectronic pod (optoelectronic pitch angle) Photoelectric yaw angle Photoelectric pitch angular velocity Photoelectric yaw rate ), mission pod status 4D (mission pitch angle) Mission yaw angle mission pitch angular velocity mission yaw rate ), core error term 6 dimensions (UAV roll angle error) UAV pitch angle error UAV yaw angle error Photoelectric pitch angle error Photoelectric yaw angle error Mission photoelectric pitch coaxial alignment error Mission photoelectric yaw coaxial alignment error ), historical action state 7 dimensions The 6-dimensional core error term is privileged information.

[0039] Seven-dimensional action vector DRL policy network π outputs seven-dimensional integrated action commands This vector, through time synchronization, issues platform attitude commands and load speed commands within the same time step.

[0040] Mid-level instruction execution and coupling suppression: Instruction translation: During high-frequency execution cycles T LLFC For the low-frequency command Linear or spline curve interpolation is performed in the time domain to generate high-frequency smoothing instructions. Real-time constraints and hard enforcement: In the case of Before being assigned to the actuator, the instructions are checked and constrained in real time. The high-frequency instructions after constraint The allocation matrix converts the commands into independent thrust commands for the UAV rotor motors and torque commands for the pod servo, which are then sent to the underlying hardware driver unit. The mid-level control module will... The attitude command is converted into rotor thrust T motors Convert angular velocity commands into joint torque τgimbal .

[0041] Active Coupling Suppression: Unlike traditional control, the DRL strategy in this invention learns during training to: maintain the stability of the hovering reference attitude of the hexarotor UAV, suppress flight attitude deviation, suppress coupling vibration of the UAV body caused by large-amplitude attitude rotation and sway of the electro-optical pod and weapon pod, maintain platform stability, and thus improve the response speed of aiming accuracy.

[0042] Physical constraint execution: During instruction translation, saturation constraints are strictly enforced to ensure T motors ≤T max and ≤τ max This ensures the portability of the strategy on real hardware.

[0043] This invention directly outputs a joint action command containing the desired attitude of the UAV platform and the desired angular velocity of the dual pods through a deep reinforcement learning policy network. This command is then converted into physically constrained rotor thrust and joint torque by a mid-level controller. This enables proactive prediction and collaborative suppression of coupling interference between the platform and the pods, avoiding overshoot and oscillations that are prone to occur under the thrust and torque saturation limits of traditional linear controllers. At the same time, through global optimization driven by a composite reward function, a dynamic trade-off between fast response and high-precision aiming is achieved. This allows the system to simultaneously meet the requirements of aiming accuracy better than 2.0 mrad and sub-second response speed in dynamic aiming tasks, effectively solving the problems of passive suppression of response delay, instability of nonlinear limit control, and lack of collaborative optimization in existing technologies.

[0044] The above-disclosed embodiments are merely one or more preferred embodiments of this application and should not be construed as limiting the scope of this application. Those skilled in the art can understand that all or part of the processes for implementing the above embodiments and equivalent changes made in accordance with the claims of this application still fall within the scope of this application.

Claims

1. A deep reinforcement learning-based whole-body aiming control method for UAVs and pods, used to control a highly coupled system consisting of a multi-rotor UAV platform, an optoelectronic pod, and a mission pod, characterized in that... Includes the following steps: The state vector is obtained in real time from the highly coupled system. The state vector includes the attitude quaternion and angular velocity of the multi-rotor UAV platform, the joint angle and joint angular velocity of the optoelectronic pod and the mission pod, and the target aiming tilt angle error. The state vector is input into a deep reinforcement learning policy network, which outputs a joint action command. The joint action command includes the desired attitude command of the multi-rotor UAV platform and the desired angular velocity command of the optoelectronic pod and the mission pod. The desired attitude command is converted into rotor thrust by the mid-level controller, and the desired angular velocity command is converted into joint torque. The rotor thrust and the joint torque are then applied to the highly coupled system. During the conversion process, the rotor thrust is limited to not exceeding the maximum thrust of the motor, and the joint torque is limited to not exceeding the maximum joint torque.

2. The UAV-pod whole-body aiming control method based on deep reinforcement learning as described in claim 1, characterized in that, The joint action instruction is a seven-dimensional continuous action vector, which is represented as follows: in, For the expected roll angle of the drone, For the desired pitch angle of the drone, For the desired yaw rate of the drone, For the desired pitch rate of the electro-optical pod, For the desired yaw rate of the electro-optical pod, For the desired pitch rate of the mission pod, The desired yaw rate for the mission pod.

3. The UAV-pod whole-body aiming control method based on deep reinforcement learning as described in claim 2, characterized in that, The desired attitude command includes the desired roll angle of the UAV. Desired pitch angle of UAV and the expected yaw rate of the drone It is used to actively adjust the attitude of the multi-rotor UAV platform during the dynamic aiming process of the target, so as to counteract the coupling torque generated by the load movement.

4. The UAV-pod whole-body aiming control method based on deep reinforcement learning as described in claim 1, characterized in that, The state vector specifically includes: the roll angle, pitch angle, yaw angle, roll rate, pitch rate, yaw rate, vertical velocity, forward horizontal velocity, and lateral velocity of the multi-rotor UAV platform; the electro-optical pitch angle, electro-optical yaw angle, electro-optical pitch rate, and electro-optical yaw rate of the electro-optical pod; the mission pitch angle, mission yaw angle, mission pitch rate, and mission yaw rate of the mission pod; the core error term; and the joint action command output at the previous moment. The core error items include UAV roll angle error, UAV pitch angle error, UAV yaw angle error, electro-optical pitch angle error, electro-optical yaw angle error, pitch coaxial alignment error between the electro-optical pod and the mission pod, and yaw coaxial alignment error between the electro-optical pod and the mission pod; the core error items are privileged information.

5. The UAV-pod whole-body aiming control method based on deep reinforcement learning as described in claim 1, characterized in that, The highly coupled system is modeled based on physical inertial parameters and geometric positioning parameters. The physical inertial parameters include the mass, center of gravity, and inertial tensor of the multi-rotor UAV platform and the mass of the mission pod. The geometric positioning parameters include the relative positions of the optoelectronic pod joint anchor points and the mission pod joint anchor points in the coordinate system of the multi-rotor UAV platform.

6. The UAV-pod whole-body aiming control method based on deep reinforcement learning as described in claim 1, characterized in that, The deep reinforcement learning policy network is obtained by training by maximizing a composite reward function R, which is expressed as: R=R 达成 +R 倾角 +R 稳定性 +R 惩罚 Among them, R 达成 Used to encourage rapid response, R 倾角 It is used to penalize aiming errors and achieve a dynamic trade-off between response speed and aiming accuracy.

7. The UAV-pod whole-body aiming control method based on deep reinforcement learning as described in claim 1, characterized in that, The deep reinforcement learning policy network is obtained by training the physical parameters and actuator saturation limits of the highly coupled system with domain randomization perturbation during simulation training.

8. The UAV-pod whole-body aiming control method based on deep reinforcement learning as described in claim 1, characterized in that, The deep reinforcement learning policy network includes an actor network and a critic network, both of which are multilayer perceptrons. The hidden layer structure of the multilayer perceptron is 512×256×128×12, the number of learning rounds is 5, the number of batches of batch data is 4, the PPO pruning factor is 0.2, the discount factor is 0.99, and the learning rate is 0.002.

Images