A multi-rotor unmanned aerial vehicle interception control system based on reinforcement learning

Abstract

The application relates to the technical field of unmanned aerial vehicles, in particular to a multi-rotor unmanned aerial vehicle interception control system based on reinforcement learning. The system comprises an image acquisition module, which is used for acquiring a target image of a target unmanned aerial vehicle; an IBVS controller, which is used for extracting a target position in the target image, calculating an image error, constructing and training a reinforcement learning model, constructing a current state based on the image error and state information of the multi-rotor unmanned aerial vehicle as input of the reinforcement learning model, acquiring an expected attitude output by the reinforcement learning model, calculating an error between a current attitude and the expected attitude to generate a control instruction for intercepting the target unmanned aerial vehicle; and a flight controller, which is used for receiving the control instruction, converting the control instruction into a pulse width modulation signal, outputting the pulse width modulation signal to a motor to adjust the rotating speed of each motor, and executing an interception task. Through the application, the target movement in a complex environment can be dynamically adapted, and real-time and efficient interception of the target unmanned aerial vehicle can be realized.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of unmanned aerial vehicles, in particular to a multi-rotor unmanned aerial vehicle interception control system based on reinforcement learning. BACKGROUND

[0002] With the opening of low-altitude airspace and the development of small unmanned aerial vehicle technology, non-cooperative targets (such as unauthorized unmanned aerial vehicles) appear more and more frequently in important airspace, posing a serious threat to flight safety and public order.

[0003] To deal with such threats, existing countermeasures mainly include radio frequency interference, high-energy weapon strikes, and physical capture technologies. Radio frequency interference technology can make the target unmanned aerial vehicle lose control or return by interfering with its communication link or satellite navigation signal, and has strong interference effect. High-energy weapons, such as lasers and microwaves, can directly destroy the target, respond quickly and have high hit rates. However, these methods generally have the risk of environmental pollution, strong electromagnetic interference, and collateral damage to surrounding personnel or equipment. In contrast, the physical capture method uses methods such as launching a net gun and laying a capture device to intercept the target, which is pollution-free and more environmentally friendly. However, this method has low interception efficiency and limited success rate when facing high-speed or maneuverable targets, especially in dynamic complex environments.

[0004] In recent years, the use of small unmanned aerial vehicles equipped with visual sensors as an interception platform has gradually become a research hotspot. Such systems have the advantages of quick response, low cost, and high safety, and especially camera-based visual systems have the characteristics of lightweight and multifunctionality, showing broad application prospects in non-cooperative target identification, tracking, and interception.

[0005] In existing technologies, some papers have proposed a method combining image-based visual servoing (IBVS) algorithm and proportional navigation guidance law (PNG). In their experiments, the circular error probable (CEP) can reach 0.089 meters, showing good interception accuracy. However, such control algorithms are often designed based on fixed parameters, lacking the ability to adapt to target maneuverability and environmental changes, leading to problems such as error accumulation and response lag in complex dynamic environments, which seriously affect the interception efficiency and system stability.

[0006] Therefore, how to improve the real-time performance and robustness of such systems in dealing with variable and maneuverable targets remains a key technical problem to be solved. SUMMARY

[0007] The present application provides a multi-rotor drone interception control system based on reinforcement learning, including:

[0008] An image acquisition module is used to obtain a target image of a target UAV;

[0009] The IBVS controller is used to extract the target position in the target image, calculate the image error, build and train a reinforcement learning model, construct the current state as the input of the reinforcement learning model based on the image error and the state information of the multi-rotor drone, obtain the desired posture output by the reinforcement learning model, calculate the error between the current posture and the desired posture, and generate control instructions for intercepting the trajectory of the target drone;

[0010] The flight controller is used to receive the control instructions, convert them into pulse width modulation signals, and output them to the motors to adjust the speed of each motor and execute the interception mission, ensuring that the UAV can execute the interception mission according to the control strategy given by the reinforcement learning model;

[0011] The IBVS controller is configured as follows:

[0012] The reinforcement learning model includes an actor network and a critic network. The actor network is used to generate actions, and the critic network is used to evaluate the actions generated by the actor network. The reinforcement learning model uses a set reward function to learn an optimization strategy. The reward function is used to encourage the multi-rotor drone to achieve the desired acceleration, track the desired posture and keep the target drone within the field of view, and punish the maximum acceleration and lift actions that approach the physical limits of the multi-rotor drone, thereby avoiding unstable or dangerous states during flight.

[0013] By combining image errors and drone state information, the above-described embodiment utilizes a reinforcement learning model to optimize interception strategies. This allows for dynamic adaptation to target movement in complex environments, enabling real-time, efficient interception of target drones. Flexible selection of image acquisition modules (monocular or binocular) further enhances control precision. During flight, the reinforcement learning model's reward function adjusts the drone's acceleration and attitude in real time to avoid unstable or dangerous flight conditions, thereby improving the success rate and safety of interception missions.

[0014] In some embodiments, the reward function is represented by the following computational model:

[0015]

[0016] in, 、 、 、 、 are weight coefficients, is the actual acceleration of the multi-rotor drone in the earth coordinate system and expected acceleration The square of the Euclidean distance reflects the motion deviation to make the actual acceleration close to the expected acceleration. is the trace of the matrix, is the transformation matrix from the body coordinate system to the earth coordinate system, which is used to represent the actual posture of the multi-rotor drone. It is used to indicate the desired attitude of the multi-rotor UAV. The desired attitude is used to ensure that the lift direction of the multi-rotor UAV is aligned with the desired acceleration, and target tracking is achieved by adjusting the pitch angle and yaw angle. The horizontal pixel error and vertical pixel error of the target drone at the center point of the target image are used to keep the target within the field of view of the multi-rotor drone by reducing the image error. is the normalized desired angular velocity, Depend on The three are the components of the desired angular velocity in each axis of the body coordinate system, is the normalized expected lift, which is used to indicate the magnitude of the action and encourage safe control.

[0017] Based on this reward function, by quantifying the acceleration error and attitude error, the guidance system can achieve more accurate acceleration tracking and attitude control. The introduction of image error terms improves the real-time and reliability of target tracking. The introduction of angular velocity and lift limit terms effectively reduces the risk of aircraft loss of control and collision, and enhances the system's adaptability to complex environments.

[0018] In some embodiments, the following constraints are used to prevent the target drone from exceeding the field of view due to pitching or altitude changes: ,in, The vertical movement range of the target drone in the target image during the process of the multi-rotor drone intercepting the target drone, that is, the difference between the maximum and minimum vertical pixel errors, is a preconfigured vertical motion threshold whose value is a small constant.

[0019] In some embodiments, the desired posture Represented by a rotation matrix, only the pitch and yaw angles are involved, and the roll angle is ignored to simplify control. The tilt rotation matrix is ​​used to represent the pitch and yaw angles required to track the target drone.

[0020] In some embodiments, building and training a reinforcement learning model further comprises:

[0021] Initialize the actor network and critic network as well as the target actor network and target critic network, initialize the experience replay buffer, and initialize the target actor network and target critic network as copies of the actor network and critic network for stable training;

[0022] Actor Networks Based on State Generating actions through deterministic policies and add noise , output action , ,Achieve motion exploration by adding noise;

[0023] Define the environment, the environment receives the action And update the output next state 、 award , recycling experience data To the experience replay buffer, used to update the actor network and critic network;

[0024] Randomly sample mini-batches of data from the experience replay buffer,

[0025] Calculate the target Q value based on the mini-batch data and the critic network to estimate the expected reward of the state-action pair;

[0026] Update the critic network by minimizing the loss function;

[0027] Update the parameters of the actor network based on the policy gradient of the critic network;

[0028] Update the parameters of the target actor network and the target critic network according to the soft update formula;

[0029] Repeat the above steps until you reach your learning goal.

[0030] By introducing an actor-critic architecture with a target network, a soft update mechanism, experience replay, and policy noise design, the training process of this application achieves excellent convergence and stability. Experience replay effectively breaks the temporal correlation between samples, improving training efficiency; the target network and soft update mechanism significantly reduce policy oscillation and enhance training robustness; the overall training architecture enables accurate policy learning in complex, continuous state and action spaces, thus enabling multi-rotor drones to possess real-time, intelligent interception capabilities in dynamic environments, significantly improving the system's autonomy and execution efficiency.

[0031] In some embodiments, building and training a reinforcement learning model further comprises:

[0032] Define the state of the multirotor drone ,action ;

[0033] Wherein, the state is defined as:

[0034] ,

[0035] in, The pitch angle, roll angle and yaw angle of the multi-rotor UAV in the body coordinate system; is the angular velocity vector of the multi-rotor UAV in the body coordinate system; is the expected acceleration component of the multi-rotor UAV in the body coordinate system, is the velocity component of the UAV in the Earth coordinate system;

[0036] The action is defined as: They are the coordinates of the multi-rotor UAV along the body coordinate system. Axis direction, Axis direction, The desired angular velocity along the axis, is the expected lift.

[0037] In some embodiments, the environment is configured as a dynamic model of a multi-rotor drone:

[0038] ,

[0039] in, Used to represent the speed of the drone in the earth coordinate system is equal to its velocity vector in the Earth coordinate system; is the actual acceleration of the multi-rotor UAV in the earth coordinate system, is the position vector of the UAV in the Earth coordinate system. The speed can be calculated based on its position change. ; is the attitude update equation, also known as the attitude change rate, which is used to reflect that the attitude change of the drone over time is determined by its angular velocity; is the angular velocity vector of the UAV in the body coordinate system, expressed as , used to describe the rotation rate of the drone, for The antisymmetric matrix of ; is the inertia matrix of the multi-rotor drone, which describes the inertial characteristics of the drone rotating around each axis, usually a diagonal matrix; is the angular acceleration vector of the UAV in the body coordinate system; is the Coriolis moment caused by angular velocity, reflecting the inertial effect inside the rotating object; is the gyroscopic torque, i.e. the additional torque generated by the rotation of the rotor; is the aerodynamic torque caused by rotor steering and air resistance, and is usually part of the control input, where:

[0040] .

[0041] In some embodiments, the actor network adopts a fully connected neural network, including 3 hidden layers, each hidden layer includes 256 neurons, and the activation function adopts ReLU.

[0042] In some embodiments, the output layer of the actor network uses a tanh activation function and performs action scaling. Specifically:

[0043] ;

[0044] .

[0045] In some embodiments, after the action is scaled, a clipping function is used Tailor the action so that:

[0046] ,

[0047] , used to limit the control amount to avoid overload,

[0048] The trimming function is expressed as: ,

[0049] but , , is the maximum angular velocity of the multi-rotor drone, is the maximum lift of the drone.

[0050] In some embodiments, the critic network adopts a fully connected neural network, including 3 hidden layers, each hidden layer includes 256 neurons, and the activation function adopts ReLU, wherein the input state and action are merged after the first hidden layer, and the output layer is set to have no activation function.

[0051] Compared with related technologies, the multi-rotor UAV interception and control system provided in the embodiment of the present application, by combining image errors and UAV status information, uses a reinforcement learning model to optimize the interception strategy, can dynamically adapt to target movement in complex environments, and achieve real-time and efficient interception of target UAVs; during flight, the reward function of the reinforcement learning model can adjust the acceleration and attitude of the UAV in real time to avoid unstable or dangerous flight states, thereby improving the success rate and safety of the interception mission.

[0052] The details of one or more embodiments of the present application are set forth in the following drawings and description to make other features, objects, and advantages of the present application more readily apparent. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0054] Figure 1 is a schematic diagram of a coordinate system for a quadrotor aircraft according to related art;

[0055] Figure 2 It is a structural block diagram of a multi-rotor UAV interception control system according to an embodiment of the present application. DETAILED DESCRIPTION

[0056] In order to make the purpose, technical solutions and advantages of this application more clearly understood, the present application is described and illustrated below in conjunction with the accompanying drawings and examples. It should be understood that the specific embodiments described herein are merely used to explain this application and are not intended to limit this application. Based on the embodiments provided in this application, all other embodiments obtained by those of ordinary skill in the art without making any creative efforts are within the scope of protection of this application.

[0057] Obviously, the drawings described below are merely examples or embodiments of the present application. Those skilled in the art can, without inventive effort, apply the present application to other similar scenarios based on these drawings. Furthermore, it is also understood that, although the effort involved in such a development process may be complex and lengthy, for those skilled in the art related to the content disclosed in this application, changes in design, manufacturing, or production based on the technical content disclosed in this application are merely conventional technical means and should not be construed as an insufficiency of the content disclosed in this application.

[0058] References to "embodiments" in this application mean that a particular feature, structure, or characteristic described in connection with the embodiment may be included in at least one embodiment of the application. The appearance of this phrase in various places in the specification does not necessarily refer to the same embodiment, nor does it refer to independent or alternative embodiments that are mutually exclusive of other embodiments. It is understood, both explicitly and implicitly, by those skilled in the art that the embodiments described in this application may be combined with other embodiments unless there is a conflict.

[0059] A quadrotor unmanned helicopter is an underactuated dynamic rotorcraft with four input forces and six coordinate outputs. This system is therefore an autonomous aircraft capable of quasi-static flight (hovering and close-range hovering). The equations of motion for any system are established with respect to a specific reference coordinate system. Drones are essentially multibody dynamic systems. The motion of a drone can be viewed as a rigid body with six degrees of freedom, including rotation about three axes and linear motion of the center of gravity along three axes.

[0060] The Earth coordinate system is an inertial coordinate system that is fixed to the ground and uses the Earth as a reference. It is commonly used to describe the position, velocity, and acceleration of multi-rotor drones and targets in the global environment. The Earth coordinate system is represented as: . The axis usually points vertically upward, opposite to the direction of gravity; and The axis is in the horizontal plane and its direction is defined by the specific application, for example Figure 1 shown.

[0061] The body coordinate system is fixed on the body of the multi-rotor drone, with the origin of the coordinate system located at the center of mass of the drone, and is used to describe the drone's posture ( ) and the position of sensors on the body (such as cameras) relative to the body. The body coordinate system is expressed as: 。 The axis is usually along the front of the body, The axis is along the side of the body, The axis is vertical to the body (usually upward), for example Figure 1 shown.

[0062] The camera coordinate system is fixed to the monocular camera on the multirotor drone, with the origin of the coordinate system located at the optical center of the camera. It is used to describe the three-dimensional position of the target in the camera's field of view and the relative posture of the camera relative to the drone. The camera coordinate system is expressed as: 。 The axis usually points vertically upward, opposite to the direction of gravity; and The axes lie in the horizontal plane and their directions are defined by the application.

[0063] This application is based on the principle of proportional navigation to control multi-rotor drones. The basic idea is to make the speed and direction of the drone change at a rate of Rate of change of sight direction Proportional:

[0064]

[0065] ：The velocity direction angle of the drone indicates the direction of the velocity vector in space.

[0066] : Line of sight angle, indicating the line of sight direction from the drone to the target.

[0067] : Proportional navigation constant, controls the sensitivity of navigation.

[0068] When the three-dimensional space is decomposed into two-dimensional horizontal and vertical planes, the changes in the line of sight direction and velocity direction can be processed separately in the horizontal and vertical planes. Assume:

[0069] The sight angle in the vertical plane is: ， ,

[0070] Sight angle in the horizontal plane:

[0071] ， ,

[0072] in, is the sight direction vector,

[0073] Velocity angle in the vertical plane:

[0074] ，

[0075] Velocity angle in the horizontal plane:

[0076] ，

[0077] in is the velocity direction vector of the UAV.

[0078] The present invention provides a multi-rotor UAV interception control system based on reinforcement learning. Figure 2 As shown, the system includes an electrically connected image acquisition module, an IBVS controller, and a flight controller, and is specifically configured as follows:

[0079] The image acquisition module is used to acquire a target image of the target drone. Optionally, the image acquisition module uses a monocular camera, such as a Jetson CSI camera. In another embodiment, the image acquisition module uses a binocular camera to provide richer depth information, thereby enhancing the accuracy of interception control;

[0080] The IBVS controller is used to extract the target position in the target image, calculate the image error, build and train a reinforcement learning model, construct the current state based on the image error and the state information of the multi-rotor drone as the input of the reinforcement learning model, obtain the desired posture output by the reinforcement learning model, calculate the error between the current posture and the desired posture, and generate control instructions for intercepting the target drone's trajectory; the reinforcement learning model generates the desired angular velocity and desired lift as the goals of the drone's flight control based on the drone's state and image error. The IBVS controller calculates the error between these values ​​and the drone's current posture, such as pitch angle error, yaw angle error, roll angle error, etc., and generates control instructions for adjusting the angular velocity (pitch angle, yaw angle, roll angle), which are used to adjust the drone's posture and trajectory to achieve the interception mission.

[0081] The flight controller is used to receive the control instructions, convert them into pulse width modulation signals, and output them to the motors to adjust the speed of each motor, execute the interception mission, and ensure that the UAV can execute the interception mission according to the control strategy given by the reinforcement learning model; optionally, the flight controller adopts Pixhawk, which can accurately control the speed of the motors and ensure that the UAV can stably execute the desired flight path;

[0082] The IBVS controller is configured as follows:

[0083] The reinforcement learning model includes an actor network and a critic network. The actor network is used to generate actions, and the critic network is used to evaluate the actions generated by the actor network. The reinforcement learning model uses a set reward function to learn an optimization strategy. The reward function is used to encourage the multi-rotor drone to achieve the desired acceleration, track the desired posture and keep the target drone within the field of view, and punish the maximum acceleration and lift actions that approach the physical limits of the multi-rotor drone, thereby avoiding unstable or dangerous states during flight.

[0084] By combining image errors and drone state information, the above-described embodiment utilizes a reinforcement learning model to optimize interception strategies. This allows for dynamic adaptation to target movement in complex environments, enabling real-time, efficient interception of target drones. Flexible selection of image acquisition modules (monocular or binocular) further enhances control precision. During flight, the reinforcement learning model's reward function adjusts the drone's acceleration and attitude in real time to avoid unstable or dangerous flight conditions, thereby improving the success rate and safety of interception missions.

[0085] In some embodiments, the reward function is represented by the following computational model:

[0086]

[0087] in, 、 、 、 、 are weight coefficients, is the actual acceleration of the multi-rotor drone in the earth coordinate system and expected acceleration The square of the Euclidean distance reflects the motion deviation to make the actual acceleration close to the expected acceleration. is the trace of the matrix, is the transformation matrix from the body coordinate system to the earth coordinate system, which is used to represent the actual posture of the multi-rotor drone. It is used to indicate the desired attitude of the multi-rotor UAV. The desired attitude is used to ensure that the lift direction of the multi-rotor UAV is aligned with the desired acceleration, and to achieve target tracking through pitch and yaw adjustments. The horizontal pixel error and vertical pixel error of the target drone at the center point of the target image are used to keep the target within the field of view of the multi-rotor drone by reducing the image error. is the normalized desired angular velocity, From the expected angular velocity The three are the components of the desired angular velocity in each axis of the body coordinate system. is the normalized expected lift, which is used to indicate the magnitude of the action and encourage safe control.

[0088] Based on this reward function, by quantifying the acceleration error and attitude error, the guidance system can achieve more accurate acceleration tracking and attitude control. The introduction of image error terms improves the real-time and reliability of target tracking. The introduction of angular velocity and lift limit terms effectively reduces the risk of aircraft loss of control and collision, and enhances the system's adaptability to complex environments.

[0089] In the above embodiment, the actual acceleration of the multi-rotor drone is determined by the acceleration of gravity and the lift generated by the rotor. The lift is controllable and is used to adjust the trajectory of the drone. , is the gravitational acceleration vector, usually expressed as ,in , the direction is vertically downward, For the quality of multi-rotor drones, The lift vector generated by the rotor of the multi-rotor drone in the earth coordinate system is usually in the same direction as the body coordinate system. The axis is opposite, , is the expected speed, indicating that the multi-rotor drone will move from the current speed to the desired speed Applied acceleration.

[0090] In the above embodiment, the desired velocity vector By speed amplitude and the normalized direction vector Composition, used to drive the multi-rotor drone to fly along the proportional navigation trajectory, speed amplitude Dynamically adjust according to target distance and drone performance, Defines the flight direction of the drone:

[0091] , , is the desired vertical plane velocity angle and horizontal plane velocity angle, through the current moment and the previous moment The change in sight angle is calculated as follows: to ensure that the drone gradually approaches the target.

[0092] In the above embodiment, the camera uses a pinhole model to project the target in the three-dimensional space onto the two-dimensional image plane: , , , ( is the pixel coordinate of the target UAV in the target image, and its origin Set to the upper left corner of the target image, ( is the center pixel coordinate of the target image, the image error can be expressed as: .

[0093] In some embodiments, to facilitate the multi-rotor drone to track the target through yaw angle adjustment and to prevent the target drone from moving significantly in the vertical direction of the target image, the following constraints are used to prevent the target drone from exceeding the field of view (FOV) due to pitch movement or altitude change: ,in, The vertical movement range of the target drone in the target image during the process of the multi-rotor drone intercepting the target drone, that is, the difference between the maximum and minimum vertical pixel errors, is a preconfigured vertical motion threshold whose value is a small constant.

[0094] In the Earth coordinate system, use the sight direction vector Defined as the relative position vector of the multirotor drone pointing to the target aircraft for interception The normalization of , which is calculated from the perspective of relative position, can also be specifically expressed as: , the sight direction can be calculated by the target UAV position and coordinate transformation in the target image, which can be expressed as follows:

[0095] , is the transformation matrix from the body coordinate system to the camera coordinate system.

[0096] The rate of change of the above image error and the instantaneous motion vector of the camera in the camera coordinate system Related, through the Jacobian matrix describe: , instantaneous motion vector : , including line speed and angular velocity ;

[0097] Jacobian matrix The six-degree-of-freedom motion of the camera is mapped to the rate of change of the image error, thereby facilitating the connection from image feedback to motion control. The Jacobian matrix Expressed as:

[0098] ;

[0099] in, is the depth of the target drone along the optical axis in the camera coordinate system; is the normalized image error, is the camera focal length, .

[0100] In some embodiments, the desired posture Represented by a rotation matrix, only the pitch and yaw angles are involved, and the roll angle is ignored to simplify control. is used to represent the tilt rotation matrix constructed based on the pitch and yaw angles required to track the target drone. ,in:

[0101] is the identity matrix;

[0102] is the current lift direction and the desired lift direction The cross product of represents the axis of rotation;

[0103] : Current lift direction and the desired lift direction The angle between represents the rotation angle; the desired lift direction The coordinate system of the multi-rotor UAV body The desired orientation of the axis, used to constrain the current lift direction and the desired lift direction Alignment, desired lift direction The net acceleration after gravity compensation is expressed as: ;

[0104] : Rotation axis The antisymmetric matrix of is used to calculate the rotation matrix;

[0105] , , , It is the body coordinate system The rotation matrix about the axis.

[0106] In the above embodiment, the present application configures an angular velocity controller to adjust the pitch angle and yaw angle so that the actual attitude Quickly approach the desired pose , and combined with the need to maintain the field of view, set the angular velocity controllers for pitch and yaw angles to:

[0107] ,

[0108] Among them, the vex() function is used to extract the corresponding three-dimensional vector from a three-dimensional antisymmetric matrix.

[0109] Furthermore, the input of a field of view maintenance controller is configured in combination with the above angular velocity controller:

[0110] ,

[0111] in, The yaw rate generated by the controller for the field of view maintenance, is a saturation function, which is defined as:

[0112] .

[0113] In order to ensure the stability of posture control, this application is based on the above Lyapunov candidate function:

[0114] ,

[0115] in, is the Frobenius norm.

[0116] In some embodiments, building and training a reinforcement learning model further comprises:

[0117] Step S1: Define the state of the multi-rotor drone ,action , define the environment;

[0118] Step S2: Initialize the actor network, critic network, target actor network, and target critic network. Specifically: initialize the actor network parameters and the parameters of the critic network and the parameters of the target actor network and target critic network 、 , initialize the experience replay buffer, the target actor network and the target critic network are initialized as copies of the actor network and the critic network for stable training;

[0119] Step S3: Current state of actor network By using a deterministic strategy, the state Generate Action and add noise , output action , ,Achieve motion exploration by adding noise, optional,noise Ornstein-Uhlenbeck noise satisfy: ; is the regression rate, is the mean (usually 0), is the volatility, It's Brownian motion.

[0120] Step S4: Environment receiving action And update the output next state 、 award , recycling experience data To the experience replay buffer, used to update the actor network and the critic network;

[0121] Step S5: Randomly sample small batches of data from the experience replay buffer,

[0122] Step S6: Calculate the target Q value based on the mini-batch data and the critic network to estimate the expected return of the state-action pair. The target Q value is expressed as: , i is the sample index;

[0123] Step S7: Update the critic network by minimizing the loss function, which is expressed as:

[0124] ;

[0125] Step S8: Update the parameters of the actor network based on the policy gradient of the critic network. The policy gradient adopts the gradient ascent method, which is expressed as: ;

[0126] Step S9: Update the parameters of the target actor network and the target critic network according to the soft update formula, which is expressed as:

[0127]

[0128]

[0129] in, is the soft update rate, which is 0.001;

[0130] Step S10: Repeat steps S3 to S9 until the learning goal is achieved;

[0131] Wherein, the state is defined as:

[0132]

[0133] ,in, The pitch angle, roll angle and yaw angle of the multi-rotor UAV in the body coordinate system; is the angular velocity vector of the multi-rotor UAV in the body coordinate system; is the expected acceleration component of the multi-rotor UAV in the body coordinate system, is the velocity component of the UAV in the Earth coordinate system;

[0134] The action is defined as: They are the coordinates of the multi-rotor UAV along the body coordinate system. Axis direction, Axis direction, The desired angular velocity in the axis direction, namely the roll angle, pitch angle, and yaw angle, The expected lift is used to ensure that the multirotor drone generates sufficient thrust to achieve the desired acceleration while being limited within the hardware capabilities of the multirotor drone. It can be calculated based on the following formula:

[0135] , is the expected lift force considering gravity compensation.

[0136] In the above embodiment, the environment is configured as a dynamic model of a multi-rotor drone:

[0137] ,

[0138] in, Used to represent the speed of the drone in the earth coordinate system is equal to its velocity vector in the Earth coordinate system; is the actual acceleration of the multi-rotor UAV in the earth coordinate system, is the position vector of the UAV in the Earth coordinate system. The speed can be calculated based on its position change. ; is the attitude update equation, also known as the attitude change rate, which is used to reflect that the attitude change of the drone over time is determined by its angular velocity; is the angular velocity vector of the UAV in the body coordinate system, expressed as , used to describe the rotation rate of the drone, is the antisymmetric matrix of the angular velocity vector, is the inertia matrix of the multi-rotor drone, which describes the inertial characteristics of the drone rotating around each axis, usually a diagonal matrix; is the angular acceleration vector of the UAV in the body coordinate system; is the Coriolis moment caused by angular velocity, reflecting the inertial effect inside the rotating object; is the gyroscopic torque, i.e. the additional torque generated by the rotation of the rotor; is the aerodynamic torque, caused by rotor steering and air resistance, and is usually part of the control input; .

[0139] In the above embodiment, is the angular velocity update equation, based on the Euler dynamics equation Construction,Considering that the external torque (such as the control torque generated by the rotor) and the internal inertia effect jointly determine the rotation behavior of the UAV, the angular velocity update equation of this embodiment is determined by the joint action of multiple torques.

[0140] Based on the above configuration, the state space of the embodiment of the present application fully characterizes the core information of the multi-rotor drone at any moment, such as flight attitude, expected acceleration, visual error and geographic speed. By capturing the pitch angle, roll angle, yaw angle and the corresponding angular velocity, the system can reflect the current attitude and rotation trend. Expected acceleration Reflects the mission objective's requirements for mobility, image error Ensure that the target is always in the center of the field of view, and the velocity vector Can be used for state dynamic modeling and navigation estimation. Action space The desired angular velocity controls attitude adjustments in the three rotational degrees of freedom, while the desired lift determines vertical thrust output. The desired lift calculation process uses upper and lower limits to prevent lift commands from exceeding the actual flight control capabilities. This ensures both the feasibility of flight maneuvers and the safety and stability of the flight process. This configuration provides a clear input and output interface for the reinforcement learning strategy, facilitating the establishment of a stable and convergent training process.

[0141] The environment of this application constructs a dynamic model of a multi-rotor drone based on the drone's position update (i.e., speed), velocity update (i.e., acceleration), attitude update, and angular velocity update equations. It can accurately simulate the complex dynamic responses of a multi-rotor drone during flight and is the key foundation for interacting with the environment, generating state transitions, calculating reward functions, and predicting effects in reinforcement learning algorithm training.

[0142] The actor network generates flight maneuvers, while the critic network evaluates the value (i.e., Q-value) of actions in a given state. Together, they form a policy optimization mechanism. The experience replay mechanism records state transitions and reward information during environmental interactions, reducing correlation between training samples and improving learning stability and efficiency. The introduced Ornstein-Uhlenbeck noise exhibits temporal correlation, enhancing policy exploration in continuous action spaces and preventing regression into local optima. In each training round, the model updates the critic network to fit the target Q-value by sampling empirical data in batches, then improves the quality of the actor network's actions using policy gradients. The soft update mechanism of the target network provides a smooth transition between policy updates, avoiding training instability caused by frequent changes in the target network. This mechanism completes the training loop, from state-action mapping to action value assessment to policy adjustment.

[0143] By introducing the actor-critic structure and target network, soft update mechanism, experience replay and policy noise design, the training process of this application has good convergence and stability. Experience replay effectively breaks the temporal correlation between samples and improves training efficiency; the target network and soft update mechanism significantly reduce the policy oscillation problem and improve training robustness; Ornstein-Uhlenbeck noise realizes effective action space exploration, improves policy search capabilities, and helps the model obtain better flight control strategies. The overall training architecture can achieve accurate policy learning in complex and continuous state space and action space, so that multi-rotor drones have real-time and intelligent interception capabilities in dynamic environments, significantly improving the autonomy and execution efficiency of the system.

[0144] In some embodiments, the actor network adopts a fully connected neural network, including 3 hidden layers, each hidden layer includes 256 neurons, and the activation function adopts ReLU.

[0145] In some embodiments, the output layer of the actor network uses a tanh activation function and performs action scaling. Specifically:

[0146] ;

[0147] .

[0148] In some embodiments, after the action is scaled, a clipping function is used Tailor the action so that:

[0149] ,

[0150] , used to limit the control amount to avoid overload,

[0151] The trimming function is expressed as: ,

[0152] but , , is the maximum angular velocity of the multi-rotor drone, is the maximum lift of the drone.

[0153] In some embodiments, the critic network adopts a fully connected neural network, including 3 hidden layers, each hidden layer includes 256 neurons, and the activation function adopts ReLU, wherein the input state and action are merged after the first hidden layer, and the output layer is set to have no activation function.

[0154] It should be noted that the above modules can be functional modules or program modules, and can be implemented through software or hardware. For modules implemented through hardware, the above modules can be located in the same processor; or the above modules can be located in different processors in any combination.

[0155] The above-described embodiments merely represent several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present invention. It should be noted that a person skilled in the art could make various modifications and improvements without departing from the spirit of the present application, all of which fall within the scope of protection of the present application. Therefore, the scope of protection of the present patent application shall be determined by the appended claims.

Claims

1. A multi-rotor UAV interception control system based on reinforcement learning, characterized in that: include: An image acquisition module is used to obtain a target image of a target UAV; The IBVS controller is used to extract the target position in the target image, calculate the image error, build and train a reinforcement learning model, construct the current state as the input of the reinforcement learning model based on the image error and the state information of the multi-rotor drone, obtain the desired posture output by the reinforcement learning model, calculate the error between the current posture and the desired posture, and generate control instructions for intercepting the trajectory of the target drone; A flight controller is used to receive the control instructions, convert them into pulse width modulation signals and output them to the motors to adjust the speed of each motor and perform the interception mission; The IBVS controller is configured as follows: The reinforcement learning model includes an actor network and a critic network. The actor network is used to generate actions, and the critic network is used to evaluate the actions generated by the actor network. The reinforcement learning model uses a set reward function to learn an optimization strategy. The reward function is used to encourage the multi-rotor drone to achieve the desired acceleration, track the desired posture, and keep the target drone within the field of view, and to punish the maximum acceleration and lift actions that approach the physical limits of the multi-rotor drone. The reward function is expressed as the following computational model: in, 、 、 、 、 are weight coefficients, is the actual acceleration of the multi-rotor drone in the earth coordinate system and expected acceleration The square of the Euclidean distance, , is the expected speed, indicating that the multi-rotor drone will move from the current speed to the desired speed The applied acceleration, is the trace of the matrix, is the transformation matrix from the body coordinate system to the earth coordinate system, which is used to represent the actual posture of the multi-rotor drone. Used to indicate the desired attitude of a multirotor drone. are the horizontal pixel error and vertical pixel error of the target UAV at the center point of the target image, is the normalized desired angular velocity, is the normalized expected lift.

2. The multi-rotor UAV interception control system based on reinforcement learning according to claim 1 is characterized in that: Use the following constraints to prevent the target drone from exceeding the field of view due to pitching or altitude changes: ， in, The vertical movement range of the target drone in the target image during the process of the multi-rotor drone intercepting the target drone, that is, the difference between the maximum and minimum vertical pixel errors, is a pre-configured vertical motion threshold.

3. The multi-rotor UAV interception control system based on reinforcement learning according to claim 2 is characterized in that: Expected Posture Represented by the rotation matrix, The tilt rotation matrix is ​​used to represent the pitch and yaw angles required to track the target drone.

4. The multi-rotor UAV interception control system based on reinforcement learning according to any one of claims 1 to 3, characterized in that: Building and training a reinforcement learning model further includes: Initialize the actor network, critic network, target actor network, target critic network, and experience replay buffer; Actor Networks Based on State Generating actions through deterministic policies and add noise , output action , ; Define the environment, the environment receives the action And update the output next state 、 award , recycling experience data To the experience replay buffer; Randomly sample mini-batches of data from the experience replay buffer, Calculate the target Q value based on the mini-batch data and the critic network; Update the critic network by minimizing the loss function; Update the parameters of the actor network based on the policy gradient of the critic network; Update the parameters of the target actor network and the target critic network according to the soft update formula; Repeat the above steps until you reach your learning goal.

5. The multi-rotor UAV interception control system based on reinforcement learning according to claim 4 is characterized in that: Building and training a reinforcement learning model further includes: Define the state of the multirotor drone ,action ; Wherein, the state is defined as: ， in, The pitch angle, roll angle and yaw angle of the multi-rotor UAV in the body coordinate system; is the angular velocity vector of the multi-rotor UAV in the body coordinate system; is the expected acceleration component of the multi-rotor UAV in the body coordinate system, is the velocity component of the UAV in the Earth coordinate system; The action is defined as: They are the coordinates of the multi-rotor UAV along the body coordinate system. Axis direction, direction, The desired angular velocity in the direction, is the expected lift.

6. The multi-rotor UAV interception control system based on reinforcement learning according to claim 5 is characterized in that: The environment configuration is a dynamic model of a multi-rotor drone: ， in, Used to represent the speed of the drone in the earth coordinate system is equal to its velocity vector in the Earth coordinate system; is the actual acceleration of the multi-rotor UAV in the Earth coordinate system; is the attitude change rate, is the angular velocity vector of the UAV in the body coordinate system, expressed as , for The antisymmetric matrix of ; is the inertial matrix of the multi-rotor drone; is the angular acceleration vector of the UAV in the body coordinate system; is the Coriolis moment due to angular velocity; is the gyroscopic torque; is the aerodynamic torque.

7. The multi-rotor UAV interception control system based on reinforcement learning according to claim 4 is characterized in that: The output layer of the actor network uses a tanh activation function and performs action scaling.

8. The multi-rotor UAV interception control system based on reinforcement learning according to claim 7 is characterized in that: After the action is scaled, use the clipping function Trim the action.

9. The multi-rotor UAV interception control system based on reinforcement learning according to claim 4, characterized in that: The actor network and critic network adopt fully connected neural networks, each including three hidden layers.

Images