Aircraft-trailer system coordination control method fusing reinforcement learning

Abstract

The invention relates to an aircraft-trailer system coordination control method fused with reinforcement learning, and the method comprises the steps: constructing an aircraft-trailer system mathematical model, and determining the coupling relation between control variables and state variables, the control variables including trailer movement control variables and system steering control variables; aiming at trailer movement control variables, designing a model-based self-adaptive controller; aiming at the steering control variable of the system, designing an attitude compensation controller based on reinforcement learning; the self-adaptive controller and the attitude compensation controller are fused, corresponding control signals are output to the aircraft and the trailer, and motion control over the aircraft and the trailer is completed. Compared with the prior art, the self-adaptive control law based on the virtual variable is designed through the decoupling kinematics model; and aiming at system parameters and external uncertainty, designing a compensation method based on reinforcement learning. A series of simulation is carried out under different conditions, and results show that the method can ensure coordinated movement of the aircraft and the trailer under different conditions.

Description

Technical Field

[0001] The present invention relates to the technical field of aircraft traction control, and in particular to an aircraft-trailer system coordinated control method integrating reinforcement learning. Background Art

[0002] Since the aircraft does not have a reverse gear function, it is necessary to use a towing operation to help the aircraft move from the parking position to the designated location. Traditional aircraft towing operations, such as dragging and taxiing, have been proven to effectively reduce operating costs and pollutant emissions. In addition, with the assistance of a towing truck, better path planning can be achieved, thereby improving airport operating efficiency.

[0003] However, the coupling characteristics of the aircraft-trailer system bring a series of challenges. To facilitate towing and steering, the aircraft and the towing trailer are mostly rigidly connected through the landing gear. This connection method needs to provide strong traction to ensure the movement of the heavy aircraft. Due to the nonlinear coupling of the aircraft-trailer system and the possible scissor effect, it is difficult to automatically control the aircraft-trailer system. In addition, the coordination of operations will also be affected by factors such as communication, air traffic control and system commands.

[0004] In summary, under the conditions of system coupling and external disturbance uncertainty, the existing control methods are difficult to ensure the coordinated movement of the aircraft and the trailer. Summary of the invention

[0005] The purpose of the present invention is to overcome the defects of the above-mentioned prior art and to provide an aircraft-trailer system coordinated control method integrating reinforcement learning, which can accurately perform trajectory tracking control on the uncertainty caused by system coupling and external interference, and realize the motion coordinated control of the aircraft-trailer system.

[0006] The object of the present invention can be achieved by the following technical solution: A coordinated control method of an aircraft-trailer system integrating reinforcement learning, comprising the following steps:

[0007] S1. Constructing a mathematical model of the aircraft-trailer system and determining the coupling relationship between control variables and state quantities, wherein the control variables include trailer movement control variables and system steering control variables;

[0008] S2. Design a model-based adaptive controller for trailer movement control variables;

[0009] Design an attitude compensation controller based on reinforcement learning for the system steering control variables;

[0010] S3, integrates the adaptive controller and the attitude compensation controller, outputs corresponding control signals to the aircraft and trailer, and completes the motion control of the aircraft and trailer.

[0011] Furthermore, the specific process of step S1 is as follows:

[0012] Define ΣOa, ΣOm, and ΣOt as frames fixed on the aircraft, steering mechanism, and trailer, respectively. Frame ΣOa is used to describe the aircraft position p a , the frame ΣOm is used to describe the steering mechanism position p m , the framework ΣOt is used to describe the trailer position p t ;

[0013] Definitions a =[x a ,y a ,γ a ] T represents the attitude of the aircraft in the global frame ΣOg, s t =[x t ,y t ,γ t ] T represents the posture of the trailer in the global frame ΣOg;

[0014] Considering the motion state of the aircraft, s a and t The relationship between them is,

[0015]

[0016] sγ a = sin(γ a )

[0017] sγ m = sin(γ m )

[0018] cγ a = cos(γ a )

[0019] cγ m = cos(γ m )

[0020] Among them, γ t is the rotation angle of the trailer, γ m is the rotation angle of the steering mechanism, γ a is the rotation angle of the aircraft, L ma and L tm They are the distance between the steering mechanism and the aircraft, and the distance between the trailer and the steering mechanism;

[0021] Targeting a and t Derivative the relationship between:

[0022]

[0023] Among them, q s is the joint space state of the aircraft-trailer system, and S(s) is the state transition matrix.

[0024] Furthermore, step S2 includes the following process:

[0025] S21, control variable x for trailer movement t and t , introduce the dummy variable μ t , to achieve x t and t After the decoupling, the adaptive control rate is designed to perform closed-loop control on the trailer moving speed;

[0026] S22, for the system steering control variable γ t and γ m , build a reinforcement learning agent to achieve smooth steering operation.

[0027] Furthermore, the virtual variable μ in step S21 t By turning variable γ t and γ m Jointly decided, and meeting: The dummy variable μ t Specifically:

[0028]

[0029] Among them, x e and e They are respectively the x of the aircraft-trailer system in the world coordinate system g and g Error in direction, c s =[c s,k1 ,c s,d1 ,c s,k2 ,c s,d2 ] is the control parameter, which is used to adjust the error x e , y e , The gain, and is the desired position of the trailer, e γ,t is the trailer rotation angle error, e γ,m is the rotation angle error of the rotating mechanism.

[0030] Furthermore, the specific process of step S21 is as follows:

[0031] For trailer speed control, and As control variables, these three control variables are coupled, and the selection and As the independent variable, and define the error as x e,t and e,t , and then introduce a virtual input μ t ,get:

[0032]

[0033] Define the Lyapunov function And find the derivative:

[0034]

[0035] Design the position control law:

[0036]

[0037] Among them, c x,k and c y,k is the control parameter, and the speed control law v t and Adaptive Law Set to respectively,

[0038]

[0039] in, is an auxiliary signal, λ k is a positive real number, c x,d is a normal number.

[0040] Furthermore, the step S22 adopts a two-layer reinforcement learning (RL) architecture to synthesize the virtual variable μ t and steering angle ω m Attitude compensation controller.

[0041] Furthermore, the two-layer reinforcement learning architecture includes a basic strategy module π and an adaptive module ρ, wherein the basic strategy module π is based on the current state, previous actions, different expected trajectories and adaptive learning rate As input, the steering motion control strategy is output, and the adaptive module ρ is trained to improve the system's adaptability to uncertain factors, including internal parameters and external changes.

[0042] Furthermore, the working process of the basic strategy module π includes:

[0043] At each step, the input state is defined as s s =[s a ,s t ,s m] is the current global attitude of the trailer, aircraft and steering mechanism, Yes s The first derivative of , s a,d is the motion reference trajectory of the aircraft;

[0044] The action is represented by a t =[ω m ,μ t ] T ,ω m is the action sequence for updating the steering mechanism angle, and the designed virtual input μ t Used to coordinate and adjust the linear and angular speed of the system;

[0045] Convert the goal-conditional reinforcement learning problem into a Markov decision process: The elements in are states, actions, transition dynamics, reward functions, and a discount factor γ. For any state-action sequence t-1 ,s t ,r t >, the RL agent is trained to construct a policy π that maximizes the expected reward, and then the agent solves the problem based on the above policy. The policy π is defined as

[0046]

[0047] where τ={(s1,a1,r1),(s2,a2,r2),…,(s t ,a t ,r t )} is the sequence of agents executing strategy π, P = [P1, P2, …, P t ] is the possible sequence of actions under π.

[0048] In order to guide the agent to coordinate motion under the coupling and nonlinearity of the aircraft-trailer system, the reward function is designed to include task orientation and performance improvement. The task orientation reward is used to encourage the aircraft-trailer system to track the desired trajectory s in different environments. a,d ; Performance improvement rewards focus on dynamic adjustment. In addition, the reward function also takes into account real-time status and predicted status.

[0049] Furthermore, the task-oriented reward is defined as:

[0050]

[0051] Among them, r0 is the initial reward, e s =[e a ,e t ,e m ] is the state error matrix, e s =e​s,t -e s,d , ξ0 is an adjustable parameter, Δ e is the threshold, f(s,a) is the defined reward rate function, P1, P2 and P3 are positive definite matrices, e s T P1e s Describes the impact of state error on the reward function;

[0052] The performance bonus is defined as:

[0053]

[0054] in, is the difference between the action values ​​at time t and t-1, e tm =e t -e m is the relative error between the trailer and the steering mechanism, P a , P e and P tm is a positive definite matrix, and They are used to adjust the smoothness of actions and states respectively. Used to improve the coordination of relative steering movements, during training, the scaling factor of each reward term is adjusted to maximize the reward.

[0055] Furthermore, the working process of the adaptive module p includes:

[0056] For the nominal inertia matrix and uncertain external friction To compensate, define for:

[0057]

[0058] Where s = [s t ,f] is the input vector, is the Gaussian basis function, W M is the learning network weight, and the adaptive learning rule is

[0059] For the adaptive module, a radial basis function-based neural network is used to approximate the uncertain dynamics of the system. The neural network consists of three feedforward layers. The input layer includes the system state and the defined nonlinear friction force. The hidden layer is a radial basis vector with 10 nodes. The output layer is the adaptive terms of the internal system parameters and the external uncertain friction. As the input of the base strategy π.

[0060] Compared with the prior art, the present invention has the following advantages:

[0061] The present invention first describes the model of the aircraft-trailer system and analyzes the coupling relationship between control states. Then, for the control variables describing the movement of the trailer, a model-based adaptive controller is designed. For the control variables describing the steering of the aircraft-trailer system, a reinforcement learning agent is constructed and a reinforcement learning-based attitude compensation controller is designed. Finally, the adaptive controller and the attitude compensation controller are integrated to achieve coordinated control of the aircraft-trailer system and accurate trajectory tracking control of the uncertainty caused by model coupling and external interference.

[0062] The present invention introduces virtual variables to design an adaptive controller through kinematic decoupling. The virtual variables are determined by the rotation angle of the trailer and the rotation angle of the rotating mechanism, which can realize system position decoupling and ensure the trailer state [x t ,y t ] and linear velocity v t tracking performance.

[0063] Aiming at the system parameter changes and external uncertainties, the present invention designs a compensation controller based on reinforcement learning, adopts a double-layer reinforcement learning architecture to synthesize virtual variables μ t and steering angle ω m The agile controller of the two-layer reinforcement learning architecture consists of two layers: the basic strategy π and the adaptive module ρ. The basic strategy π is used to ensure coordinated steering motion, and the task orientation and performance improvement rewards are designed, which can guide the agent to coordinate motion under the coupling and nonlinearity of the aircraft-trailer system; the adaptive module ρ considers the time-varying characteristics of the system parameters and the external uncertainty, and can effectively compensate for the inertia matrix and uncertain external friction, thereby improving the coordination of motion. BRIEF DESCRIPTION OF THE DRAWINGS

[0064] Figure 1 It is a schematic diagram of the method flow of the present invention;

[0065] Figure 2 It is a schematic diagram of the application process of the embodiment;

[0066] Figure 3 The aircraft-trailer system model involved in the present invention;

[0067] Figure 4 A coordinated control diagram for an aircraft-trailer system incorporating reinforcement learning;

[0068] Figure 5 is the control strategy diagram for reinforcement learning;

[0069] Figure 6 This is the simulation scenario under different mass and friction parameters (simulation scenario 1);

[0070] Figure 7 This is a reward picture for simulation scenario 1;

[0071] Figure 8 It is a trajectory tracking simulation scenario (simulation scenario 2);

[0072] Fig. 9 This is the control rate change diagram for simulation scenario 2;

[0073] Fig.10 is the trajectory tracking error of simulation scenario 2;

[0074] Fig.11 This is the reward picture for simulation scenario 2. DETAILED DESCRIPTION

[0075] The present invention is described in detail below with reference to the accompanying drawings and specific embodiments.

[0076] Example

[0077] like Figure 1 As shown, a coordinated control method of an aircraft-trailer system integrating reinforcement learning comprises the following steps:

[0078] S1. Constructing a mathematical model of the aircraft-trailer system and determining the coupling relationship between control variables and state quantities, wherein the control variables include trailer movement control variables and system steering control variables;

[0079] S2. Design a model-based adaptive controller for trailer movement control variables;

[0080] Design an attitude compensation controller based on reinforcement learning for the system steering control variables;

[0081] S3, integrates the adaptive controller and the attitude compensation controller, outputs corresponding control signals to the aircraft and trailer, and completes the motion control of the aircraft and trailer.

[0082] This embodiment applies the above solution, and the process is as follows Figure 2 As shown in the figure, the aircraft-trailer system is first modeled and the coupling relationship between control states is analyzed. t and t , design a model-based adaptive control strategy. For the control variable γ describing the steering of the aircraft-trailer system t and γ m , build a reinforcement learning agent to achieve smooth steering operation.

[0083] Control variable x for trailer movement t and t , introduce the dummy variable μ t , to achieve xt and t After decoupling, an adaptive control rate is designed to perform closed-loop control on the trailer moving speed.

[0084] Dummy variable μ t From aircraft tug system to variable γ t and γ m Joint decisions should satisfy:

[0085] To ensure that the dummy variable μ t Convergence, design reinforcement learning agent, define the state as Agent behavior is defined as a t =[ω m ,μ t ] T , by interacting with the aircraft-trailer model environment, attitude stability control is ensured.

[0086] The model-based adaptive controller and reinforcement learning-based controller are integrated to achieve coordinated control of the aircraft-trailer system.

[0087] Specifically include:

[0088] 1. Aircraft-trailer system model construction

[0089] like Figure 3 As shown in the system description section, the system consists of an aircraft and a trailer, which are connected by a designed steering mechanism. In order to effectively describe the modeling process, the preparation work and related definitions are as follows: ΣOg is taken as the global frame. Let ΣOa, ΣOm and ΣOt represent the frames fixed on the aircraft, steering mechanism and trailer respectively. The frame ΣOa is used to describe the aircraft position p a , the frame ΣOm is used to describe the steering mechanism position p m , the framework ΣOt is used to describe the trailer position p t .s a =[x a ,y a ,γ a ] T represents the attitude of the aircraft in the frame ΣOg, s t =[x t ,y t ,γ t ] T Represents the pose of the trailer in the global frame ΣOg. Similarly, other states in the world coordinate system are also marked with superscripts. a,m is the relative angle between the aircraft and the steering mechanism, γ m,t is the relative angle between the steering mechanism and the trailer. a and Dt are the wheelbases of the aircraft and trailer respectively. ma and L tm They are point p a 、p m and p t The distance between.

[0090] Different from the independent Ackerman kinematics model, this system needs to consider the motion state of the aircraft, and the control goal is also to achieve the desired motion of the aircraft. a and t The relationship between can be derived as follows:

[0091]

[0092] Where sγ a = sin(γ a ), sγ m = sin(γ m ), cγ a = cos(γ a ), cγ m = cos(γ m ), γ t is the rotation angle of the trailer, γ m is the rotation angle of the steering mechanism, γ a is the rotation angle of the aircraft, L ma and L tm They are the distance between the steering mechanism and the aircraft, and the distance between the trailer and the steering mechanism.

[0093] By taking the derivative of formula (1), we can obtain:

[0094]

[0095] Where S(s) is the state transition matrix, and S(s) is defined as follows:

[0096]

[0097] It can be seen from formula (1) and formula (2) that the aircraft-trailer system is in a coupling with a defined state. Specifically, the angular velocity of the trailer depends on its own attitude and its relative state with the aircraft. In addition, formula (1) and formula (2) describe the kinematic characteristics of the system. If factors such as joint uncertainty and motion friction are considered, the dynamic characteristics of the system will become more complicated. For this coupling variable, this scheme design uses reinforcement learning method to deal with it.

[0098] 2. Design of Adaptive Compensation Controller

[0099] Based on the above analysis, a coordinated control scheme integrating reinforcement learning (RL) and closed-loop feedback is proposed. t ), an adaptive control method is adopted. For the coupled angular velocity (system variable γ t and γ m ), an agent is used to better implement steering decisions under different conditions and uncertainties. The control framework is as follows Figure 4 shown.

[0100] For trailer speed control, you can and As the control variable. These three variables are coupled, here we choose and As the independent variable, and define the error as x e,t and e,t . Therefore, it can be regarded as point p t Position control, and then introduce a virtual input μ t ,get:

[0101]

[0102] Define the Lyapunov function And the derivation can be obtained,

[0103]

[0104] in, and is the desired position of the trailer.

[0105] Design the position control law,

[0106]

[0107] Among them, c x,k and c y,k is the control parameter.

[0108] Therefore, the speed control law v t and Adaptive Law Set to:

[0109]

[0110] In the formula, is an auxiliary signal, λ k is a positive real number, c x,d is a normal number.

[0111] After the aircraft and trailer form a positive body, the designed virtual input μ t We need to consider the system's posture, and we can get

[0112]

[0113] In the formula, x e and e They are respectively the x of the aircraft-trailer system in the world coordinate system h and g Error in direction. c s =[c s,k1 ,c s,d1 ,c s,k2 ,c s,d2 ] is the control parameter.

[0114] Formula (7) is designed based on Lyapunov stability, which can ensure the trailer state [x t ,y t ] and linear velocity v t But it cannot guarantee the tracking performance of the dummy variable μ t In the aircraft-trailer system, μ t By γ t and γ m Joint decisions should satisfy:

[0115] The parameter matrix c in formula (9) s Only the convergence rate can be adjusted but the stability cannot be guaranteed. Therefore, the coupled directional variables and uncertainties are intended to be controlled by a designed reinforcement learning (RL) agent.

[0116] 3. Reinforcement Learning Coordination Strategy

[0117] The goal of this proposal is to let the agent control the directional variables of the aircraft-trailer system, and then train the policy to ensure more flexible steering under the constraints of large size and inertia. To achieve this, this proposal proposes a two-layer reinforcement learning (RL) architecture to synthesize the virtual variable μ t and steering angle ω m The learning model is summarized as follows: Figure 5 shown.

[0118] The RL architecture consists of two layers, the base policy π and the adaptive module ρ. The base policy π is based on the current state, previous actions, different desired trajectories, and adaptive learning rate. as input. The adaptive module ρ is trained to improve the system's adaptability to uncertainties, including internal parameters (such as mass changes) and external changes (friction in different terrains). They work together to achieve strong adaptability to diverse environments and strongly coupled uncertainties within the system.

[0119] (1) Basic Strategy π

[0120] At each step, the input state is defined as Here s =[s a ,s t ,s m ] is the current global attitude of the trailer, aircraft and steering mechanism, Yes s In addition, unlike the traditional towing system, this scheme directly defines the motion trajectory of the aircraft as the reference trajectory s a,d , which ensures the aircraft's kinematic performance. At the speed level, the analysis is performed based on formula (2), where the linear speed is achieved through a model-based control law (see formula (7)), while the designed policy π only cares about how to ensure the steering motion more coordinated.

[0121] The action is represented by a t =[ω m ,μ t ] T , here, ω m is the action sequence to update the steering mechanism angle. The designed virtual input μ t Used to coordinate the linear and angular speeds of the system. Unlike differential drive trailers, the variable μ t The coupling effect of the aircraft and the trailer needs to be considered. RL provides an adaptive learning framework that can be applied to control decisions of coupled and nonlinear models. This embodiment uses the advantage actor-critic (A2C) to achieve control of the system and formulates this problem as a goal-conditioned reinforcement learning problem. Of course, other RL algorithms are also applicable.

[0122] This problem is then transformed into a Markov decision process, expressed as: The elements are state, action, transition dynamics, reward function and a discount factor γ. For any state-action sequence t-1 ,s t ,r t >. The RL agent is trained to construct a policy π that maximizes the expected reward, and then the agent solves the problem based on the above policy. The policy π is defined as,

[0123]

[0124] Where τ={(s1,a1,r1),(s2,a2,r2),…,(s t ,a t ,r t )} is the sequence of agents executing strategy π, P = [P1, P2, …, P t ] is the possible sequence of actions under π.​

[0125] In order to guide the agent to coordinate motion under the coupling and nonlinearity of the aircraft-trailer system, the reward is designed to have two parts: task-oriented and performance-enhancing. The task-oriented reward encourages the aircraft-trailer system to track the desired trajectory s in different environments. a,d The performance-enhancing reward focuses on dynamic adjustments, such as the crawling problem caused by the huge mass of the aircraft and the instability caused by external uncertainties. In addition, the reward function also takes into account the real-time state and the predicted state to ensure the convergence of the RL agent.

[0126] The task-oriented part is defined as,

[0127]

[0128] In the formula, r0 is the initial reward, e s =[e a ,e t ,e m ] is the state error matrix, e s =e s,t -e s,d , ξ0 is an adjustable parameter, Δ e is the threshold. f(s,a) is the reward rate function defined,

[0129]

[0130] Where P1, P2 and P3 are positive definite matrices. s T P1e s Describes the impact of state error on the reward function.

[0131] The performance improvement part is defined as,

[0132]

[0133] In the formula, is the difference between the action values ​​at time t and t-1, and similarly e tm =e t -e m is the relative error between the trailer and the steering mechanism. a , P e and P tm is a positive definite matrix. Formula (12) describes the impact of the state error and its derivative on the agent. The term in formula (3) and Adjust the smoothness of actions and states respectively. This improves the coordination of the relative steering motion. During training, the scaling factor of each reward term is adjusted to maximize the reward.

[0134] (2) Adaptive module

[0135] The basic policy model based on state feature analysis and expected motion trajectory enables the agent to make decisions at the kinematic level. The basic policy π enables motion decisions at the kinematic level. However, the time-varying characteristics of system parameters and external uncertainties may affect the coordination of motion. To address this issue, this scheme designs an adaptive module ρ, which takes effect through a set of learned adaptive weights to ensure stability and convergence.

[0136] The aircraft has a very large and variable mass, which will cause changes in system parameters and affect the dynamic characteristics of the system. In addition, the change in mass will also affect the friction parameters, resulting in problems such as slow crawling or steady-state oscillation during the system movement. In addition, since the aircraft-trailer system is in a low-speed motion state when taxiing, this design has a certain influence on the nominal inertia matrix. and uncertain external friction Compensation. Definition for,

[0137]

[0138] Where s = [s t ,f] is the input vector, is the Gaussian basis function, W M is the learning network weight, and the adaptive learning rule is

[0139] For the adaptive learning module, a neural network based on radial basis functions is used to approximate the uncertain dynamics of the system. The neural network consists of three feed-forward layers. The input layer includes the system state and the defined nonlinear friction. The hidden layer is a radial basis vector with 10 nodes. The output layer is the adaptive terms of the internal system parameters and the external uncertain friction. Finally, is used as input to the base policy π.

[0140] To verify the effectiveness of this solution, this embodiment verifies the solution in simulation. Two scenarios are considered in the simulation environment: one is that when the mass of the transported load changes suddenly, the load parameters of the coordinated transport system will change suddenly, and the smooth adjustment ability of the coordinated impedance model under this condition is verified. The other is trajectory tracking motion, by designing different motion trajectories, the control accuracy of the proposed control method is verified.

[0141] In the above simulation scenario, the aircraft-trailer system is implemented in a simulation platform based on dynamics engine interaction, and the model scale is scaled 25 times according to the real aircraft-trailer system.

[0142] The following are two specific cases:

[0143] Case 1: Simulation verification of mass and friction change scenarios

[0144] Simulation scenario setup: The aircraft-trailer system moves in scenarios with different ground frictions, and is expected to move along a sinusoidal trajectory. The mass of the aircraft changes during the motion, and this simulation verifies the robustness of the controller under conditions of sudden changes in mass and friction parameters. In this training scenario, the angle between the aircraft and the trailer is arbitrary. The agent's goal is to reduce the angle to the desired tracking state.

[0145] like Figure 6 As shown in the figure, the agent was trained for 2000 rounds, with 50 steps of iterations per round, and the average reward was output. Under different mass and friction conditions, the agent was able to converge well under the designed reward strategy. As the number of training times increases, the fluctuation of the reward also gradually decreases, which shows that the agent has memory and adaptability. In addition, under different internal conditions, the reward also shows different fluctuations. In particular, in the fourth case (random changes in friction), the step reward also has a certain fluctuation, but the average reward that determines the learning ability of the agent shows higher smoothness. When the mass of the aircraft is 25 kg, the agent needs about 150 rounds to stabilize, while when the mass is 15 kg, it only takes 80 rounds to adapt. This shows that the internal parameters have a certain influence on the training and learning of the agent.

[0146] Figure 7 This is a simulation of the reward changes in a scene under different terrain conditions. It can be seen that under the fusion control proposed in this scheme, the reward converges quickly and stabilizes, indicating that the proposed method has a faster convergence speed. In addition, under different terrains, although the step reward fluctuates, the average reward is generally stable, indicating that the algorithm has high robustness.

[0147] Case 2: Trajectory tracking simulation verification

[0148] In order to better observe the control performance under different speeds and trajectories, double circle trajectory tracking is selected as the trajectory tracking task. The diameters of the two circles are D1 = 9 meters and D2 = 6 meters respectively. The tangents of the two circles can verify the accuracy of repeated positioning. In this simulation, the initial position of the trailer is set to [ g x t,0 , g y t,0 , g γ t,0 ]=[-0.2m,-0.3m,12°], the expected initial position is [ g xt,des , g y t,des , g γ t,des ]=[0m,0m,0°]. This is to verify the convergence ability and dynamic response ability of the proposed algorithm.

[0149] like Figure 8 As shown, the aircraft-trailer system should achieve the desired trajectory tracking within a given time t = 400 seconds. At about t = 15 seconds, the tug pulls the aircraft into the desired trajectory. The green and red solid lines represent the trajectories of the tug and the aircraft, respectively. At t = 100 seconds, half of the trajectory tracking of a circle with a diameter of D1 = 9m is completed. The black dashed line represents the desired trajectory. At t = 200 seconds, the system completes half of the trajectory tracking. At t = 200 seconds, the system completes the trajectory tracking. The proposed controller achieves effective trajectory tracking and a high level of repeatability accuracy.

[0150] The control law of the trailer, such as Fig. 9 As shown, the relevant characteristics can also be reflected. In the first step (within about 10 seconds), fast linear and angular velocities are used to correct the initial state error. The second and third steps correspond to the trajectory tracking of the large circle and the small circle, respectively. It can be observed that the control law at the switching point (t = 200 seconds) is very smooth. Fig.10 is the tracking error of the aircraft and trailer in Cartesian space. The position and attitude errors are distinguished by solid and dashed lines, and different colors are used to distinguish different dimensions. Despite the large mass and inertia of the aircraft and the lack of trajectory correction capability, the control of the tracking error using the proposed controller is still very effective. Especially in the initial stage, the tracking error of the aircraft is smaller than the tracking error of the trailer.

[0151] Fig.11 is the reward of the designed reinforcement learning (RL) agent, which shows better convergence in a dynamically coupled environment. Fig.11 From the reward convergence diagram shown, we can see that when the aircraft completes two circles, the reinforcement learning algorithm is mainly used to achieve stable control of different angular velocities. The angular velocity needs to converge quickly to the desired angle. The reward convergence speed along the way can reflect the error tracking of the angular velocity. Fig.11 The faster convergence speed in ensures the trajectory tracking accuracy. Despite various unstable factors in this simulation, the cumulative reward shows good convergence and stability. This is because the coupled state of the system is fully considered in the design of the agent and further adjustments are adopted in the controller design process.

[0152] In summary, this scheme proposes a method combining a model-based controller and a reinforcement learning algorithm for the trajectory tracking control problem of the taxiing aircraft-trailer system. This method first designs an adaptive speed controller based on virtual variables through kinematic decoupling, and then designs a reinforcement learning agent to handle system coupling and uncertainty for attitude control, so that the controller can accurately track the uncertainty caused by model coupling and external interference, thereby realizing the motion coordination of the aircraft-trailer system.

[0153] This scheme achieves precise control of the system coupling characteristics by combining model foundation and data-driven technology. The method first establishes a mathematical model and analyzes the coupling characteristics, then designs virtual variables to achieve system position decoupling, then builds a speed controller based on Lyapunov stability, and realizes intelligent control of the system direction through reinforcement learning agents. Finally, by integrating the closed-loop control law and the RL agent output, the control strategy is optimized, the control performance is improved, and the complexity and training time of the RL agent are reduced. In the simulation test, it is verified that the aircraft can achieve precise trajectory tracking according to this method, showing significant advantages in improving control accuracy, enhancing system stability, and optimizing control strategies.

Claims

1. A coordinated control method for an aircraft-trailer system integrating reinforcement learning, characterized in that: The following steps are involved: S1. Constructing a mathematical model of the aircraft-trailer system and determining the coupling relationship between control variables and state quantities, wherein the control variables include trailer movement control variables and system steering control variables; S2. Design a model-based adaptive controller for trailer movement control variables; Design an attitude compensation controller based on reinforcement learning for the system steering control variables; S3, integrates the adaptive controller and the attitude compensation controller, outputs corresponding control signals to the aircraft and trailer, and completes the motion control of the aircraft and trailer.

2. The aircraft-trailer system coordinated control method integrating reinforcement learning according to claim 1 is characterized in that: The specific process of step S1 is as follows: Define ΣOa, ΣOm, and ΣOt as frames fixed on the aircraft, steering mechanism, and trailer, respectively. Frame ΣOa is used to describe the aircraft position p a , the frame ΣOm is used to describe the steering mechanism position p m , the framework ΣOt is used to describe the trailer position p t ; Definitions a =[x a ,y a ,γ a ] T represents the attitude of the aircraft in the global frame ΣOg, s t =[x t ,y t ,γ t ] T represents the posture of the trailer in the global frame ΣOg; Considering the motion state of the aircraft, s a and t The relationship between them is, sg a =sin(γ a ) sg m =sin(γ m ) cg a =cos(γ a ) cg m =cos(γ m ) Among them, γ t is the rotation angle of the trailer, γ m is the rotation angle of the steering mechanism, γ a is the rotation angle of the aircraft, L ma and L tm They are the distance between the steering mechanism and the aircraft, and the distance between the trailer and the steering mechanism; Targeting a and t Derivate the relationship between: Among them, q s is the joint space state of the aircraft-trailer system, and S(s) is the state transition matrix.

3. The aircraft-trailer system coordinated control method integrating reinforcement learning according to claim 2 is characterized in that: The step S2 includes the following process: S21, control variable x for trailer movement t and t , introduce the dummy variable μ t , to achieve x t and t After the decoupling, the adaptive control rate is designed to perform closed-loop control on the trailer moving speed; S22, for the system steering control variable γ t and γ m , build a reinforcement learning agent to achieve smooth steering operation.

4. The aircraft-trailer system coordinated control method integrating reinforcement learning according to claim 3 is characterized in that: The virtual variable μ in step S21 t By turning variable γ t and γ m Jointly decided, and meeting: The dummy variable μ t Specifically: Among them, x e and e They are respectively the x of the aircraft-trailer system in the world coordinate system g and g Error in direction, c s =[c s,k1 ,c s,d1 ,c s,k2 ,c s,d2 ] is the control parameter, which is used to adjust the error x e , y e , The gain, and is the desired position of the trailer, e γ,t is the trailer rotation angle error, e γ,m is the rotation angle error of the rotating mechanism.

5. The aircraft-trailer system coordinated control method integrating reinforcement learning according to claim 4 is characterized in that: The specific process of step S21 is as follows: For trailer speed control, and As control variables, these three control variables are coupled, and the selection and As the independent variable, and define the error as x e,t and e,t , and then introduce a virtual input μ t ,get: Define the Lyapunov function And find the derivative: Design the position control law: Among them, c x,k and c y,k is the control parameter, and the speed control law v t and adaptive law Set to respectively, in, is an auxiliary signal, λ k is a positive real number, c x,d is a normal number.

6. The aircraft-trailer system coordinated control method integrating reinforcement learning according to claim 5 is characterized in that: The step S22 adopts a two-layer reinforcement learning architecture to synthesize the virtual variable μ t and steering angle ω m Attitude compensation controller.

7. The aircraft-trailer system coordinated control method integrating reinforcement learning according to claim 6 is characterized in that: The two-layer reinforcement learning architecture includes a basic strategy module π and an adaptive module ρ. The basic strategy module π is based on the current state, previous actions, different expected trajectories and adaptive learning rate. As input, the steering motion control strategy is output, and the adaptive module ρ is trained to improve the system's adaptability to uncertain factors, including internal parameters and external changes.

8. The aircraft-trailer system coordinated control method integrating reinforcement learning according to claim 7 is characterized in that: The working process of the basic strategy module π includes: At each step, the input state is defined as s s =[s a ,s t ,s m ] is the current global attitude of the trailer, aircraft and steering mechanism, Yes s The first derivative of , s a,d is the motion reference trajectory of the aircraft; The action is represented by a t =[ω m ,μ t ] T ,ω m is the action sequence for updating the steering mechanism angle, and the designed virtual input μ t Used to coordinate and adjust the linear and angular speed of the system; Convert the goal-conditional reinforcement learning problem into a Markov decision process: The elements in are states, actions, transition dynamics, reward functions, and a discount factor γ. For any state-action sequence t-1 ,s t ,r t >, the RL agent is trained to construct a policy π that maximizes the expected reward, and then the agent solves the problem based on the above policy. The policy π is defined as​ where τ={(s1,a1,r1),(s2,a2,r2),…,(s t ,a t ,r t )} is the sequence of agents executing strategy π, P = [P1, P2, …, P t ] is the possible sequence of actions under π; In order to guide the agent to coordinate motion under the coupling and nonlinearity of the aircraft-trailer system, the reward function is designed to include task orientation and performance improvement. The task orientation reward is used to encourage the aircraft-trailer system to track the desired trajectory s in different environments. a,d ; Performance improvement rewards focus on dynamic adjustment. In addition, the reward function also takes into account real-time status and predicted status.

9. The aircraft-trailer system coordinated control method integrating reinforcement learning according to claim 8 is characterized in that: The task-oriented reward is defined as: Among them, r0 is the initial reward, e s =[e a ,e t ,e m ] is the state error matrix, e s =e s,t -e s,d , ξ0 is an adjustable parameter, Δ e is the threshold, f(s,a) is the defined reward rate function, P1, P2 and P3 are positive definite matrices, e s T P1e s Describes the impact of state error on the reward function; The performance bonus is defined as: in, is the difference between the action values ​​at time t and t-1, e tm =e t -e m is the relative error between the trailer and the steering mechanism, P a , P e and P tm is a positive definite matrix, and They are used to adjust the smoothness of actions and states respectively. Used to improve the coordination of relative steering movements, during training, the scaling factor of each reward term is adjusted to maximize the reward.

10. The aircraft-trailer system coordinated control method integrating reinforcement learning according to claim 8, characterized in that: The working process of the adaptive module p includes: For the nominal inertia matrix and uncertain external friction To compensate, define for: Where s = [s t ,f] is the input vector, is the Gaussian basis function, W M is the learning network weight, and the adaptive learning rule is For the adaptive module, a radial basis function-based neural network is used to approximate the uncertain dynamics of the system. The neural network consists of three feedforward layers. The input layer includes the system state and the defined nonlinear friction force. The hidden layer is a radial basis vector with 10 nodes. The output layer is the adaptive terms of the internal system parameters and the external uncertain friction. As the input of the base strategy π.

Images