Cluster cooperative trajectory optimization method and device based on multi-agent reinforcement learning

Abstract

Embodiments of the present application relate to the technical field of multi-agent systems, reinforcement learning and autonomous control, and particularly relate to a cluster cooperative trajectory optimization method and device based on multi-agent reinforcement learning. A specific embodiment of the method comprises: a global planning layer generating global reference flight path information based on a preset heuristic evolutionary genetic algorithm; a local tracking layer generating a desired attack angle and a roll angle instruction; and a bottom control layer generating a sequence of rudder deflection angle values. The embodiment can effectively integrate global planning and local control and achieve hierarchical cooperation.

Description

Technical Field

[0001] The embodiments of this application relate to the fields of multi-agent systems, reinforcement learning, and autonomous control technology, specifically to a cluster cooperative trajectory optimization method and apparatus based on multi-agent reinforcement learning. Background Technology

[0002] With the continuous expansion of civil aviation and UAV applications and the increasing complexity of missions, the limitations of traditional single-aircraft independent operation are becoming increasingly apparent. Not only are they susceptible to weather, obstacles, and dynamic traffic flow in complex airspace, hindering mission efficiency, but flight safety also faces severe challenges, making them unsuitable for the modern high-density and diverse flight environment. Multi-aircraft cooperative flight can significantly improve airspace utilization and mission completion capabilities, and will inevitably become an important operating mode for future air traffic and UAV systems. The technological evolution of cooperative trajectory planning has always revolved around the two core requirements of global optimality and dynamic adaptability, and has currently formed a dual-track development pattern of global planning algorithms and local control algorithms.

[0003] At the global collaborative planning level, genetic algorithms (GA) and their improved variants are among the mainstream technologies. Their core advantage lies in simulating the selection-crossover-mutation mechanism of biological evolution, enabling efficient searching of globally approximate optimal trajectories in high-dimensional solution spaces with multiple constraints and objectives. Meanwhile, model predictive control (MPC) and its derivative technologies have become research frontiers: MPC updates environmental information in real time through rolling time-domain optimization, solving finite-time-domain optimization problems within each control cycle to achieve a "prediction-optimization-execution" closed loop, enabling UAVs to maintain adaptability in highly dynamic and multi-constraint environments. However, traditional methods still have significant shortcomings in complex dynamic environments, multi-agent collaboration, and real-time performance.

[0004] This application constructs a simulation environment for collaborative flight of aircraft swarms. The mission scenario consists of 10 aircraft that must traverse a three-dimensional airspace containing multiple dynamic obstacle zones and achieve coordinated arrival under limited communication conditions. Mission objectives include multiple indicators such as the number of aircraft reaching the target area, the distance to the target center, the arrival speed, and the arrival time difference. The aircraft motion is subject to stringent physical constraints, including dynamic constraints, heat flux density constraints, dynamic pressure constraints, overload constraints, and obstacle avoidance.

[0005] Existing methods generally suffer from low planning efficiency, insufficient tracking accuracy, poor coordination consistency, and high computational complexity when dealing with highly dynamic environments, multi-machine collaboration, and complex constraints. Traditional genetic algorithms are prone to getting trapped in local optima, and the fixed step size of MPC is difficult to adapt to dynamic changes. While multi-agent reinforcement learning has adaptive potential, how to effectively integrate global planning and local control and achieve hierarchical collaboration remains a technical challenge.

[0006] Existing technologies suffer from the following problems: Traditional single-aircraft independent operation modes are susceptible to weather, obstacles, and dynamic traffic flow in complex airspaces, resulting in limited flight safety and mission execution efficiency, making them unsuitable for modern high-density and diverse flight environments. Existing global planning algorithms (such as genetic algorithms) have low search efficiency in multi-constraint, high-dimensional solution spaces, are prone to getting trapped in local optima, and struggle to guarantee multi-aircraft collaborative consistency. Local control algorithms (such as MPC) lack adaptability in dynamic environments, and fixed prediction step sizes cannot simultaneously meet computational efficiency and control performance requirements. In multi-agent systems, independent planning by each unit easily leads to conflicts, lacking effective collaborative mechanisms and communication constraint handling. Traditional methods have limited ability to handle complex physical constraints (such as dynamic pressure, heat flux, and overload), making it difficult to guarantee safety and accuracy under high-speed flight conditions. Existing learning-based methods are highly dependent on models, have insufficient generalization ability, and are costly to train, making them difficult to deploy in practical systems. Summary of the Invention

[0007] The summary section of this application is intended to provide a brief overview of the concepts, which will be described in detail in the detailed description section below. This summary section is not intended to identify key or essential features of the claimed technical solutions, nor is it intended to limit the scope of the claimed technical solutions.

[0008] Some embodiments of this application propose a cluster cooperative trajectory optimization method, apparatus, computer device, and computer-readable storage medium based on multi-agent reinforcement learning to solve one or more of the technical problems mentioned in the background section above.

[0009] In a first aspect, some embodiments of this application provide a cluster cooperative trajectory optimization method based on multi-agent reinforcement learning, applied to a cluster cooperative trajectory optimization system. The system comprises a global planning layer, a local tracking layer, and a bottom-level control layer. The method includes: the global planning layer, in response to receiving the coordinates of the task target point, a global environment map, and the cluster initial state vector, generating global reference trajectory information based on a preset heuristic evolutionary genetic algorithm, and sending the global reference trajectory information to the local tracking layer; the local tracking layer, in response to receiving the global reference trajectory information sent by the global planning layer and in response to receiving node cooperative state information, generating desired angle of attack and bank angle commands, and sending the desired angle of attack and bank angle commands to the bottom-level control layer; and the bottom-level control layer, in response to receiving the desired angle of attack and bank angle commands sent by the local tracking layer and in response to receiving aerodynamic parameters and physical state information, generating a numerical sequence of control surface deflection angles.

[0010] Secondly, some embodiments of this application provide a cluster cooperative trajectory optimization device based on multi-agent reinforcement learning. The device includes: a global generation unit configured to, in response to receiving the coordinates of the task target point, a global environment map, and the initial state vector of the cluster, generate global reference trajectory information based on a preset heuristic evolutionary genetic algorithm, and send the global reference trajectory information to the local tracking layer; a local generation unit configured to, in response to receiving the global reference trajectory information sent by the global planning layer and in response to receiving node cooperative state information, generate desired angle of attack and roll angle commands, and send the desired angle of attack and roll angle commands to the lower control layer; and a lower generation unit configured to, in response to receiving the desired angle of attack and roll angle commands sent by the local tracking layer and in response to receiving aerodynamic parameters and physical state information, generate a numerical sequence of control surface deflection angles.

[0011] Thirdly, this application also provides a computer device, which includes a processor, a memory, and a computer program stored in the memory and executable by the processor, wherein when the computer program is executed by the processor, it implements the method described in any implementation of the first aspect above.

[0012] Fourthly, this application also provides a computer-readable storage medium storing a computer program, wherein when the computer program is executed by a processor, it implements the method described in any implementation of the first aspect above.

[0013] The various embodiments of this application have the following beneficial effects: First, by designing a heuristic evolutionary genetic algorithm (HEGA) to achieve global cooperative trajectory planning, multi-agent cluster cooperative trajectory optimization based on multi-agent reinforcement learning in some embodiments of this application can ensure obstacle avoidance, collision avoidance, and synchronous arrival for multiple aircraft. Second, by introducing reinforcement learning agents to dynamically adjust the MPC prediction step size and cost function weights, adaptive local optimization can be achieved. Next, a distributed model predictive control (DMPC) architecture is adopted, and safe cooperation is achieved through collision avoidance constraints and communication-maintaining constraints. Then, dynamic inverse control and PID control are integrated at the lower control layer, with a built-in constraint monitoring module to monitor boundaries such as dynamic pressure, thermal flux, and overload in real time. Furthermore, a hierarchical cooperative intelligent control framework is constructed, combining optimization algorithms, predictive control, and artificial intelligence technologies to achieve system self-organization, self-adaptation, and self-learning. Therefore, the cluster cooperative trajectory optimization method based on multi-agent reinforcement learning in this application can effectively integrate global planning and local control to achieve hierarchical cooperation. Attached Figure Description

[0014] The above and other features, advantages, and aspects of the embodiments of this application will become more apparent from the accompanying drawings and the following detailed description. Throughout the drawings, the same or similar reference numerals denote the same or similar elements. It should be understood that the drawings are schematic, and elements are not necessarily drawn to scale.

[0015] Figure 1 This is a flowchart of some embodiments of the cluster cooperative trajectory optimization method based on multi-agent reinforcement learning according to this application;

[0016] Figure 2 These are simulation diagrams of path planning for ten aircraft in different no-fly zones according to some embodiments of the cluster cooperative trajectory optimization method based on multi-agent reinforcement learning according to this application.

[0017] Figure 3 These are platform verification diagrams of some embodiments of the cluster cooperative trajectory optimization method based on multi-agent reinforcement learning according to this application;

[0018] Figure 4 These are schematic diagrams of some embodiments of the cluster cooperative trajectory optimization device based on multi-agent reinforcement learning according to this application;

[0019] Figure 5 This is a schematic diagram of the structure of a computer device suitable for implementing some embodiments of this application. Detailed Implementation

[0020] Embodiments of this application will now be described in more detail with reference to the accompanying drawings. While some embodiments of this application are shown in the drawings, it should be understood that this application can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this application. It should be understood that the drawings and embodiments of this application are for illustrative purposes only and are not intended to limit the scope of protection of this application.

[0021] It should also be noted that, for ease of description, only the parts relevant to the invention are shown in the accompanying drawings. Unless otherwise specified, the embodiments and features described herein can be combined with each other.

[0022] It should be noted that the concepts of "first" and "second" mentioned in this application are only used to distinguish different devices, modules or units, and are not used to limit the order of functions performed by these devices, modules or units or their interdependencies.

[0023] It should be noted that the terms "a" and "a plurality of" used in this application are illustrative rather than restrictive, and those skilled in the art should understand that, unless otherwise expressly indicated in the context, they should be understood as "one or more".

[0024] The names of the messages or information exchanged between multiple devices in the embodiments of this application are for illustrative purposes only and are not intended to limit the scope of these messages or information.

[0025] The present application will now be described in detail with reference to the accompanying drawings and embodiments.

[0026] Figure 1 A flowchart 100 is shown, illustrating some embodiments of a cluster cooperative trajectory optimization method based on multi-agent reinforcement learning according to this application. This cluster cooperative trajectory optimization method based on multi-agent reinforcement learning, applied to a cluster cooperative trajectory optimization system, includes the following steps:

[0027] Step 101: In response to receiving the coordinates of the task target point, the global environment map, and the initial state vector of the cluster, the global planning layer generates global reference trajectory information based on a preset heuristic evolutionary genetic algorithm, and sends the global reference trajectory information to the local tracking layer.

[0028] In some embodiments, the global planning layer, in response to receiving the coordinates of the task target point, the global environment map, and the initial state vector of the cluster, generates global reference trajectory information based on a preset heuristic evolutionary genetic algorithm, and sends the global reference trajectory information to the local tracking layer. The cluster cooperative trajectory optimization system includes a global planning layer, a local tracking layer, and a low-level control layer. Logically, this application can be divided into three core algorithm layers: a global planning layer, a local tracking layer, and a low-level control layer. Each layer interacts with data through shared memory or a message interface. Here, the global planning layer is the top-level decision-making unit of the cluster cooperative trajectory optimization system, responsible for handling long-term path optimization problems. The local tracking layer is the core computational unit of the cluster cooperative trajectory optimization system, responsible for handling short-term dynamic interference and trajectory correction. The low-level control layer is the instruction execution unit of the cluster cooperative trajectory optimization system, responsible for converting optimization instructions into specific control variables.

[0029] The mission target point coordinates can be the final three-dimensional spatial location the cluster needs to reach (including mission indicators such as landing speed and landing time difference). The global environment map can be digital map data containing a static no-fly zone mathematical model, terrain features, and mission airspace boundaries. The mission target point coordinates and the global environment map are usually issued by the superior command system or pre-stored in the mission database. They are received by the global planning layer before the simulation or actual mission begins. The cluster's initial state vector can be obtained through onboard sensors (such as GPS and inertial navigation) or input through system configuration parameters in the simulation environment.

[0030] As an example, the input data for the global planning layer can be: the coordinates of the mission target point, a global environment map (including a static no-fly zone mathematical model), and the initial state vector of the cluster. The core algorithm can be: running a heuristic evolutionary genetic algorithm, which does not rely on specific hardware instructions but iteratively calculates the fitness function and outputs global reference track information that satisfies time and space constraints. The output data can be: global reference track information (which can be a discrete point sequence). The input data for the local tracking layer can be: global reference track information and the cooperative state information of neighboring nodes. The core algorithm can be: using a coupled computation strategy of "reinforcement learning (RL) + model predictive control (MPC)," and the output data can be: extracting the first instruction of the control sequence, the desired angle of attack, and the desired bank angle. Here, the RL proxy module acts as a trained neural network function, taking the state error as input and outputting the hyperparameters of the MPC optimization problem. The MPC solver module, based on the parameters output by RL, constructs a quadratic or nonlinear programming model containing cooperative potential field constraints and calls a numerical optimization solver to calculate the optimal future control sequence. The input data for the bottom control layer can be: desired angle of attack and roll angle commands, current aerodynamic parameters and physical state of the aircraft; the core algorithm can be: constraint monitoring algorithm: based on physical equations (dynamic pressure, heat flow formulas) to pre-simulate the state at the next moment, and use SoftLimit (smoothing limit function) to numerically correct the input commands; the output data can be: the final numerical sequence of control surface deflection angles.

[0031] Here, the aforementioned preset heuristic evolutionary genetic algorithm may include: preset heuristic hybrid strategy, preset fitness function, preset elite selection algorithm, preset crossover algorithm, and preset mutation algorithm.

[0032] The preset heuristic hybrid strategy can be a hybrid strategy employing 80% heuristic rules and 20% historical experience transfer. The heuristic rules (ensuring feasibility) are as follows: a basic framework is generated based on "line-of-sight (LOS)" logic, connecting the starting and ending points to form a baseline. Based on the current position deviation, a basic control sequence is inversely derived using PID (proportional-integral-derivative) logic, and Gaussian noise is superimposed on it. To generate diversity. Historical experience transfer (accelerating convergence): A historical task database is established. When a new task is assigned, the cosine similarity between the start and end vectors of the new task and the historical tasks is calculated. The chromosome with the highest similarity is retrieved and injected into the current population. This ensures the feasibility of the initial individuals.

[0033] The preset fitness function is determined based on the coordinates of the task target point and the global environment map. The preset fitness function can be:

[0034] ,

[0035] in, Indicates fitness. Indicates the individual aircraft code. Indicates the weighting coefficient. This indicates the terminal position error (the deviation between the terminal position of the aircraft and the coordinates of the mission target point). , Indicates the actual position coordinates of the aircraft's terminal phase. Indicates the coordinates of the mission objective point. Indicates the weighting coefficient. This indicates a constraint violation (accumulating all degrees of violation of physical boundaries during the integration process). , Indicates the stagnation heat flux density. Indicates the weighting coefficient. This indicates a no-fly zone penalty (if the trajectory point falls into the no-fly zone cylinder defined in the global environment map, an exponential penalty is applied based on the depth of intrusion) (this is done by comparing the trajectory point with the no-fly zone model in the global environment map, calculating the depth of intrusion, and applying an exponential penalty). , This represents the weight matrix (used to adjust the influence of different directions or regions). This indicates the radius of the no-fly zone (or the boundary value of the no-fly zone). This indicates the distance to the trajectory boundary (the distance from the trajectory point to the center of the no-fly zone or the perpendicular distance from the trajectory point to the boundary of the no-fly zone). Indicates the weighting coefficient. Smoothness is represented by calculating the sum of squared differences between adjacent control commands. , Indicates time, Indicates in At any given moment (the action instructions generated by the reinforcement learning agent (including the executor of the global planning layer, local tracking layer, and low-level control layer) based on the new system feedback state), Indicates in Action instructions at any given moment (output by the reinforcement learning agent based on the current state).

[0036] Therefore, a penalty function can be used to construct the fitness function.

[0037] Therefore, this application constructs a multi-objective weighted fitness function that comprehensively considers terminal accuracy, path safety, physical constraints, control smoothness, and task efficiency. It innovatively introduces a progressive penalty mechanism and a reward-guided strategy, imposing gradual penalties for constraint violations while simultaneously rewarding excellent performance. Through carefully adjusted weight parameters, the algorithm reflects the priority differences among various objectives while maintaining a suitable order-of-magnitude balance, ensuring that it can evenly optimize all key performance indicators and efficiently search for high-quality solutions that meet various engineering requirements in a complex multi-objective space.

[0038] The preset elite selection algorithm can be: 30% elite retention + roulette wheel selection.

[0039] The default crossover algorithm can be: single-point crossover. Randomly select a crossover point k, and swap the data after index k in the two parent arrays.

[0040] The preset mutation algorithm can be: adaptive mutation. The mutation rate decreases linearly with the number of generations. , Indicates the rate of variation. This represents the current generation. In the iterative process of a genetic algorithm, it represents the current iteration count and is used to identify which stage of evolution the algorithm is currently in. The maximum number of generations is one of the preset algorithm termination conditions, representing the total number of iterations the algorithm is allowed to run. (This refers to the process of stacking selected genes.) Disturbance.

[0041] Therefore, this application designs an improved crossover and mutation mechanism in the genetic algorithm evolution process. The crossover operation employs a single-point crossover strategy, randomly selecting crossover points in the parent chromosomes for gene fragment exchange, promoting the combination and inheritance of high-quality genes. The mutation operation uses an adaptive mutation mechanism, with the mutation probability gradually decreasing with each generation. An initial mutation rate of 10% is set to enhance global exploration capabilities, and then a linear decay mechanism is used to gradually reduce the mutation rate, enhancing local exploitation capabilities in the later stages of the algorithm. This adaptive mechanism effectively balances the algorithm's exploration and exploitation capabilities, realizing a trajectory optimization process from coarse-grained to fine-grained.

[0042] The objective function of the aforementioned pre-defined heuristic evolutionary genetic algorithm can be:

[0043] ,

[0044] in, Represents the total cost function. Indicates the start time of flight ( The state at that time is the cluster's initial state vector. Indicates the end time of the flight. Indicates the path length (the length of the complete trajectory from the starting point to the task target point). Indicates control input, Indicates the energy weighting coefficient. The terminal state error penalty (obtained by calculating the Euclidean distance or deviation between the flight trajectory endpoint and the mission target point) is defined by the inequality constraints of the objective function of the pre-defined heuristic evolutionary genetic algorithm:

[0045] ,

[0046] in, Indicates dynamic pressure. , This represents the physically permissible limit. Indicates the stagnation heat flux density. This represents the limiting value of the stagnation point heat flux density. Indicates normal overload, This indicates the limit value of normal overload.

[0047] In practice, in response to receiving the coordinates of the mission objective point, the global environment map, and the initial state vector of the cluster, the aforementioned global planning layer can generate global reference track information based on a preset heuristic evolutionary genetic algorithm through the following steps:

[0048] The first step involves initializing the initial state vector of the cluster based on a pre-defined heuristic hybrid strategy to generate an initial aircraft population. This strategy includes heuristic rule generation and historical experience transfer. Specifically, heuristic rule generation accounts for 80% of the strategy, while historical experience transfer accounts for 20%. Heuristic rule generation can include: establishing a baseline: using the initial state vector as the starting point and the target coordinates as the ending point, connecting them to form a baseline; inversely deriving the control sequence: based on line-of-sight (LOS) logic, using PID logic to derive the basic control sequence based on the deviation between the current position and the baseline; and increasing diversity: superimposing Gaussian noise onto these basic sequences to generate a large number of "feasible" individuals that, while different, all point towards the target. Historical experience transfer can involve: retrieving successful cases similar to the current task's start and end points from a historical task database; and directly injecting these validated superior individuals into the current population to accelerate algorithm convergence.

[0049] Here, each "individual" (i.e., chromosome) in the initial aircraft swarm is represented in memory as a one-dimensional double-precision floating-point array. Its core information is the time series of control variables: a dual-sequence structure; the individual contains the control commands for the aircraft throughout its entire flight time; Information A: the desired angle of attack sequence (e.g., indices 0-999 store the angle of attack values ​​at different times); Information B: the desired bank angle sequence (e.g., indices 1000-1999 store the bank angle values ​​at the corresponding times). One chromosome represents a complete "flight plan." The global planning layer evolves these angle of attack and bank angle sequences to ultimately form global reference track information that satisfies the constraints.

[0050] The second step involves chromosome encoding the initial spacecraft population to generate a spacecraft-coded population. This coded population includes at least one spacecraft-coded individual. In practice, the global planning layer can employ a real-number encoded double-sequence chromosome structure to directly map the initial spacecraft population (spacecraft control sequences) to genes.

[0051] As an example, in computer memory, an individual is represented as a one-dimensional double-precision floating-point array. For instance, the total chromosome length is set to 2 × 1000 = 2000 (assuming a simulation step size of 0.1s and a total duration of 100s). The array layout is as follows: indices 0~999: storing the angle of attack sequence. The unit is degrees (°); index 1000~1999: stores the tilt angle sequence. The unit is degrees (°).

[0052] The third step involves performing the following iterative steps based on the aircraft's coded population:

[0053] In the first iteration step, based on a preset fitness function, the fitness of each aircraft coded individual in the aircraft coded population is determined, and a fitness set is obtained.

[0054] The second iteration step involves selecting individual aircraft codes from the aircraft coding population based on the fitness set and a pre-defined elite selection algorithm, thus forming the aircraft selection population.

[0055] The third iteration step, in response to determining that the selected population of the aircraft meets a preset termination condition, designates the selected population as the global reference trajectory information. The preset termination condition can be: reaching a maximum iteration limit or fitness function convergence. The maximum iteration limit can be: a preset upper limit on the number of generations (e.g., 100 or 200 generations); when the iteration reaches this value, the calculation stops and the current best individual is output. Fitness function convergence can be: when the optimal fitness value in the population no longer significantly improves over several consecutive generations (e.g., 10 consecutive generations), i.e., the rate of change is lower than a preset threshold (e.g., the preset threshold could be 10). -6 The iteration is terminated prematurely, assuming that a globally approximate optimal solution has been found.

[0056] Optionally, the global planning layer may also perform the following determination steps in response to determining that the selected population of the aircraft does not meet the above-mentioned preset termination conditions:

[0057] The first step involves cross-processing the selected aircraft population based on a preset cross-processing algorithm to generate an aircraft cross-processing population.

[0058] The second determination step involves adaptively mutating the crossover population of the aircraft based on a preset mutation algorithm to generate a mutated population of the aircraft.

[0059] The third determination step involves using the aircraft variant population as the aircraft coding population for repeating the above iterative steps.

[0060] Therefore, the global planning layer is responsible for generating a globally optimal or suboptimal cooperative reference track based on global information at the start of the mission. This layer employs a heuristic evolutionary genetic algorithm as its core planning method.

[0061] Therefore, to address the issues of low efficiency and poor coordination in traditional planning algorithms, this application designs a heuristic evolutionary genetic algorithm (HEGA) to achieve global collaborative trajectory planning, ensuring that multiple aircraft can bypass obstacles, avoid collisions, and arrive synchronously.

[0062] Therefore, this application designs a global trajectory planning algorithm that integrates heuristic rules and adaptive genetic operations. The trajectory sets of multiple aircraft are integrated into a single chromosome through a cooperative coding mechanism, and the control variables are directly represented by a double-sequence structure encoded with real numbers. An attention-driven heuristic crossover operator and an environment-adaptive mutation mechanism are innovatively introduced, enabling automatic adjustment of the search strategy based on dynamic environmental characteristics. This algorithm integrates random key encoding bias and learning-driven adaptive adjustment on the basis of traditional genetic mechanisms. Through an elite preservation strategy and a hybrid initialization method, it significantly accelerates the convergence process while ensuring population diversity, solving the global optimization problem of multi-aircraft cooperative trajectory planning under multiple constraints. Simulation diagrams of path planning for ten aircraft under different no-fly zones are shown below. Figure 2 As shown. Figure 2 Simulation diagrams of path planning for ten aircraft in different no-fly zones are shown, illustrating some embodiments of the cluster cooperative trajectory optimization method based on multi-agent reinforcement learning according to this application.

[0063] Therefore, the HEGA algorithm demonstrates excellent global search capabilities in complex, multi-constraint environments. The planning process can complete the planning of ten flight paths in approximately 10 minutes, improving efficiency by about 40% compared to traditional methods. The generated trajectories exhibit high smoothness and reasonable obstacle avoidance behavior; all planned paths effectively avoid no-fly zones, meeting safety distance and spatial constraints, thus laying a solid foundation for subsequent path tracking and cooperative control. Furthermore, the multi-objective integrated fitness model, through a carefully designed weight allocation and penalty mechanism, achieves balanced optimization of multiple key indicators such as terminal accuracy, path safety, physical constraints, and control smoothness. While maintaining high accuracy, the system also considers trajectory smoothness, energy efficiency, and task timeliness, resulting in a significant improvement in overall performance.

[0064] Step 102: In response to receiving global reference track information from the global planning layer and node cooperative status information, the local tracking layer generates desired angle of attack and bank angle commands and sends the desired angle of attack and bank angle commands to the lower control layer.

[0065] In some embodiments, the local tracking layer, in response to receiving global reference track information from the global planning layer and in response to receiving node cooperative status information, generates desired angle of attack and bank angle commands, and sends these commands to the underlying control layer. The node cooperative status information may be information such as the position, velocity, and estimated arrival time of neighboring nodes (other aircraft) in the cluster, used to ensure that multiple aircraft do not collide and arrive simultaneously / in sequence. Here, the node cooperative status information can be obtained by receiving status messages sent by neighboring nodes through a wireless data link between aircraft.

[0066] Optionally, the local tracking layer can collect the local state (position, velocity) and receive neighbor states via data links to construct a local cooperative situation map. Here, the local cooperative situation map can be a data structure or spatial model built by the aircraft in memory. Centered on the local aircraft, the local cooperative situation map labels the relative positions, relative velocities, and potential collision risk areas of surrounding neighbor aircraft. (This can be understood as the aircraft's "digital radar vision.") After receiving neighbor states, the local tracking layer constructs the local cooperative situation map using a pre-defined algorithm (e.g., a pre-defined graph theory algorithm), and then extracts cooperative errors and constraints from this map (e.g., using cooperative error extraction algorithms and cooperative constraint extraction algorithms) as input features for the reinforcement learning model.

[0067] In practice, in response to receiving global reference track information from the global planning layer and in response to receiving node cooperative state information, the aforementioned local tracking layer can generate desired angle of attack and bank angle commands through the following steps:

[0068] The first step is to determine the system state deviation based on the aforementioned global reference track information. In practice, firstly, the aforementioned local tracking layer can collect the current actual state information. The current actual state information can be information including position, velocity, and attitude collected in real time by onboard sensors (GPS, IMU, etc.). Secondly, the aforementioned local tracking layer can determine the system state deviation as the difference between the corresponding values ​​of the aforementioned current actual state information and the aforementioned global reference track information.

[0069] Therefore, the aforementioned local tracking layer can collect the "machine state" in real time through onboard sensors (GPS, IMU, etc.) and subtract the value of the global reference track information to obtain the system state deviation. Here, the system state deviation can be the difference between the actual state (position, velocity, attitude) of the aircraft at the current moment and the ideal state of the "global reference track information" given by the global planning layer at the corresponding moment. The system state deviation and node cooperative state information are the core inputs of the local tracking layer, used to achieve high-precision trajectory tracking and multi-aircraft synchronization.

[0070] The second step involves inputting the aforementioned system state deviation and node cooperative state information into a pre-trained reinforcement learning model to obtain hyperparameter information. The pre-trained reinforcement learning model can be a neural network model employing a Deep Q-Network (DQN). The input to the reinforcement learning model can be: the system state vector perceived by the model in real time, specifically including system state deviation, local cooperative situation map, and node cooperative state information. The output of the reinforcement learning model can be: hyperparameter information. Here, hyperparameter information can include: weight matrix adjustment coefficients (used to dynamically change the weight allocation of control energy or tracking accuracy in the cost function), prediction time domain length (although the formula of the prediction planning model itself describes a single-step state transition, the reinforcement learning model will determine how many such steps the MPC will predict), and control deviation compensation (a correction term directly applied to the optimal control sequence solution).

[0071] Therefore, the reinforcement learning agent, acting as a "parameter scheduler," can dynamically adjust MPC parameters. The inputs to RL can be trajectory tracking error (system state deviation), environmental disturbance intensity (local cooperative situation map), and relative distance (node ​​cooperative state information), while the output can be the prediction time-domain length of the MPC. Control deviation compensation amount and weight matrix adjustment coefficient As an example, when the agent detects a strong disturbance or target maneuver, the output instruction shortens the prediction time domain (e.g., from...). =Reducing from 50 to 10), sacrificing long-term optimality in exchange for increased single-step computation speed (frequency increased from 10Hz to 50Hz), thus achieving agile response.

[0072] The third step is to construct the planning objective function based on the hyperparameter information described above. In practice, based on the hyperparameter information, a quadratic programming or nonlinear programming objective function incorporating the synergistic potential field is constructed:

[0073] ,

[0074] in, Represents the MPC cost function. Indicates the serial number. Indicates the agent's ID. This represents the current intelligent agent (including the execution entity of the global planning layer, local tracking layer, and low-level control layer). This indicates the length of the prediction time domain included in the hyperparameter information. Indicates the first The state vector at time t, Represents the desired state vector. Indicates the first The control input at any given time (with angle of attack and roll angle as the independent variables to be solved) ), This indicates that the hyperparameter information includes the control deviation compensation amount. This indicates that the hyperparameter information includes the weight matrix adjustment coefficients. Represents intelligent agents The neighborhood set, Indicates weight, This represents a non-negative monotonically increasing function. Represents intelligent agents In the The state vector at time t, Represents intelligent agents In the The state vector at time t, Indicates the first The state vector at time t, The order identifier of the norm.

[0075] The fourth step is to construct a predictive programming model. The predictive programming model can be:

[0076] ,

[0077] in, This represents the predicted state vector for the next time step. This represents the state vector at the current moment. Indicates the current moment. Indicates the discrete time step. , , , This represents the four slope values ​​(rates of change of state) in the Runge-Kutta algorithm, where, , , , The position coordinates represent the X, Y, and Z axes of the aircraft in three-dimensional space (either inertial or mission coordinate system; the inertial coordinate system can be the absolute reference describing the aircraft's actual physical motion and is a prerequisite for the dynamic equations to hold; the mission coordinate system can be the relative reference describing the aircraft's mission geometry and is a mathematical tool for simplifying constraint handling and collaborative problem modeling). , , Represents the velocity components (the velocity components of the aircraft on the X, Y, and Z axes in three-dimensional space). This indicates the mass of the aircraft.

[0078] Therefore, the fourth-order Runge-Kutta method (RK4) can be used for numerical integration discretization.

[0079] The fifth step is to determine the optimal control sequence based on the above-mentioned planning objective function and predictive planning model. In practice, the above-mentioned local tracking layer can call a numerical optimization solver to obtain the set of control inputs that minimizes the MPC cost function as the optimal control sequence.

[0080] Here, the optimal control sequence can be .

[0081] The sixth step is to determine the desired angle of attack and bank angle commands based on the above optimal control sequence. In practice, the local tracking layer can use the first command item in the above optimal control sequence as the desired angle of attack and bank angle commands at the current moment.

[0082] Therefore, the local tracking layer is responsible for real-time dynamic optimization and tracking based on global reference track information, and adopts a strategy that combines MPC and RL.

[0083] Therefore, to address the issue of insufficient adaptability of MPC in dynamic environments, this application introduces a reinforcement learning agent to dynamically adjust the MPC prediction step size and cost function weights to achieve adaptive local optimization. To address the multi-agent cooperative conflict problem, this application adopts a distributed model predictive control (DMPC) architecture, achieving safe cooperation through collision avoidance constraints and communication-maintaining constraints.

[0084] Therefore, this application creatively integrates reinforcement learning agents with model predictive control (MPC) to construct a predictive control closed loop with online self-learning capabilities. The reinforcement learning agent, through an Actor-Critic (policy-value network) framework or a deep Q-network (combining deep learning and reinforcement learning methods), models the MPC dynamic step size adjustment problem as a Markov decision process, sensing the system state vector in real time (including system state deviation, local cooperative situation map, and node cooperative state information), and dynamically adjusting the cost function weights, control deviation compensation, and prediction time domain length of the MPC. This fusion mechanism overcomes the shortcomings of traditional MPC's fixed prediction step size in adapting to complex environments, achieving intelligent decision-making that adaptively adjusts the control strategy based on real-time situation. Furthermore, for the multi-agent cooperative conflict problem, this application adopts a distributed model predictive control architecture as the core control framework. Each aircraft independently operates its own MPC controller, ensuring a preset safe distance between any two aircraft and reliable communication connections between cluster members by introducing cooperative-related terms (including collision avoidance constraints and communication maintenance constraints) into the individual cost function. By coordinating the control strategies of each aircraft through a consensus algorithm, the system achieves optimization and consistency of the overall swarm objective while maintaining the autonomy of each aircraft, effectively solving the conflict avoidance and cooperative optimization problems in multi-agent systems. Then, addressing the communication constraints in swarm cooperation, this application innovatively introduces communication maintenance constraints and a limited information interaction mechanism within the DMPC framework. By optimizing the communication topology and data transmission strategy, communication overhead is minimized while ensuring cooperative performance. The system supports cooperative control under constrained communication conditions; even in the event of communication delays or interruptions, it can still maintain basic cooperative capabilities through local prediction and historical information, significantly enhancing the system's robustness in complex electromagnetic environments.

[0085] Therefore, the fusion strategy of MPC and reinforcement learning enables the trajectory tracking algorithm to achieve high-precision terminal guidance under long-term, high-speed flight conditions. The final landing point of the aircraft is 0.187 km from the target center, far less than the target area radius of 5 km, improving the hit accuracy by approximately 60%. Control errors are minimal, and attitude changes are smooth, demonstrating excellent reliability and practical value. Furthermore, the reinforcement learning agent dynamically adjusts MPC parameters according to the real-time environment, ensuring stable output even under complex conditions such as changing lighting, occlusion, sensor noise, and model uncertainty. Even when inertial signals are interfered with or images are blurred, the system can still automatically select the optimal control strategy through an adaptive weighting mechanism, ensuring the continuity and reliability of control results. Simultaneously, the distributed model predictive control architecture combined with a consensus algorithm enables the aircraft swarm to maintain excellent collaborative performance in high-speed dynamic environments. Test results show that the system can effectively maintain formation geometry, achieve multi-aircraft collaboration, significantly reduce landing time differences, and improve the collaborative hit rate by approximately 35%, fully demonstrating the overall advantages of swarm collaboration.

[0086] Step 103: In response to receiving the desired angle of attack and roll angle commands from the local tracking layer, and in response to receiving aerodynamic parameters and physical state information, the bottom control layer generates a numerical sequence of control surface deflection angles.

[0087] In some embodiments, the aforementioned lower control layer generates a numerical sequence of control surface deflection angles in response to receiving desired angle of attack and roll commands from the aforementioned local tracking layer, and in response to receiving aerodynamic parameters and physical state information. The aerodynamic parameters can be pre-defined parameters describing the force characteristics of the aircraft, and may include aerodynamic efficiency models, lift coefficients, drag coefficients, etc. The physical state information can be the current kinematic and dynamic state of the aircraft, specifically including dynamic pressure, stagnation heat flux density, normal overload, and current flight speed and altitude. Some parameters in the physical state information that cannot be directly measured (such as stagnation heat flux density) can be calculated online based on the measured physical state information using preset physical equations (heat flux formulas).

[0088] In practice, in response to receiving the desired angle of attack and roll angle commands from the local tracking layer, and in response to receiving aerodynamic parameters and physical state information, the aforementioned lower control layer can generate a numerical sequence of control surface deflection angles through the following steps:

[0089] The first step is to perform motion conversion processing on the aforementioned desired angle of attack and heel angle commands to generate physical motion information. In practice, the aforementioned lower-level control layer can perform motion conversion processing on the aforementioned desired angle of attack and heel angle commands to generate physical motion information through the following steps:

[0090] The first sub-step involves determining the desired torque command (based on dynamic inverse calculation) based on the aforementioned physical state information, the desired angle of attack, and the roll angle commands. In practice, the aforementioned lower-level control layer can determine the desired torque command based on the aforementioned physical state information, the desired angle of attack, and the roll angle commands using a preset inverse algorithm. For example, the preset inverse algorithm could be a dynamic inverse algorithm. Therefore, the lower-level control layer first calculates the desired three-axis control torque required to maintain the attitude based on the received desired angle of attack and desired roll angle, combined with the currently acquired physical state information (such as angular velocity, altitude, etc.), using the dynamic inverse algorithm. This achieves the physical meaning of determining the required pitching or plunging torque for the aircraft to achieve the target angle of attack.

[0091] The second sub-step involves constructing an aerodynamic efficiency matrix based on the aforementioned aerodynamic parameters. In practice, the lower-level control layer retrieves preset aerodynamic parameters (aerodynamic efficiency model) from memory and constructs a control allocation matrix according to the current dynamic pressure and Mach number. Therefore, the calculation logic at this point is: this matrix quantitatively describes the torque increment generated by each control surface (aileron, horizontal stabilizer, rudder, etc.) when deflected by a unit angle under the current flight condition.

[0092] The third sub-step involves mapping the aforementioned desired torque commands to generate a numerical sequence of control surface deflection angles (control allocation solution (control surface command generation)). In practice, the aforementioned lower-level control layer maps the total desired torque (desired three-axis control torque) to each specific aerodynamic control surface through the control allocation module, obtaining a numerical sequence of control surface deflection angles for each aerodynamic control surface. In practice, this can be achieved by solving a system of linear equations (or a constrained optimization problem) to allocate the total torque requirement to the aileron, horizontal stabilizer, and rudder.

[0093] The fourth sub-step involves mapping the aforementioned control surface deflection angle numerical sequence to generate a PWM (Pulse Width Modulation) signal (PWM signal mapping). In practice, the underlying control layer maps the calculated control surface deflection angle values ​​(digital signals) to corresponding PWM (Pulse Width Modulation) voltage signals based on the physical characteristic curves of the servo actuator. In practice, this can be achieved by linearly mapping the angle range (e.g., -25° to +25°) to the PWM duty cycle range (e.g., 5% to 10%).

[0094] The fifth sub-step involves determining the aforementioned desired torque command, aerodynamic efficiency matrix, control surface deflection angle numerical sequence, and PWM pulse width modulation signal as physical motion information.

[0095] Optionally, the actuator is driven to move. In practice, the aforementioned lower-level control layer outputs a PWM signal to the servo actuator. The servo actuator drives the control surface to undergo actual physical deflection based on the received electrical signal, thereby changing the aerodynamic forces acting on the aircraft and achieving trajectory tracking.

[0096] Therefore, the desired torque command can be calculated into the deflection angle of each aerodynamic control surface (such as aileron and rudder) by the control distribution module according to the aerodynamic efficiency model. The flight control computer (the aforementioned underlying control layer) converts the angle into a PWM (pulse width modulation) voltage signal to drive the servo actuator to move.

[0097] The second step is to determine the preset constraints. These preset constraints can be:

[0098] ,

[0099] in, Indicates atmospheric density. Indicates flight speed. Indicates the stagnation radius of the aircraft. This indicates the lift force experienced by the aircraft. Indicates resistance. Indicates the mass of the aircraft. Represents gravitational acceleration. Indicates kilopascals, It represents kilowatts per square meter.

[0100] The third step involves, in response to the determination that the aforementioned physical motion information does not meet the aforementioned preset constraints, performing smoothing correction processing on the aforementioned desired angle of attack and roll angle commands based on a preset smoothing correction formula to generate smoothing correction commands. The preset smoothing correction formula may be:

[0101] ,

[0102] in, This indicates a smooth correction instruction. This indicates the allowable boundary for the current heat flux / dynamic pressure back calculation. This indicates the desired angle of attack and yaw angle commands.

[0103] Therefore, when the predicted state shows a tendency to exceed limits, soft limiting is performed using the hyperbolic tangent function (Tanh) from the C++ standard library to avoid jitter caused by instruction truncation. As an example, The raw instructions for calculating MPC (e.g., 25°); This is the allowable boundary (e.g., 20°) calculated based on the current heat flux / dynamic pressure. The function will output approximately 19.8° for a smooth transition.

[0104] The fourth step involves performing dynamic inverse calculations on the smoothing correction commands based on the aforementioned aerodynamic parameters and physical state information to generate a numerical sequence of control surface deflection angles. In practice, based on the aforementioned aerodynamic parameters and physical state information, the underlying control layer uses a dynamic inverse control algorithm to perform dynamic inverse calculations on the smoothing correction commands to generate a numerical sequence of control surface deflection angles.

[0105] Therefore, the underlying control layer converts commands into physical actions through dynamic inversion and PID control, and has a built-in high-priority constraint monitoring module.

[0106] Therefore, to address the challenges of handling complex physical constraints, this application integrates dynamic inverse control and PID control at the bottom control layer, and incorporates a constraint monitoring module to monitor dynamic pressure, heat flow, overload, and other boundaries in real time.

[0107] Therefore, at the underlying control execution layer, this application integrates a hybrid strategy of dynamic inverse control and PID control, and innovatively incorporates a high-priority constraint monitoring module. Based on a precise physical mathematical model, this module performs forward prediction and real-time calculation of the flight state, proactively safeguarding critical physical boundaries such as dynamic pressure, heat flux, and overload. When it predicts that a control command may reach the 150 kPa dynamic pressure limit, the 1500 kW / m² heat flux extreme value, or the 10g overload constraint, it will autonomously soften and limit the command to ensure the absolute safety of the aircraft. This proactive safety mechanism significantly reduces the risk of the system under high-speed flight conditions.

[0108] Therefore, the underlying constraint monitoring module, based on the forward prediction capability of an accurate physical model, can identify potential risks in advance and autonomously adjust commands. Experiments show that the system can strictly control key parameters such as dynamic pressure, heat flux, and overload within safe ranges under various extreme conditions, fundamentally ensuring the safety of the aircraft.

[0109] Optionally, the above-mentioned cluster collaborative trajectory optimization system waits for the next clock cycle and executes step 102 again until the task ends.

[0110] Therefore, to address the problem of insufficient generalization ability of learning methods, this application constructs a hierarchical collaborative intelligent control framework, which combines optimization algorithms, predictive control and artificial intelligence technologies to achieve system self-organization, self-adaptation and self-learning.

[0111] Therefore, this application proposes a three-layer collaborative control framework consisting of global planning, local tracking, and low-level execution, clearly defining the decision space and time scale. The global planning layer is responsible for generating collaborative reference tracks, the local tracking layer performs real-time dynamic optimization, and the low-level execution layer implements precise physical control. Each layer interacts efficiently through a unified interface and data flow, forming a complete self-organizing, adaptive, and self-learning system. This architecture ensures both the global optimality of the system and the real-time performance and accuracy of local control, significantly improving the overall performance of the system in complex dynamic environments. Furthermore, this application constructs an unsupervised training mechanism entirely based on a simulation platform. The system autonomously learns the optimal strategy through interaction with the environment, without the need for manually labeled data. The training process is based on actual physical models and environmental data, utilizing multiple rounds of simulation testing to optimize algorithm parameters and network weights. This mechanism not only significantly reduces deployment costs but also endows the model with strong generalization capabilities, enabling it to adapt to different sensor configurations, environmental conditions, and task requirements, achieving true autonomous learning and cross-platform transfer. Figure 3 As shown, Figure 3 Platform verification diagrams are shown for some embodiments of the cluster cooperative trajectory optimization method based on multi-agent reinforcement learning according to this application. The flight status sphere shows the cooperative motion trajectories of a group of multiple aircraft in a highly dynamic environment. The red line trajectory represents the flight path of the aircraft, avoiding no-fly zones.

[0112] Therefore, this application employs a layered architecture and parallel computing mechanism to significantly reduce the system's computational load and improve inference speed by approximately 50%. The trajectory tracking algorithm can complete the entire flight mission tracking within approximately 30 to 40 minutes, with stable overall tracking and timely control response, meeting the real-time requirements of highly dynamic environments. Furthermore, the modular design and parameter sharing strategy enable the system to quickly adapt to different hardware platforms and sensor configurations. Tests show that the system can run in real-time on embedded platforms such as NVIDIA Jetson (embedded system platform), supports the ROS (Robot Operating System) communication protocol and standard API (Application Programming Interface) interfaces, facilitating integration with third-party systems and greatly reducing the deployment threshold. Meanwhile, this system features a simple structure, low power consumption, high accuracy, strong robustness, and real-time performance, making it widely applicable in fields such as autonomous vehicle localization, UAV trajectory calculation, indoor navigation of mobile robots, and spatial tracking in augmented reality and virtual reality systems.

[0113] In civilian applications, it is particularly suitable for scenarios such as drone logistics swarms, collaborative environmental monitoring, and urban air traffic management, and has significant application value and industrialization potential.

[0114] The various embodiments of this application have the following beneficial effects: First, by designing a heuristic evolutionary genetic algorithm (HEGA) to achieve global cooperative trajectory planning, multi-agent cluster cooperative trajectory optimization based on multi-agent reinforcement learning in some embodiments of this application can ensure obstacle avoidance, collision avoidance, and synchronous arrival for multiple aircraft. Second, by introducing reinforcement learning agents to dynamically adjust the MPC prediction step size and cost function weights, adaptive local optimization can be achieved. Next, a distributed model predictive control (DMPC) architecture is adopted, and safe cooperation is achieved through collision avoidance constraints and communication-maintaining constraints. Then, dynamic inverse control and PID control are integrated at the lower control layer, with a built-in constraint monitoring module to monitor boundaries such as dynamic pressure, thermal flux, and overload in real time. Furthermore, a hierarchical cooperative intelligent control framework is constructed, combining optimization algorithms, predictive control, and artificial intelligence technologies to achieve system self-organization, self-adaptation, and self-learning. Therefore, the cluster cooperative trajectory optimization method based on multi-agent reinforcement learning in this application can effectively integrate global planning and local control to achieve hierarchical cooperation.

[0115] Further reference Figure 4 As an implementation of the methods shown in the above figures, this disclosure provides some embodiments of a cluster cooperative trajectory optimization device based on multi-agent reinforcement learning. These embodiments of the cluster cooperative trajectory optimization device based on multi-agent reinforcement learning are similar to... Figure 1 Corresponding to the method embodiments shown, this cluster cooperative trajectory optimization device based on multi-agent reinforcement learning can be specifically applied to various electronic devices.

[0116] like Figure 4 As shown, a cluster cooperative trajectory optimization device 400 based on multi-agent reinforcement learning in some embodiments includes: a global generation unit 401, a local generation unit 402, and a bottom-level generation unit 403. The global generation unit 401 is configured such that, in response to receiving the coordinates of the mission target point, a global environment map, and the cluster initial state vector, the global planning layer generates global reference trajectory information based on a preset heuristic evolutionary genetic algorithm and sends the global reference trajectory information to the local tracking layer. The local generation unit 402 is configured such that, in response to receiving the global reference trajectory information sent by the global planning layer and in response to receiving node cooperative state information, the local tracking layer generates desired angle of attack and bank angle commands and sends the desired angle of attack and bank angle commands to the bottom-level control layer. The bottom-level generation unit 403 is configured such that, in response to receiving the desired angle of attack and bank angle commands sent by the local tracking layer and in response to receiving aerodynamic parameters and physical state information, the bottom-level control layer generates a numerical sequence of control surface deflection angles.

[0117] It is understandable that the units and reference units described in the multi-agent reinforcement learning-based cluster cooperative trajectory optimization device 400 are related to... Figure 1The steps described in the method correspond to each other. Therefore, the operations, features, and beneficial effects described above for the method also apply to the cluster cooperative trajectory optimization device 400 based on multi-agent reinforcement learning and the units contained therein, and will not be repeated here.

[0118] This application also provides a computer device 500. For example... Figure 5 As shown, the computer device 500 includes a bus 501, a processor 502, a memory 503, and a communication interface 504. The processor 502, the memory 503, and the communication interface 504 communicate with each other via the bus 501. The computer device 500 can be a server or a terminal device. It should be understood that this application does not limit the number of processors and memories in the computer device 500.

[0119] Bus 501 can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of representation, Figure 5 The bus 501 may be represented by a single line, but this does not mean that there is only one bus or one type of bus. The bus 501 may include a path for transmitting information between various components of the computer device 500 (e.g., memory 503, processor 502, communication interface 504).

[0120] Processor 502 may include any one or more processors such as a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP).

[0121] Memory 503 may include volatile memory, such as random access memory (RAM). Memory 503 may also include non-volatile memory, such as read-only memory (ROM), flash memory, hard disk drive (HDD), or solid state drive (SSD).

[0122] The memory 503 stores executable program code, and the processor 502 executes this executable program code to implement the functions of the aforementioned global generation unit, local generation unit, and low-level generation unit, thereby realizing the aforementioned cluster cooperative trajectory optimization method based on multi-agent reinforcement learning. That is, the memory 503 stores instructions for executing the aforementioned cluster cooperative trajectory optimization method based on multi-agent reinforcement learning.

[0123] The communication interface 504 uses transceiver modules such as, but not limited to, network interface cards and transceivers to enable communication between the computer device 500 and other devices or communication networks.

[0124] This application also provides a chip, which includes a processor and a data interface. The processor reads instructions stored in the memory through the data interface to execute the above-described cluster cooperative trajectory optimization method based on multi-agent reinforcement learning.

[0125] This application also provides a computer-readable storage medium. The computer-readable storage medium can be any available medium that a computing device can store, or a data storage device such as a data center containing one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state drive). The computer-readable storage medium includes instructions that instruct the computing device to execute the aforementioned cluster cooperative trajectory optimization method based on multi-agent reinforcement learning.

[0126] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0127] The above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the protection scope of the technical solutions of the embodiments of this application.

Claims

1. A cluster cooperative trajectory optimization method based on multi-agent reinforcement learning, applied to a cluster cooperative trajectory optimization system, characterized in that, The cluster collaborative trajectory optimization system includes: a global planning layer, a local tracking layer, and a low-level control layer, including: In response to receiving the coordinates of the task target point, the global environment map, and the cluster initial state vector, the global planning layer generates global reference trajectory information based on a preset heuristic evolutionary genetic algorithm, and sends the global reference trajectory information to the local tracking layer. The local tracking layer, in response to receiving global reference track information sent by the global planning layer and in response to receiving node cooperative status information, generates desired angle of attack and bank angle commands, and sends the desired angle of attack and bank angle commands to the underlying control layer. The underlying control layer generates a numerical sequence of control surface deflection angles in response to receiving the desired angle of attack and roll angle commands sent by the local tracking layer, and in response to receiving aerodynamic parameters and physical state information.

2. The cluster cooperative trajectory optimization method based on multi-agent reinforcement learning according to claim 1, characterized in that, The preset heuristic evolutionary genetic algorithm includes: a preset heuristic hybrid strategy, a preset fitness function, a preset elite selection algorithm, a preset crossover algorithm, and a preset mutation algorithm; And in response to receiving the coordinates of the mission target point, the global environment map, and the initial state vector of the cluster, the system generates global reference track information based on a preset heuristic evolutionary genetic algorithm, including: Based on a preset heuristic hybrid strategy, the initial state vector of the cluster is initialized to generate an initial aircraft population. The initial spacecraft population is subjected to chromosome encoding processing to generate a spacecraft-coded population, wherein the spacecraft-coded population includes at least one spacecraft-coded individual; Based on the aircraft-coded population, the following iterative steps are performed: Based on a preset fitness function, the fitness of each aircraft coded individual in the aircraft coded population is determined to obtain a fitness set. The preset fitness function is determined based on the coordinates of the mission target point and the global environment map. Based on fitness sets and a pre-defined elite selection algorithm, each aircraft code individual is selected from the aircraft coding population to form the aircraft selection population. In response to the determination that the selected aircraft population meets the preset termination conditions, the selected aircraft population is determined as the global reference track information.

3. The cluster cooperative trajectory optimization method based on multi-agent reinforcement learning according to claim 2, characterized in that, The method further includes: In response to the determination that the selected population of the aircraft does not meet the preset termination condition, the following determination steps are performed: Based on a preset crossover algorithm, the selected population of aircraft is crossover processed to generate a crossover population of aircraft. Based on a preset mutation algorithm, adaptive mutation processing is performed on the crossover population of aircraft to generate a mutated population of aircraft. The aircraft variant population is used as the aircraft coding population for repeating the iterative steps.

4. The cluster cooperative trajectory optimization method based on multi-agent reinforcement learning according to claim 2, characterized in that, The preset fitness function included in the preset heuristic evolutionary genetic algorithm is: ， in, Indicates fitness. Indicates the aircraft's coded individual, Indicates the weighting coefficient. Indicates the terminal position error. , This indicates the actual position coordinates of the aircraft's terminal phase. Indicates the coordinates of the mission objective point. Indicates the weighting coefficient. Indicates a violation of the constraint. , Indicates the stagnation heat flux density. Indicates the weighting coefficient. Indicates no-fly zone penalties. , Represents the weight matrix. Indicates the radius of the no-fly zone. Indicates the distance to the trajectory boundary. Indicates the weighting coefficient. Indicates smoothness, , Indicates time, Indicates in Momentary action commands Indicates in Action instructions at any given moment; The objective function of the preset heuristic evolutionary genetic algorithm is: ， in, Represents the total cost function. Indicates the start time of the flight. Indicates the end time of the flight. Indicates the path length. Indicates control input, Indicates the energy weighting coefficient. The terminal state error penalty is represented by the inequality constraint of the objective function of the pre-defined heuristic evolutionary genetic algorithm: ， in, Indicates dynamic pressure. , This represents the physically permissible limit. Indicates the stagnation heat flux density. This represents the limiting value of the stagnation point heat flux density. Indicates normal overload, This indicates the limit value of normal overload.

5. The cluster cooperative trajectory optimization method based on multi-agent reinforcement learning according to claim 1, characterized in that, The step of generating desired angle of attack and bank angle commands in response to receiving global reference track information from the global planning layer and in response to receiving node cooperative state information includes: Based on the global reference track information, the system state deviation is determined; The system state deviation and the node collaborative state information are input into a pre-trained reinforcement learning model to obtain hyperparameter information; Based on the hyperparameter information, a planning objective function is constructed; Construct a predictive planning model; Based on the planning objective function and the predictive planning model, the optimal control sequence is determined; Based on the optimal control sequence, the desired angle of attack and tilt angle commands are determined.

6. The cluster cooperative trajectory optimization method based on multi-agent reinforcement learning according to claim 5, characterized in that, The predictive programming model is: ， in, This represents the predicted state vector for the next time step. This represents the state vector at the current moment. Indicates the current moment. Indicates the discrete time step. , , , Represents the four slope values ​​in the Runge-Kutta algorithm, where, , , , Indicates position coordinates, , , , Represents the velocity component. This indicates the mass of the aircraft.

7. The cluster cooperative trajectory optimization method based on multi-agent reinforcement learning according to claim 1, characterized in that, The step of generating a numerical sequence of control surface deflection angles in response to receiving the desired angle of attack and roll angle commands from the local tracking layer, and in response to receiving aerodynamic parameters and physical state information, includes: The desired angle of attack and roll angle commands are processed for motion conversion to generate physical motion information; Define the preset constraints; In response to determining that the physical motion information does not meet the preset constraint conditions, the desired angle of attack and roll angle commands are smoothed and corrected based on a preset smoothing correction formula to generate smoothing correction commands; Based on the aerodynamic parameters and physical state information, the smoothing correction command is dynamically inversely calculated to generate a numerical sequence of control surface deflection angles.

8. The cluster cooperative trajectory optimization method based on multi-agent reinforcement learning according to claim 7, characterized in that, The preset constraint is: ， in, Indicates atmospheric density. Indicates flight speed. Indicates the stagnation radius of the aircraft. This indicates the lift force experienced by the aircraft. Indicates resistance. Indicates the mass of the aircraft. Represents gravitational acceleration. Indicates kilopascals, It represents kilowatts per square meter; The preset smoothing correction formula is: ， in, This indicates a smooth correction instruction. This indicates the allowable boundary for the current heat flux / dynamic pressure back calculation. This indicates the desired angle of attack and yaw angle commands.

9. A cluster cooperative trajectory optimization device based on multi-agent reinforcement learning, characterized in that, include: The global generation unit is configured to, in response to the global planning layer receiving the coordinates of the task target point, the global environment map, and the cluster initial state vector, generate global reference trajectory information based on a preset heuristic evolutionary genetic algorithm, and send the global reference trajectory information to the local tracking layer. The local generation unit is configured such that, in response to receiving global reference track information sent by the global planning layer and in response to receiving node cooperative state information, the local tracking layer generates desired angle of attack and bank angle commands, and sends the desired angle of attack and bank angle commands to the underlying control layer. The underlying generation unit is configured to generate a numerical sequence of control surface deflection angles in response to receiving desired angle of attack and roll angle commands from the local tracking layer, and in response to receiving aerodynamic parameters and physical state information.

10. A computer device, characterized in that, The computer device includes a processor, a memory, and a computer program stored in the memory and executable by the processor, wherein when the computer program is executed by the processor, it implements the steps of the method as described in any one of claims 1-8.

Images