Multi-agent reinforcement learning formation control method and system for rarefied Mars atmosphere

Abstract

The invention relates to the crossing field of multi-agent control and space exploration technologies, and provides a multi-agent reinforcement learning formation control method and system for a rarefied Mars atmosphere in order to overcome the applicability defect of an existing multi-agent formation control method in an extreme Mars environment. The method comprises the following steps: firstly, sensing and accurately estimating a mars rarefied atmosphere state in real time, and providing reliable environment input for a control decision; secondly, an optimal control instruction is generated for each unmanned aerial vehicle through a multi-agent reinforcement learning framework; state information asynchronization caused by communication time delay and packet loss is processed through a communication and time delay compensation strategy, and real-time performance and stability of formation control are guaranteed; and finally, formation topology dynamic adjustment in a fault scene is realized through formation topology management and an adaptive reconstruction strategy, so that the fault-tolerant capability of the system to member faults is improved, and continuous execution of a Mars exploration task is ensured. The method can be widely applied to deep space exploration tasks such as Mars surface exploration, topographic mapping and resource exploration.

Description

Technical Field

[0001] This invention relates to the intersection of multi-agent control and space exploration technology, specifically to a method and system for controlling unmanned aerial vehicle (UAV) swarm formations in the thin atmosphere, strong wind shear, and unstable communication environment of Mars. Background Technology

[0002] Mars exploration is one of the core directions of human deep space exploration, and drone swarms, with their advantages of wide coverage, high mission fault tolerance, and high exploration efficiency, have become key equipment for detailed exploration of the Martian surface. However, the unique and extreme environment of Mars brings many intractable technical challenges to the control of drone swarm formations. Existing multi-agent control methods based on Earth's atmospheric environment cannot be directly applied. Specific problems are as follows: The Martian atmosphere is extremely unique: its density is only about 1% of that of Earth's surface atmosphere, and it experiences severe wind shear and gusts. Traditional aerodynamic calculation and control algorithms based on fixed atmospheric models cannot match the dynamic changes in the Martian atmosphere in real time, which can easily lead to problems such as stall and attitude loss during UAV maneuvers, seriously threatening formation stability.

[0003] Severe energy and communication constraints exist in Mars exploration missions: UAVs rely entirely on their onboard batteries or solar panels for refueling, resulting in extremely limited energy resources. Furthermore, communication between the Martian surface and Earth suffers from significant latency, and short-range communication between UAVs is susceptible to terrain obstruction and electromagnetic interference, leading to packet loss and latency fluctuations. Existing control methods often fail to consider energy consumption optimization and communication latency compensation, potentially causing premature power outages or formation coordination failures.

[0004] Insufficient formation robustness and reconfiguration capability: In Mars exploration missions, drones may leave the formation due to equipment failure, energy depletion, or damage from extreme environments. Existing formation control methods mostly employ fixed topology management patterns and lack adaptive reconfiguration mechanisms based on real-time fault states. Once a member fails, it can easily trigger the collapse of the entire formation, making it impossible to guarantee the continuous execution of the mission.

[0005] Low efficiency of multi-agent collaborative decision-making: Traditional multi-agent control often adopts the "centralized decision-making + distributed execution" mode, which requires a large amount of global state information interaction. In the scenario of limited communication bandwidth on Mars, this can easily lead to increased decision-making latency. At the same time, existing reward functions mostly focus on formation error control and do not take into account the aerodynamic characteristics and energy consumption constraints of Mars, making it difficult to guide agents to learn a "safe-efficient-energy-saving" collaborative strategy.

[0006] In summary, for multi-agent formation control in the thin atmosphere of Mars, there is an urgent need for an integrated solution that can combine environmental perception, adaptive decision-making, communication compensation and topology reconstruction to overcome existing technical bottlenecks and meet the high robustness, high autonomy and high efficiency requirements of Mars exploration missions for cluster control. Summary of the Invention

[0007] The core objective of this invention is to overcome the limitations of existing multi-agent formation control methods in the extreme environment of Mars, and to provide a multi-agent reinforcement learning formation control method and system for the thin Martian atmosphere, thereby achieving the following objectives: real-time perception and accurate estimation of the Martian thin atmosphere state, providing reliable environmental input for control decisions; efficient collaborative control of UAV swarms under low communication bandwidth based on distributed reinforcement learning, balancing formation accuracy, energy consumption optimization, and aerodynamic safety; compensation for communication latency and packet loss, ensuring the real-time performance and stability of formation control; and adaptive reconstruction of formation topology, improving the system's fault tolerance to member failures and ensuring the continuous execution of the Mars exploration mission.

[0008] To achieve the above objectives, the present invention provides the following technical solution: In a first aspect, the present invention provides a multi-agent reinforcement learning formation control method for the thin atmosphere of Mars, the control method comprising the following steps: Step 1: Acquire and process raw observational data of the Martian atmosphere to jointly estimate the state of the Martian atmosphere and obtain key parameters of the Martian atmosphere; Step 2: Based on key parameters of the Martian atmosphere, generate optimal control commands for each UAV using a multi-agent reinforcement learning framework; Step 3: Handle asynchronous state information caused by communication latency and packet loss through communication and latency compensation strategies; Step 4: Dynamically adjust the formation topology under fault scenarios through formation topology management and adaptive reconfiguration strategies.

[0009] Furthermore, an atmospheric state estimation is performed using a Kalman / particle hybrid filter, specifically as follows: By processing linear variation data of the Martian atmosphere, including air pressure and temperature, using Kalman filtering, estimates of basic atmospheric pressure and temperature are obtained. The linearity error of the Kalman filter is corrected by approximating the real wind field distribution through particle resampling. The equivalent air density is calculated based on the ideal gas law and air velocity measurements, and the local wind field gradient is output.

[0010] Furthermore, the aforementioned multi-agent reinforcement learning framework includes a hierarchical policy network architecture, neighbor interaction modeling, and an integrated propulsion-aerodynamic-energy reward function; A layered policy network architecture includes: Local Actor Network: Deploy an independent Actor network for each drone in the formation. The inputs of this network include the drone's own state, the environmental perception information processed in step 1, and the state of neighboring drones obtained through communication. The output is the low-level control action of the drone. Global Critic Network: During the training phase, a centralized Critic network is used. This network is used to access the status and actions of all UAVs and evaluate the overall behavior of the formation by calculating the temporal difference error, which guides the parameter updates of each local Actor network. During the execution phase, the Critic network only retains the local evaluation function.

[0011] Furthermore, a graph attention network is used to model the neighbor interactions of multiple drone formations, specifically as follows: Each drone in the formation is considered a node in a graph, and the communication links between adjacent drones are considered edges. A graph attention network is used to dynamically assign attention weights to each node.

[0012] Furthermore, step 3 above specifically includes: Latency prediction and buffering: Maintain a neighbor latency buffer for each UAV, record historical communication data packets with each neighbor, and estimate the communication latency with neighbors in real time using a sliding window algorithm; State alignment and extrapolation compensation: When drones i exist t Receive neighboring machines at all times j historical status At that time, based on neighboring machines j The dynamic model and the action at the previous moment Extrapolate its state forward to calculate the current time. t estimated state To realize drones i In the Actor network, the states of neighboring machines are all based on the same time reference when making decisions.

[0013] Furthermore, step 4 above specifically includes: Each drone periodically broadcasts a heartbeat signal containing its own health status. If a specific member's heartbeat signal is not received for a preset period of time, that member is considered to be faulty. At the same time, the packet loss rate of communication with neighboring drones is monitored in real time. If the packet loss rate continues to exceed the threshold, the corresponding communication link is considered to be permanently interrupted. When a fault or interruption is detected, the topology reconfigurator is triggered, and the optimal formation is selected from the candidate formations based on a preset formation robustness index.

[0014] Furthermore, the aforementioned topology reconstructor generates a smooth transition trajectory for the remaining UAVs and sends out new target positions through the communication module; each UAV moves to the new position along the transition trajectory based on the distributed reinforcement learning control module, completing the formation reconstruction.

[0015] Secondly, the present invention also provides a multi-agent reinforcement learning formation control system for the thin atmosphere of Mars, used to implement the control method proposed above. The control system includes: The environmental perception and thin atmosphere state estimation module is used to output key parameters of the Martian atmosphere in real time through multi-source sensor data fusion and hybrid filtering algorithms, providing high-precision input for subsequent control decisions. The distributed reinforcement learning control module is used to generate optimal control commands for each UAV based on key parameters of the Martian atmosphere using a multi-agent reinforcement learning framework. The communication and delay compensation module is used to handle asynchronous status information caused by communication delay and packet loss; The formation topology management and adaptive reconfiguration module is used to dynamically adjust the formation topology in fault scenarios.

[0016] Thirdly, the present invention also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, performs the control method described above.

[0017] Fourthly, the present invention also provides a computer device, the device including a memory and a processor, wherein the memory stores a computer program, and when the processor runs the computer program stored in the memory, the processor executes the control method described above.

[0018] The beneficial effects of this invention are as follows: (1) This invention uses multi-source sensor data fusion and hybrid filtering algorithm to perceive and accurately estimate the state of the thin atmosphere of Mars in real time, providing a reliable environmental input for control decisions. It solves the problem that existing control methods cannot match the dynamic changes of the Martian atmosphere in real time, which leads to problems such as stall and attitude loss during UAV maneuvers.

[0019] (2) Based on distributed reinforcement learning, this invention achieves efficient collaborative control of UAV swarms under low communication bandwidth, taking into account formation accuracy, energy consumption optimization and aerodynamic safety.

[0020] (3) The present invention compensates for communication delay and packet loss through communication and delay compensation strategies, thereby ensuring the real-time performance and stability of formation control.

[0021] (4) This invention achieves adaptive reconfiguration of the formation topology through formation topology management and adaptive reconfiguration strategy, thereby improving the system’s fault tolerance to member failures and ensuring the continuous execution of the Mars exploration mission.

[0022] (5) This invention can be widely applied to deep space exploration missions such as Mars surface exploration, topographic mapping, and resource exploration, providing a stable and efficient control solution for collaborative operations of multiple agents in extreme planetary environments. Attached Figure Description

[0023] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0024] Figure 1 This is a diagram illustrating the architecture of a multi-agent reinforcement learning formation control system for the thin atmosphere of Mars, as described in this embodiment of the invention. Figure 2 This is a flowchart of a multi-agent reinforcement learning formation control method for the thin atmosphere of Mars, as described in an embodiment of the present invention. Detailed Implementation

[0025] The specific implementation details (such as experimental setup, operation procedures, data processing steps, and example parameters) of "A Multi-Agent Reinforcement Learning Formation Control Method and System for the Thin Atmosphere of Mars" provided in this specification are primarily intended for illustrative purposes rather than limiting definitions, aiming to help those skilled in the art thoroughly understand the principles and implementation of the invention. However, those skilled in the art should understand that these details represent only one feasible embodiment, and the core concept of the invention can be fully realized through other technical means or alternative solutions not described in detail, without departing from its spirit and essence. Furthermore, the omission of details of conventional experimental methods and apparatus known in the art in the specification is to avoid redundant information interfering with the understanding of the innovation points. This does not mean that these known technologies are not required during implementation, and those skilled in the art should be able to supplement and apply them based on their professional knowledge.

[0026] The specific embodiments of the present invention will be further described in detail below with reference to the accompanying drawings. The following examples will help those skilled in the art to further understand the present invention, but do not limit the present invention in any way. It should be noted that those skilled in the art can make several changes and improvements without departing from the concept of the present invention, and these all fall within the protection scope of the present invention.

[0027] Example 1: This example aims to overcome the limitations of existing multi-agent formation control methods in the extreme Martian environment, including: the inability of existing control methods to match the dynamic changes of the Martian atmosphere in real time, leading to problems such as stall and attitude loss during UAV maneuvers; the lack of consideration for energy consumption optimization and communication latency compensation, which can easily lead to premature power outages or formation coordination failures; the lack of an adaptive reconstruction mechanism based on real-time fault states; and the low efficiency of multi-agent collaborative decision-making. Therefore, this example provides a multi-agent reinforcement learning formation control method for the thin Martian atmosphere, comprising the following steps: Step 1: Acquire and process raw observational data of the Martian atmosphere to jointly estimate the state of the Martian atmosphere and obtain key parameters of the Martian atmosphere; Step 2: Based on key parameters of the Martian atmosphere, generate optimal control commands for each UAV using a multi-agent reinforcement learning framework; Step 3: Handle asynchronous state information caused by communication latency and packet loss through communication and latency compensation strategies; Step 4: Dynamically adjust the formation topology under fault scenarios through formation topology management and adaptive reconfiguration strategies.

[0028] Specifically: In this embodiment, a Kalman / particle hybrid filter is used for atmospheric state estimation, specifically as follows: By processing linear variation data of the Martian atmosphere, including air pressure and temperature, using Kalman filtering, estimates of basic atmospheric pressure and temperature are obtained. The linearity error of the Kalman filter is corrected by approximating the real wind field distribution through particle resampling. The equivalent air density is calculated based on the ideal gas law and air velocity measurements, and the local wind field gradient is output.

[0029] In this embodiment, the multi-agent reinforcement learning framework includes a hierarchical policy network architecture, neighbor interaction modeling, and an integrated propulsion-aerodynamic-energy consumption reward function; A layered policy network architecture includes: Local Actor Network: Deploy an independent Actor network for each drone in the formation. The inputs of this network include the drone's own state, the environmental perception information processed in step 1, and the state of neighboring drones obtained through communication. The output is the low-level control action of the drone. Global Critic Network: During the training phase, a centralized Critic network is used. This network is used to access the status and actions of all UAVs and evaluate the overall behavior of the formation by calculating the temporal difference error, which guides the parameter updates of each local Actor network. During the execution phase, the Critic network only retains the local evaluation function.

[0030] In this embodiment, a graph attention network is used to model the neighbor interactions of multiple UAV formations, specifically as follows: Each drone in the formation is considered a node in a graph, and the communication links between adjacent drones are considered edges. A graph attention network is used to dynamically assign attention weights to each node.

[0031] In this embodiment, the integrated reward function is:

[0032] in, ( i For the first i (One drone) is used to design a multi-objective distributed reward function for the Martian environment. For the first i The deviation of the position of each drone from the target formation. This is a weighting coefficient, and the reward is for formation accuracy. For the first i The instantaneous increase in energy consumption of a drone. As a weighting factor, this item penalizes high-energy-consuming maneuvers; For aerodynamic safety margin, based on Calculate the torque required for the current maneuver. With the maximum available torque The difference, As a weighting factor, this item ensures that the drone stays away from its aerodynamic limits; For the first i The quality of the communication link between the drone and its neighboring drones. These are the weighting coefficients; For communication delay, This is the weighting coefficient, and the penalty increases the latency.

[0033] In this embodiment, step 3 above is used to solve the problem of asynchronous state information caused by communication latency and packet loss in the Martian environment, specifically as follows: Latency prediction and buffering: Maintain a neighbor latency buffer for each UAV, record historical communication data packets with each neighbor, and estimate the communication latency with neighbors in real time using a sliding window algorithm; State alignment and extrapolation compensation: When drones i exist t Receive neighboring machines at all times j historical status At that time, based on neighboring machines j The dynamic model and the action at the previous moment Extrapolate its state forward to calculate the current time. t estimated state To realize drones iIn the Actor network, the states of neighboring machines are all based on the same time reference when making decisions.

[0034] in,

[0035] in, For neighboring machines j The dynamic state transition function; through state extrapolation, ensure the UAV's... i When making decisions, the Actor network uses neighboring machine states based on the same time reference, eliminating control deviations caused by time delays.

[0036] In this embodiment, step 4 above is used to realize dynamic adjustment of the formation topology under fault scenarios, specifically as follows: Fault and link interruption detection: Each drone periodically broadcasts a heartbeat signal containing its own health status. If a specific member's heartbeat signal is not received for a consecutive preset period, the member is determined to be faulty, thus achieving health status monitoring. At the same time, the packet loss rate of communication with neighboring drones is monitored in real time. If the packet loss rate continues to exceed the threshold, the corresponding communication link is determined to be permanently interrupted, thus achieving communication link detection. When a fault or interruption is detected, the topology reconfigurator is triggered, and the optimal formation is selected from the candidate formations based on the preset formation robustness index, thus realizing the reconfiguration trigger and formation selection.

[0037] The robustness index is calculated as follows:

[0038] in, The average energy consumption of the formation under the new configuration. Maximum permissible energy consumption; The average connectivity probability between drones under the new configuration; These are the weighting coefficients.

[0039] The reconfigurator generates a smooth transition trajectory for the remaining UAVs and sends a new target position through the communication module; each UAV moves to the new position along the transition trajectory based on the distributed reinforcement learning control module, thus completing the formation reconfiguration.

[0040] Example 2, Combination Figure 1 and Figure 2 This embodiment describes a multi-agent reinforcement learning formation control system for the thin atmosphere of Mars, used to implement the control method described in Embodiment 1 above. The control system includes: The environmental perception and thin atmosphere state estimation module is used to output key parameters of the Martian atmosphere in real time through multi-source sensor data fusion and hybrid filtering algorithms, providing high-precision input for subsequent control decisions. The distributed reinforcement learning control module is used to generate optimal control commands for each UAV based on key parameters of the Martian atmosphere using a multi-agent reinforcement learning framework. The communication and delay compensation module is used to handle asynchronous status information caused by communication delay and packet loss; The formation topology management and adaptive reconfiguration module is used to dynamically adjust the formation topology in fault scenarios.

[0041] Specifically, the overall system architecture includes an environmental perception and thin atmosphere state estimation module, a distributed reinforcement learning control module, a communication and latency compensation module, and a formation topology management and adaptive reconstruction module. These four modules work collaboratively, and their functions and interaction logic are as follows: Figure 1 and Figure 2 As shown, specifically: 1. Environmental perception and thin atmosphere state estimation module: This module is used to output key parameters of the Martian atmosphere in real time through multi-source sensor data fusion and hybrid filtering algorithms, providing high-precision input for subsequent control decisions.

[0042] Hardware configuration: Each drone is equipped with a Pitot tube, a miniature temperature sensor, a high-precision barometer, and a six-axis inertial measurement unit, forming a multi-source sensing unit; Data processing flow: (1) Collect raw sensor data in real time and preprocess abnormal data; (2) Atmospheric state estimation using a Kalman / particle hybrid filter: Kalman filter: processes linearly changing data such as air pressure and temperature, and outputs estimated values ​​of basic atmospheric pressure P and temperature T; Particle filtering: For strongly nonlinear parameters such as wind field gradient, particle resampling is used to approximate the real wind field distribution and correct the linearity error of Kalman filtering. (3) Based on the ideal gas state equation and the air velocity measurement, calculate the equivalent air density and output the local wind field gradient. Transmit the above parameters to the distributed reinforcement learning control module in real time.

[0043] 2. Distributed Reinforcement Learning Control Module: This module employs a multi-agent reinforcement learning (MARL) framework to generate optimal control commands for each UAV. The core design is as follows: (1) Layered strategy network architecture: Local Actor Network: Each drone independently deploys an Actor network. The network's input parameters include the drone's own state, the output of the environmental perception module, and the state of neighboring drones. The output is the underlying control action, ensuring the real-time nature and distributed characteristics of decision-making. Global Critic Network: During the training phase, a centralized Critic network is deployed, which can access the status and actions of all UAVs. It evaluates the overall behavior of the formation by calculating the temporal difference error and guides the update of the Actor network parameters. During the execution phase, the Critic network only retains the local evaluation function to avoid global communication bandwidth occupation.

[0044] (2) Neighbor interaction modeling: A graph attention network (GAT) is used to construct a neighbor information interaction model: Each drone is considered a node in a graph, and the communication links between neighboring drones are considered edges. GAT dynamically assigns "attention weights" to each node: it assigns higher weights to neighboring nodes with stable states and high communication quality, prioritizes their state information for decision-making, and resists the impact of communication packet loss and neighboring node state noise.

[0045] (3) Propulsion-Aerodynamics-Energy Consumption Integrated Reward Function: Design a multi-objective distributed reward function for the Martian environment. (i is the first) i (using one drone) to guide the agent in learning the optimal strategy, using the following formula:

[0046] in, ( i For the first i (One drone) is used to design a multi-objective distributed reward function for the Martian environment. For the first i The deviation of the position of each drone from the target formation. This is a weighting coefficient, and the reward is for formation accuracy. For the first i The instantaneous increase in energy consumption of a drone. As a weighting factor, this item penalizes high-energy-consuming maneuvers; For aerodynamic safety margin, based on Calculate the torque required for the current maneuver. With the maximum available torque The difference, As a weighting factor, this item ensures that the drone stays away from its aerodynamic limits; For the first i The quality of the communication link between the drone and its neighboring drones. These are the weighting coefficients; For communication delay, This is the weighting coefficient, and the penalty increases the latency.

[0047] 3. Communication and Delay Compensation Module: This module is designed to address the asynchronous status information issue caused by communication latency and packet loss in the Martian environment. Its specific functions are as follows: (1) Delay prediction and buffering: Each UAV maintains a neighbor delay buffer, records historical communication data packets with each neighbor, and estimates the delay with each neighbor in real time using a sliding window algorithm. j Communication latency ; (2) State alignment and extrapolation compensation: When the UAV i exist t Receive neighboring machines at all times j historical status At that time, based on neighboring machines j The dynamic model and the action at the previous moment Extrapolate its state forward to calculate the current time. t estimated state The formula is as follows:

[0048] in, For neighboring machines j The dynamic state transition function; through state extrapolation, ensure the UAV's... i When making decisions, the Actor network uses neighboring machine states based on the same time reference, eliminating control deviations caused by time delays.

[0049] 4. Formation Topology Management and Adaptive Reconfiguration Module: This module is used to dynamically adjust the formation topology under fault scenarios. The specific process is as follows: (1) Fault and link interruption detection: Health status monitoring: Each drone periodically sends a "heartbeat signal" to other members, including its own battery voltage and sensor working status; if no heartbeat signal is received from a member for 3 consecutive cycles, that member is considered to be malfunctioning; Communication link detection: The packet loss rate with neighboring machines is monitored in real time through the communication module. If the packet loss rate exceeds 30% for 5 consecutive seconds, the link is determined to be permanently interrupted. (2) Reconstruction triggering and form selection: When a fault or link interruption is detected, the topology reconstructor is triggered, and the optimal form is selected based on the preset formation robustness index; The robustness index is calculated as follows:

[0050] in, The average energy consumption of the formation under the new configuration. Maximum permissible energy consumption; The average connectivity probability between drones under the new configuration; These are the weighting coefficients.

[0051] (3) Smooth Reconstruction Execution: The reconstructor generates a smooth transition trajectory for the remaining UAVs and sends out new target positions through the communication module; each UAV moves to the new position along the transition trajectory based on the distributed reinforcement learning control module, completing the formation reconstruction.

[0052] In summary, it can be seen that the multi-agent reinforcement learning formation control method and system proposed in this invention for the thin atmosphere of Mars can achieve the following objectives: (1) Real-time sensing and accurate estimation of the thin Martian atmosphere to provide reliable environmental input for control decisions; (2) Based on distributed reinforcement learning, efficient collaborative control of UAV swarms is achieved under low communication bandwidth, taking into account formation accuracy, energy consumption optimization and aerodynamic safety. (3) Compensate for communication delay and packet loss to ensure the real-time performance and stability of formation control; (4) To achieve adaptive reconfiguration of the formation topology, improve the system’s fault tolerance to member failures, and ensure the continuous execution of the Mars exploration mission.

[0053] Example 3: This example provides a computer-readable storage medium storing a computer program. When the computer program is run by a processor, it executes the multi-agent reinforcement learning formation control method for the thin atmosphere of Mars described in any of the above examples.

[0054] Those skilled in the art will understand that implementing all or part of the processes in the above embodiments can be accomplished by a computer program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. The storage medium can be a magnetic disk, optical disk, read-only memory (ROM), random access memory (RAM), flash memory, hard disk drive (HDD), or solid-state drive (SSD), etc.; the storage medium can also include combinations of the above types of memory.

[0055] Example 4: This example provides a computer device, which includes a memory and a processor. The memory stores a computer program. When the processor runs the computer program stored in the memory, the processor executes a multi-agent reinforcement learning formation control method for the thin atmosphere of Mars as described in any of the above examples.

[0056] This embodiment provides a computer device. This part of the hardware device is a general model and is not shown in the figure. The system includes a processor and a memory, which can be connected by a bus or other means. The memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs, non-transitory computer-executable programs and modules, as well as corresponding program instructions / modules. The processor executes various functional applications and data processing by running the non-transitory software programs, instructions and modules stored in the memory, so as to realize the multi-agent reinforcement learning formation control method and steps for the thin atmosphere of Mars in the above method embodiment.

[0057] The memory may include a program storage area and a data storage area, wherein the program storage area may store the operating system and applications required for at least one function; the data storage area may store data created by the processor, etc. Furthermore, the memory may include high-speed random access memory and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, the memory may optionally include memory remotely located relative to the processor, which can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, mobile communication networks, and combinations thereof.

[0058] One or more modules are stored in the memory. When the processor executes, it performs the method steps in the embodiments. In this way, the invention objective can be achieved through the method, apparatus and process of the present invention. The specific details of the computer device described above can be understood by referring to the relevant descriptions and effects in the embodiments, and will not be repeated here.

[0059] The above description of the technical solution provided by the present invention through several specific embodiments is intended to highlight the advantages and benefits of the technical solution provided by the present invention. However, the above-described specific embodiments are not intended to limit the present invention. Any reasonable modifications and improvements to the present invention, reasonable combinations of implementation methods and equivalent substitutions based on the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A multi-agent reinforcement learning formation control method for the thin atmosphere of Mars, characterized in that, The method is as follows: Step 1: Acquire and process raw observational data of the Martian atmosphere to jointly estimate the state of the Martian atmosphere and obtain key parameters of the Martian atmosphere; Step 2: Based on key parameters of the Martian atmosphere, generate optimal control commands for each UAV using a multi-agent reinforcement learning framework; Step 3: Handle asynchronous state information caused by communication latency and packet loss through communication and latency compensation strategies; Step 4: Dynamically adjust the formation topology under fault scenarios through formation topology management and adaptive reconfiguration strategies.

2. The control method according to claim 1, characterized in that, Atmospheric state estimation is performed using a Kalman / particle hybrid filter, specifically as follows: By processing linear variation data of the Martian atmosphere, including air pressure and temperature, using Kalman filtering, estimates of basic atmospheric pressure and temperature are obtained. The linearity error of the Kalman filter is corrected by approximating the real wind field distribution through particle resampling. The equivalent air density is calculated based on the ideal gas law and air velocity measurements, and the local wind field gradient is output.

3. The control method according to claim 1, characterized in that, The multi-agent reinforcement learning framework includes a hierarchical policy network architecture, neighbor interaction modeling, and an integrated propulsion-aerodynamic-energy reward function; A layered policy network architecture includes: Local Actor Network: Deploy an independent Actor network for each drone in the formation. The inputs of this network include the drone's own state, the environmental perception information processed in step 1, and the state of neighboring drones obtained through communication. The output is the low-level control action of the drone. Global Critic Network: During the training phase, a centralized Critic network is used. This network is used to access the status and actions of all UAVs and evaluate the overall behavior of the formation by calculating the temporal difference error, which guides the parameter updates of each local Actor network. During the execution phase, the Critic network only retains the local evaluation function.

4. The control method according to claim 3, characterized in that, Modeling neighbor interactions in multi-drone formations using graph attention networks, specifically: Each drone in the formation is considered a node in a graph, and the communication links between adjacent drones are considered edges. A graph attention network is used to dynamically assign attention weights to each node.

5. The control method according to claim 1, characterized in that, Step 3 specifically involves: Latency prediction and buffering: Maintain a neighbor latency buffer for each UAV, record historical communication data packets with each neighbor, and estimate the communication latency with neighbors in real time using a sliding window algorithm; State alignment and extrapolation compensation: When drones i exist t Receive neighboring machines at all times j historical status At that time, based on neighboring machines j The dynamic model and the action at the previous moment Extrapolate its state forward to calculate the current time. t estimated state To realize drones i In the Actor network, the states of neighboring machines are all based on the same time reference when making decisions.

6. The control method according to claim 1, characterized in that, Step 4 is as follows: Each drone periodically broadcasts a heartbeat signal containing its own health status. If a specific member's heartbeat signal is not received for a preset period of time, that member is considered to be faulty. At the same time, the packet loss rate of communication with neighboring drones is monitored in real time. If the packet loss rate continues to exceed the threshold, the corresponding communication link is considered to be permanently interrupted. When a fault or interruption is detected, the topology reconfigurator is triggered, and the optimal formation is selected from the candidate formations based on a preset formation robustness index.

7. The control method according to claim 6, characterized in that, The topology reconfigurator generates a smooth transition trajectory for the remaining UAVs and sends a new target position through the communication module; each UAV moves to the new position along the transition trajectory based on the distributed reinforcement learning control module, completing the formation reconfiguration.

8. A multi-agent reinforcement learning formation control system for the thin atmosphere of Mars, characterized in that, include: The environmental perception and thin atmosphere state estimation module is used to output key parameters of the Martian atmosphere in real time through multi-source sensor data fusion and hybrid filtering algorithms, providing high-precision input for subsequent control decisions. The distributed reinforcement learning control module is used to generate optimal control commands for each UAV based on key parameters of the Martian atmosphere using a multi-agent reinforcement learning framework. The communication and delay compensation module is used to handle asynchronous status information caused by communication delay and packet loss; The formation topology management and adaptive reconfiguration module is used to dynamically adjust the formation topology in fault scenarios.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, performs the control method according to any one of claims 1-7.

10. A computer device, characterized in that, The device includes a memory and a processor, wherein the memory stores a computer program, and when the processor runs the computer program stored in the memory, the processor performs the control method according to any one of claims 1-7.

Images