Method for dynamic collaborative allocation of airspace resources based on multi-agent reinforcement learning

Abstract

This application provides a method for dynamic collaborative allocation of airspace resources based on multi-agent reinforcement learning, belonging to the field of airspace resource allocation technology. The method includes: constructing a low-altitude airspace simulation environment and determining the environmental state space; constructing a set of perception state spaces and combining it with an action space set to construct a multi-objective reward mechanism; performing policy optimization analysis on the agent set to obtain a set of regulating action spaces and a set of regulating perception state spaces; performing conflict detection and resolution on the regulating action space set; converting the target action space set into control command execution, updating the environmental state space, and returning to construct the perception state space set. This application addresses the technical problem in existing technologies where the inability of multiple agents to form spatiotemporally integrated collaborative cognition during policy optimization restricts the efficiency and security of dynamic collaborative allocation of airspace resources, thus achieving the technical effect of improving the efficiency and security of dynamic collaborative allocation of airspace resources.

Description

Technical Field

[0001] This application relates to the field of airspace resource allocation technology, and in particular to a dynamic collaborative allocation method for airspace resources based on multi-agent reinforcement learning. Background Technology

[0002] With the rapid development of the air transport industry and the widespread application of drone technology, the demand for airspace resources has increased dramatically, and the airspace environment is increasingly characterized by high density, high dynamics, and high complexity. Therefore, how to efficiently and safely allocate limited airspace resources has become a core issue in ensuring the safe operation of air traffic and improving airspace capacity.

[0003] Currently, existing spatial resource allocation methods typically focus only on the state-action value assessment of individual agents, neglecting the spatial proximity and interactivity of agent groups, as well as the temporal continuity and correlation of action sequences. This spatiotemporally fragmented approach prevents agents from accurately understanding the impact of their actions on the future trajectories of other agents when learning cooperative strategies, and also makes it difficult to capture the spatiotemporal evolution of group behavior. Due to the lack of a comprehensive representation of spatiotemporal dependencies, existing methods often only perform isolated value assessments in the design of multi-objective reward mechanisms, making agents prone to falling into suboptimal strategies during local optimization and unable to achieve a balance between task objectives, safety objectives, efficiency objectives, and cooperative objectives.

[0004] In summary, existing technologies suffer from the technical problem that the inability of multiple agents to form a spatiotemporally integrated collaborative cognition during policy optimization restricts the efficiency and security of dynamic collaborative allocation of airspace resources. Summary of the Invention

[0005] The purpose of this application is to provide a method for dynamic collaborative allocation of airspace resources based on multi-agent reinforcement learning, in order to solve the technical problem in the prior art that the efficiency and security of dynamic collaborative allocation of airspace resources are restricted because multi-agents cannot form spatiotemporal fusion collaborative cognition during policy optimization.

[0006] In view of the above problems, this application provides a method for dynamic collaborative allocation of airspace resources based on multi-agent reinforcement learning, including: constructing a low-altitude airspace simulation environment and determining the environment state space; based on the environment state space, constructing a set of perception state spaces corresponding to a set of agents, and constructing a multi-objective reward mechanism in combination with the action space set; performing policy optimization analysis on the set of agents based on the multi-objective reward mechanism to obtain an adjustment action space set and an adjustment perception state space set; performing conflict detection and resolution on the adjustment action space set based on safety interval constraints and the adjustment perception state space set to obtain a target action space set; converting the target action space set into control command execution, obtaining actual state feedback data to update the environment state space, and returning to execute the construction of the perception state space set to achieve dynamic closed-loop allocation.

[0007] Preferably, the method for dynamic collaborative allocation of airspace resources based on multi-agent reinforcement learning further includes: delineating the airspace range, importing geographic information data, and defining route nodes, wherein the route nodes include origin, landing point, and intermediate nodes; setting dynamic environmental factors, wherein the dynamic environmental factors include low-altitude air traffic flow, meteorological conditions, low-altitude temporary controlled airspace, communication spectrum bandwidth, electromagnetic environment restrictions, and ground population density restrictions; constructing the airspace simulation environment based on the route nodes and the dynamic environmental factors; and determining the environmental state space according to the airspace simulation environment.

[0008] Preferably, the method for dynamic collaborative allocation of spatial resources based on multi-agent reinforcement learning further includes: defining the agent set and determining the action space set, wherein each action space includes discrete actions and continuous actions; combining the agent set and the environmental state space to determine the perception state space set, wherein each perception state space includes its own state, environmental perception state, and resource perception state; defining multiple objectives and constructing an interaction logic that quantifies the action space set and the perception state space set into positive incentives and negative penalties to generate a multi-objective system, wherein the multiple objectives include task objectives, safety objectives, efficiency objectives, and collaborative objectives; using the agent set as the decision-making body, the perception state space set as the decision-making basis, and the multi-objective system as the constraint condition, performing decision deduction based on the action space set to construct the multi-objective reward mechanism.

[0009] Preferably, the method for dynamic collaborative allocation of airspace resources based on multi-agent reinforcement learning further includes: dividing the airspace into grids, forming a distributed grid group with a preset number of grids, and distributing the agent set in the distributed grid group according to the set of perception state spaces to obtain a distributed agent set; based on the distributed agent set, analyzing the distributed adjustment action space set and the distributed adjustment perception state space set under the multi-objective reward mechanism within the distributed grid group; and based on the central controller, combining the distributed adjustment action space set and the distributed adjustment perception state space set, analyzing the adjustment action space set and the adjustment perception state space set of the agent set under the multi-objective reward mechanism.

[0010] Preferably, the method for dynamic collaborative allocation of spatial resources based on multi-agent reinforcement learning further includes: constructing a distributed topology graph structure based on the set of perception state spaces; extracting temporal features by combining the distributed topology graph structure and the set of action spaces to obtain a distributed temporal dependency feature vector; and obtaining a distributed adjustment action space set and a distributed adjustment perception state space set corresponding to the distributed optimal target value based on the distributed temporal dependency feature vector and according to the multi-objective reward mechanism.

[0011] Preferably, the method for dynamic collaborative allocation of spatial resources based on multi-agent reinforcement learning further includes: constructing a graph neural network layer to encode the spatial topological relationships of the agent set based on the distributed topological graph structure to obtain a spatial feature vector; constructing a temporal neural network layer to extract temporal dependencies from the spatial feature vector based on the action sequence of the action space set to obtain a spatiotemporal fusion feature vector; and mapping the spatiotemporal fusion feature vector through a fully connected layer to generate the distributed temporal dependency feature vector.

[0012] Preferably, the method for dynamic collaborative allocation of spatial resources based on multi-agent reinforcement learning further includes: evaluating the value of the distributed temporal dependency feature vector based on the multi-objective reward mechanism to obtain a distributed multi-objective value; introducing an ideal target value, generating a distributed multi-objective value to be adjusted based on the value deviation between the distributed multi-objective value and the ideal target value, and mapping it to obtain a distributed temporal dependency feature vector to be adjusted; generating a distributed action space set to be adjusted and a distributed perception state space set to be adjusted based on the distributed action space set and the distributed perception state space set to be adjusted; iteratively adjusting the distributed temporal dependency feature vector to be adjusted based on the distributed action space set and the distributed perception state space set to be adjusted to obtain a distributed optimal target value; and obtaining a distributed adjustment action space set and a distributed adjustment perception state space set corresponding to the distributed adjustment temporal dependency feature vector based on the distributed optimal target value.

[0013] Preferably, the method for dynamic collaborative allocation of airspace resources based on multi-agent reinforcement learning further includes: in distributed regulation, the multi-objective reward mechanism takes the first objective as the primary objective, and in clustered regulation, the multi-objective reward mechanism takes the second objective as the primary objective.

[0014] Preferably, the method for dynamic collaborative allocation of spatial resources based on multi-agent reinforcement learning further includes: combining the set of adjustment perception states to obtain the predicted trajectory of the agent set after executing the action corresponding to the set of adjustment action spaces; performing conflict detection on the predicted trajectory based on a safety interval rule, and triggering a conflict resolution mechanism if a conflict is detected; generating an emergency maneuver action based on the conflict resolution mechanism, and using the emergency maneuver action to cover the corresponding action in the set of adjustment action spaces to obtain a safety-optimized target action space set.

[0015] Preferably, the method for dynamic collaborative allocation of spatial resources based on multi-agent reinforcement learning further includes: converting the target action space set into control commands and sending them to the corresponding agents for execution; after a preset time step, obtaining the actual state feedback data after the agent set executes the control commands; updating the environmental state space of the spatial simulation environment based on the actual state feedback data, and returning the perception state space set of the agent set to be constructed, thereby realizing dynamic closed-loop allocation.

[0016] The technical solution provided in this application has at least the following technical effects or advantages: by achieving the technical goal of constructing a spatiotemporal fusion perception and decision-making mechanism based on distributed topological graph structure and temporal dependency feature extraction, it enhances the comprehensive representation ability of multi-agents on spatial interaction relationships and action evolution laws, improves the global optimality of the set of adjustment action space and the set of adjustment perception state space obtained under the multi-objective reward mechanism, and thus improves the efficiency and security of dynamic collaborative allocation of airspace resources.

[0017] The above description is merely an overview of the technical solution of this application. To enable a clearer understanding of the technical means of this application and to facilitate its implementation according to the description, and to make the above and other objects, features, and advantages of this application more apparent, specific embodiments of this application are described below. It should be understood that the content described in this section is not intended to identify key or important features of the embodiments of this application, nor is it intended to limit the scope of this application. Other features of this application will become readily apparent through the following description. Attached Figure Description

[0018] To more clearly illustrate the technical solutions in this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are merely exemplary. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0019] Figure 1 This is a flowchart illustrating the dynamic collaborative allocation method for spatial resources based on multi-agent reinforcement learning proposed in this application.

[0020] Figure 2 This is a schematic diagram illustrating the process of determining the environmental state space in the dynamic collaborative allocation method for spatial resources based on multi-agent reinforcement learning in this application. Detailed Implementation

[0021] This application provides a method for dynamic collaborative allocation of airspace resources based on multi-agent reinforcement learning. It addresses the technical problem in existing technologies where the inability of multiple agents to achieve spatiotemporal fusion in collaborative cognition during policy optimization hinders the efficiency and security of dynamic collaborative allocation of airspace resources. The method aims to construct a spatiotemporal fusion perception and decision-making mechanism based on a distributed topological graph structure and temporal dependency feature extraction. This enhances the comprehensive representation ability of multiple agents regarding spatial interaction relationships and action evolution patterns, improves the global optimality of the set of adjustment action spaces and the set of adjustment perception state spaces obtained under a multi-objective reward mechanism, and ultimately improves the efficiency and security of dynamic collaborative allocation of airspace resources.

[0022] The technical solutions of this application will now be clearly and completely described with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. It should be understood that this application is not limited to the exemplary embodiments described herein. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort are within the scope of protection of this application. It should also be noted that, for ease of description, only the parts related to this application are shown in the accompanying drawings, not all of them.

[0023] Please see Figure 1 and Figure 2 This application provides a method for dynamic collaborative allocation of spatial resources based on multi-agent reinforcement learning, which specifically includes the following steps: Construct a low-altitude airspace simulation environment and determine the environmental state space.

[0024] Furthermore, this application also includes: delineating the airspace range, importing geographic information data, and defining route nodes, wherein the route nodes include origin, landing point, and intermediate nodes; setting dynamic environmental factors, wherein the dynamic environmental factors include low-altitude air traffic flow, meteorological conditions, low-altitude temporary controlled airspace, communication spectrum bandwidth, electromagnetic environment restrictions, and ground population density restrictions; constructing the airspace simulation environment based on the route nodes and the dynamic environmental factors; and determining the environmental state space according to the airspace simulation environment.

[0025] Specifically, this involves defining the geographical boundaries of low-altitude airspace, i.e., delineating the airspace range of low-altitude airspace, and importing geographic information data including topographical and geomorphological elements, while also defining route nodes. Route nodes specifically include the starting point of the flight mission (origin), the ending point of the mission (landing point), and key locations along the route (intermediate nodes).

[0026] Based on the defined airspace boundaries, dynamic environmental factors affecting airspace use are further established. These dynamic environmental factors include low-altitude air traffic flow, meteorological conditions, temporary low-altitude controlled airspace, communication spectrum bandwidth, electromagnetic environment limitations, and ground population density limitations. Low-altitude air traffic flow refers to the set of aircraft position, speed, heading, and other motion state variables over time within the three-dimensional airspace, reflecting the real-time distribution characteristics and evolution trends of multiple agents in the low-altitude environment. Furthermore, meteorological conditions encompass physical parameters such as wind speed, wind direction, precipitation probability, visibility, and turbulence intensity. These physical parameters directly affect the aerodynamic characteristics of aircraft and the accuracy of sensor perception, constituting key external physical variables affecting flight safety and trajectory planning. Meanwhile, temporary low-altitude controlled airspace refers to areas with clearly defined time and spatial boundaries designated by management authorities for major event support, emergency rescue, or hazardous materials leaks, prohibiting or restricting entry. Communication spectrum bandwidth characterizes the channel transmission capacity for data exchange between multiple agents and ground base stations or among themselves per unit time. The sufficiency of bandwidth resources determines the transmission delay and throughput of situational awareness information and collaborative control commands. Electromagnetic environment limitations manifest as background radiation interference, signal blockage in the same frequency band, or signal-to-noise ratio degradation caused by multipath effects, severely weakening the positioning accuracy of satellite navigation systems and the anti-interference redundancy of data links. Ground population density limitations map the probability distribution of potential risks to the life and property safety of people on the ground when low-altitude aircraft crash or generate noise pollution. High-density areas correspond to stricter upper limits on flight altitude and no-fly zones. In summary, dynamic environmental factors such as low-altitude air traffic flow, meteorological conditions, temporary low-altitude controlled airspace, communication spectrum bandwidth, electromagnetic environment limitations, and ground population density limitations jointly construct a high-dimensional, nonlinear low-altitude airspace operation constraint system. This provides multi-agent reinforcement learning models with comprehensive state inputs covering macroscopic traffic conditions, microscopic physical interference, spatial isolation policies, and ground safety baselines, forming the environmental foundation for realizing dynamic collaborative allocation methods of airspace resources.

[0027] Based on defined route nodes and established dynamic environmental factors, an airspace simulation environment is comprehensively constructed to simulate real airspace operations. Following the completion of the airspace simulation environment, the state space, representing all possible states of this environment, is further determined.

[0028] Based on the environmental state space, a set of perception state spaces corresponding to the set of intelligent agents is constructed, and a multi-objective reward mechanism is constructed by combining the set of action spaces.

[0029] Furthermore, this application also includes: defining the set of intelligent agents and determining the set of action spaces, wherein each action space includes discrete actions and continuous actions; combining the set of intelligent agents and the environmental state space to determine the set of perception state spaces, wherein each perception state space includes its own state, environmental perception state, and resource perception state; defining multiple objectives and constructing an interaction logic that quantifies the set of action spaces and the set of perception state spaces into positive incentives and negative penalties to generate a multi-objective system, wherein the multiple objectives include task objectives, safety objectives, efficiency objectives, and collaborative objectives; using the set of intelligent agents as the decision-making body, the set of perception state spaces as the decision-making basis, and the multi-objective system as the constraint condition, performing decision deduction based on the set of action spaces to construct the multi-objective reward mechanism.

[0030] Specifically, we define an intelligent agent set, which refers to the totality of multiple independent decision-making units participating in the dynamic collaborative allocation of airspace resources. We also define an action space set, which is the complete set of all actions that each intelligent agent may perform. Each action space specifically includes discrete actions and continuous actions. Discrete actions involve decision-making behaviors with explicit options, such as route selection or communication channel switching. Continuous actions involve operational behaviors that can vary within a continuous range, such as speed adjustment or heading fine-tuning.

[0031] By combining the set of intelligent agents and the environmental state space, a set of perception state spaces is determined. The set of perception state spaces is the set of local information that each intelligent agent can obtain based on its own observation capabilities. Each perception state space includes its own state, i.e., the internal information of the intelligent agent, such as position, velocity and remaining energy; environmental perception state, i.e., the external information detected by the intelligent agent, such as the surrounding airspace traffic conditions and weather conditions; and resource perception state, i.e. the resource information available to the intelligent agent, such as communication spectrum bandwidth and navigation resources.

[0032] We define multi-objectives, which refer to multiple objective dimensions that need to be optimized simultaneously during the allocation of airspace resources. We construct an interactive logic that quantifies the action space set and the perception state space set into positive incentives and negative penalties. Through quantification, we generate a multi-objective system. The multi-objective system specifically includes the mission objective, namely the success rate of completing the flight mission; the safety objective, namely the degree of guarantee of maintaining a safe distance; the efficiency objective, namely the efficiency of airspace use and time optimization; and the cooperation objective, namely the degree of cooperation between intelligent agents.

[0033] Subsequently, using the set of intelligent agents as the decision-making body, the set of perceptual state spaces as the decision-making basis, and the multi-objective system as the constraint, a decision deduction based on the action space set is carried out. Decision deduction refers to the process by which intelligent agents reason and select within the action space set based on the set of perceptual state spaces and the multi-objective system. Through this process, a multi-objective reward mechanism is constructed. The multi-objective reward mechanism provides an immediate reward signal for each action to guide the intelligent agent in learning how to make the optimal decision under multi-objective constraints.

[0034] The set of intelligent agents is subjected to policy optimization analysis based on the multi-objective reward mechanism to obtain the set of adjustment action space and the set of adjustment perception state space.

[0035] Furthermore, this application also includes: dividing the spatial domain into grids, forming a distributed grid group with a preset number of grids, and distributing the agent set in the distributed grid group according to the set of perception state spaces to obtain a distributed agent set; based on the distributed agent set, analyzing the distributed adjustment action space set and the distributed adjustment perception state space set under the multi-objective reward mechanism within the distributed grid group; and based on the central controller, combining the distributed adjustment action space set and the distributed adjustment perception state space set, analyzing the adjustment action space set and the adjustment perception state space set of the agent set under the multi-objective reward mechanism.

[0036] Furthermore, this application also includes: constructing a distributed topology graph structure based on the set of perception state spaces; performing temporal feature extraction by combining the distributed topology graph structure and the set of action spaces to obtain a distributed temporal dependency feature vector; and obtaining a distributed adjustment action space set and a distributed adjustment perception state space set corresponding to the distributed optimal target value based on the distributed temporal dependency feature vector and according to the multi-objective reward mechanism.

[0037] Furthermore, this application also includes: constructing a graph neural network layer to encode the spatial topological relationships of the agent set based on the distributed topological graph structure to obtain a spatial feature vector; constructing a temporal neural network layer to extract temporal dependencies from the spatial feature vector based on the action sequence of the action space set to obtain a spatiotemporal fusion feature vector; and mapping the spatiotemporal fusion feature vector through a fully connected layer to generate the distributed temporal dependency feature vector.

[0038] Furthermore, this application also includes: evaluating the value of the distributed temporal dependency feature vector based on the multi-objective reward mechanism to obtain a distributed multi-objective value; introducing an ideal target value, generating a distributed multi-objective value to be adjusted based on the value deviation between the distributed multi-objective value and the ideal target value, and mapping it to obtain a distributed temporal dependency feature vector to be adjusted; generating a distributed action space set to be adjusted and a distributed perception state space set to be adjusted based on the distributed action space set to be adjusted and the distributed perception state space set to be adjusted; iteratively adjusting the distributed temporal dependency feature vector to be adjusted based on the distributed action space set to be adjusted and the distributed perception state space set to be adjusted to obtain a distributed optimal target value; and obtaining the distributed adjustment action space set and the distributed adjustment perception state space set corresponding to the distributed adjustment temporal dependency feature vector based on the distributed optimal target value.

[0039] Furthermore, this application also includes: in distributed regulation, the multi-objective reward mechanism takes the first objective as the primary objective, and in clustered regulation, the multi-objective reward mechanism takes the second objective as the primary objective.

[0040] Specifically, the spatial domain is divided into grids, which refers to dividing a continuous spatial domain into multiple discrete grid units. A distributed grid group is formed with a predetermined number of grid units. The agent set is then distributed according to the perceptual state space set, resulting in a distributed agent set. Each agent is assigned to a corresponding grid group based on the local information contained in its perceptual state space set, thus forming a distributed agent set. The distributed agent set refers to the group of agents divided according to the grid distribution, with each agent belonging to a specific grid group to facilitate subsequent local optimization.

[0041] Furthermore, based on a distributed agent ensemble, the distributed adjustment action space set and distributed adjustment perception state space set under a multi-objective reward mechanism are analyzed within a distributed grid cluster. Based on the perception state space set, a distributed topological graph structure is constructed; the perception state space set contains local information observed by each agent, and the distributed topological graph structure refers to a graph structure constructed with agents as nodes and spatial proximity or communication relationships between agents as edges, used to quantitatively describe the spatial distribution and mutual influence relationships of agents within the distributed grid cluster.

[0042] A graph neural network layer is constructed to encode the spatial topological relationships of the agent set based on a distributed topological graph structure, thereby obtaining spatial feature vectors. Here, the graph neural network layer refers to the computational unit using a graph neural network structure; the distributed topological graph structure is a graph structure with agents as nodes and relationships between agents as edges; spatial topological relationships refer to the relative positions and connections of agents in the spatial domain; and the spatial feature vector is the vector representation generated after encoding the spatial topological relationships through the graph neural network layer, containing spatial interaction information between agents.

[0043] A temporal neural network layer is constructed to extract temporal dependencies from spatial feature vectors based on action sequences within the action space set, resulting in a spatiotemporal fusion feature vector. Here, the temporal neural network layer refers to computational units employing structures such as recurrent neural networks or long short-term memory networks; the action sequence is an ordered set of actions performed by the agent at consecutive time steps; temporal dependency extraction captures the temporal dependencies from the action sequence; and the spatiotemporal fusion feature vector is a vector representation generated by fusing the spatial feature vector with the temporal features of the action sequence, containing both the agent's spatial relationship information and the temporal evolution information of the actions.

[0044] The spatiotemporal fusion feature vector is mapped through a fully connected layer to generate a distributed temporally dependent feature vector. The fully connected layer mapping refers to the process of performing linear transformations and nonlinear activations on the input vector using a fully connected neural network. The distributed temporally dependent feature vector is the final feature vector obtained after the fully connected layer mapping, serving as input features for subsequent value evaluation and decision optimization, comprehensively representing the spatiotemporal dependencies of the distributed intelligent agent.

[0045] Value evaluation of distributed temporal dependency feature vectors is performed based on a multi-objective reward mechanism to obtain distributed multi-objective value. The multi-objective reward mechanism refers to a quantitative evaluation system that integrates task objectives, safety objectives, efficiency objectives, and collaborative objectives. The distributed temporal dependency feature vector is a feature representation containing information on the agent's spatial topological relationships and action temporal dependencies. Value evaluation refers to scoring the feature vectors using a pre-defined value function. The distributed multi-objective value is the numerical result output by this evaluation process, quantifying the degree to which the agent's decisions satisfy multi-objective constraints under the current distributed spatiotemporal characteristics.

[0046] An ideal target value is introduced, and a distributed multi-objective value to be adjusted is generated based on the value deviation between the distributed multi-objective value and the ideal target value. This is then mapped to obtain a distributed time-dependent feature vector to be adjusted. Here, the ideal target value refers to the pre-set optimal expected value within the multi-objective reward mechanism framework; the value deviation refers to the difference between the distributed multi-objective value and the ideal target value; the distributed multi-objective value to be adjusted refers to the target value to be optimized generated based on this deviation; mapping refers to transforming the target value to be adjusted back into the feature space through a specific functional relationship; and the distributed time-dependent feature vector to be adjusted refers to the spatiotemporal feature representation that needs further optimization after mapping.

[0047] Based on the distributed time-dependent feature vectors to be adjusted, a distributed action space set and a distributed perception state space set are generated. The distributed action space set refers to the set of action candidates that need to be adjusted corresponding to the feature vector to be adjusted, and the distributed perception state space set refers to the set of perception information that needs to be updated corresponding to the feature vector to be adjusted. Together, they form the input basis for the next iteration of optimization.

[0048] Based on the distributed set of actions to be adjusted and the distributed set of perception states to be adjusted, the distributed time-dependent feature vectors to be adjusted are iteratively adjusted to obtain the distributed optimal target value. Iterative adjustment refers to the process of gradually approaching the optimal state by repeatedly optimizing action selection and perception updates. The distributed optimal target value refers to the best value that satisfies multi-objective constraints after multiple iterative adjustments.

[0049] Based on the distributed optimal target value, obtain the distributed adjustment action space set and the distributed adjustment perception state space set corresponding to the distributed adjustment time-dependent feature vector. Here, the distributed adjustment time-dependent feature vector refers to the final spatiotemporal feature representation corresponding to the optimal target value; the distributed adjustment action space set refers to the final action set determined under this optimal feature; and the distributed adjustment perception state space set refers to the final perception state set determined under this optimal feature.

[0050] Subsequently, based on the central controller, and combining the distributed adjustment action space set and the distributed adjustment perception state space set, the adjustment action space set and adjustment perception state space set of the agent ensemble under the multi-objective reward mechanism are analyzed. The central controller refers to the control unit used for global collaborative decision-making. The adjustment action space set refers to the final action space set determined by the agent ensemble after global analysis by the central controller. The adjustment perception state space set refers to the final perception state space set updated by the agent ensemble after global collaborative optimization, integrating the local optimization results of each distributed grid group to achieve the collaborative allocation of overall airspace resources.

[0051] In distributed regulation, the multi-objective reward mechanism prioritizes the primary objective. Distributed regulation refers to a local optimization process within a distributed grid cluster. The multi-objective reward mechanism is a quantitative evaluation system that integrates task objectives, safety objectives, efficiency objectives, and collaborative objectives. The primary objective refers to the objective dimension assigned the highest weight in this local optimization phase, such as the efficiency objective. Prioritizing the primary objective in distributed regulation means that agents within the grid prioritize optimizing this dimension to quickly respond to local needs.

[0052] In clustered regulation, the multi-objective reward mechanism prioritizes the second objective. Clustered regulation refers to a global collaborative optimization process conducted through a central controller. Similarly, the multi-objective reward mechanism refers to a quantitative evaluation system that includes multiple objective dimensions. The second objective refers to another objective dimension that differs from the first objective, such as a collaborative objective. In clustered regulation, prioritizing the second objective means that during the global collaborative phase, the agent prioritizes optimizing this dimension to achieve coordinated allocation of overall airspace resources.

[0053] Based on the safety interval constraint and the set of adjustment perception state space, conflict detection and resolution are performed on the set of adjustment action space to obtain the target action space set.

[0054] Furthermore, this application also includes: combining the set of adjustment perception states to obtain the predicted trajectory of the set of agents after executing the action corresponding to the set of adjustment action space; performing conflict detection on the predicted trajectory based on the safety interval rule, and triggering a conflict resolution mechanism if a conflict is detected; generating an emergency maneuver action based on the conflict resolution mechanism, and using the emergency maneuver action to cover the corresponding action in the set of adjustment action space, to obtain a safety-optimized target action space set.

[0055] Specifically, by combining the set of adjusted perception states, the predicted trajectory of the agent set after executing actions corresponding to the set of adjusted action spaces is obtained. Here, the set of adjusted perception states refers to the set of local information of the agent updated after policy optimization analysis, the set of adjusted action spaces refers to the set of agent action candidates determined after optimization, and the predicted trajectory refers to the spatial position sequence of the agent in the future time step calculated by the motion model based on the current perception state and preset actions.

[0056] The predicted trajectory is subjected to conflict detection based on safety interval rules. If a conflict is detected, a conflict resolution mechanism is triggered. Safety interval rules refer to the minimum distance standards that must be maintained between agents or between an agent and an obstacle to ensure flight safety. Conflict detection refers to comparing the predicted trajectory with the safety interval rules to determine whether there is a violation of the interval requirements. The conflict resolution mechanism refers to a pre-set set of processing procedures to eliminate conflicts. A conflict refers to an event in the predicted trajectory that violates the safety interval rules.

[0057] Emergency maneuvers are generated based on the conflict resolution mechanism, and these emergency maneuvers are then used to override the corresponding actions in the adjustment action space set, resulting in a safety-optimized target action space set. Emergency maneuvers refer to temporary avoidance maneuvers such as emergency climbs or rapid turns, generated to quickly evade conflict. Overriding refers to replacing the corresponding actions in the original adjustment action space set with emergency maneuvers. The target action space set is the final set of executable actions determined after conflict detection and resolution, ensuring that all actions satisfy safety interval constraints.

[0058] The target action space set is transformed into control command execution, and the actual state feedback data is obtained to update the environmental state space. The execution is then returned to construct the perception state space set, realizing dynamic closed-loop allocation.

[0059] Furthermore, this application also includes: converting the target action space set into control commands and sending them to the corresponding intelligent agents for execution; after a preset time step, obtaining the actual state feedback data of the intelligent agent set after executing the control commands; updating the environmental state space of the spatial simulation environment based on the actual state feedback data, and returning the perception state space set of the intelligent agent set to be constructed, thereby realizing dynamic closed-loop allocation.

[0060] Specifically, the target action space set is transformed into control commands and sent to the corresponding intelligent agents for execution. The target action space set refers to the set of actions finally determined after conflict detection and resolution. Control commands refer to the formatted commands that convert each action in the action space into a form that the intelligent agent can recognize and execute. An intelligent agent refers to an independent decision-making unit in the allocation of airspace resources. Execution refers to the process by which the intelligent agent completes the corresponding operation according to the received control commands.

[0061] After a preset time step, the actual state feedback data after the intelligent agent set executes the control command is obtained; the preset time step refers to the pre-set time interval used to sample the execution results; the intelligent agent set refers to the whole composed of all intelligent agents participating in the allocation of airspace resources; the actual state feedback data refers to the actual state information returned by the intelligent agent after executing the command, including changes in position update speed and resource consumption, etc.

[0062] The environment state space of the spatial simulation environment is updated based on actual state feedback data, and the collection of perception state spaces for constructing the intelligent agent set is returned to achieve dynamic closed-loop allocation. The actual state feedback data is used to correct the state parameters in the spatial simulation environment. The environment state space refers to the information set that represents the global state of the spatial domain. Returning to execution means restarting the construction process of the perception state space set. Dynamic closed-loop allocation refers to the continuous optimization allocation process formed by updating the environment state through execution feedback and restarting the perception and decision-making loop.

[0063] In summary, the spatial resource dynamic collaborative allocation method based on multi-agent reinforcement learning provided in this application has the following technical effects: by achieving the technical goal of constructing a spatiotemporal fusion perception and decision-making mechanism based on distributed topological graph structure and temporal dependency feature extraction, it enhances the comprehensive representation ability of multi-agents on spatial interaction relationships and action evolution laws, improves the global optimality of the set of adjustment action space and the set of adjustment perception state space obtained under the multi-objective reward mechanism, and thus improves the efficiency and security of spatial resource dynamic collaborative allocation.

[0064] The above description of the disclosed embodiments enables those skilled in the art to make or use this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

[0065] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of this application and its equivalents, this application also intends to include such modifications and variations.

Claims

1. A method for dynamic collaborative allocation of spatial resources based on multi-agent reinforcement learning, characterized in that, include: Construct a low-altitude airspace simulation environment and determine the environmental state space; Based on the environmental state space, a set of perception state spaces corresponding to the set of intelligent agents is constructed, and a multi-objective reward mechanism is constructed by combining the set of action spaces. The set of intelligent agents is subjected to policy optimization analysis based on the multi-objective reward mechanism to obtain the set of adjustment action space and the set of adjustment perception state space; Based on the safety interval constraint and the set of adjustment perception state space, conflict detection and resolution are performed on the set of adjustment action space to obtain the target action space set. The target action space set is transformed into control command execution, and the actual state feedback data is obtained to update the environmental state space. The execution is then returned to construct the perception state space set, realizing dynamic closed-loop allocation.

2. The method for dynamic collaborative allocation of spatial resources based on multi-agent reinforcement learning as described in claim 1, characterized in that, Construct a low-altitude airspace simulation environment and determine the environmental state space, including: Delineate the airspace area, import geographic information data, and define route nodes, including origin, landing point and intermediate node; Dynamic environmental factors are set, including low-altitude air traffic flow, meteorological conditions, low-altitude temporary controlled airspace, communication spectrum bandwidth, electromagnetic environment restrictions, and ground population density restrictions. Based on the route nodes and the dynamic environmental factors, the airspace simulation environment is constructed; Based on the spatial simulation environment, the environmental state space is determined.

3. The method for dynamic collaborative allocation of spatial resources based on multi-agent reinforcement learning as described in claim 1, characterized in that, Based on the environmental state space, a set of perception state spaces corresponding to the set of intelligent agents is constructed. A multi-objective reward mechanism is then constructed by combining this set with the action space set, including: Define the set of intelligent agents and determine the set of action spaces, wherein each action space includes discrete actions and continuous actions; By combining the set of intelligent agents and the environmental state space, the set of perception state spaces is determined, wherein each perception state space includes its own state, environmental perception state, and resource perception state. Define multiple objectives and construct an interaction logic that quantifies the action space set and the perception state space set into positive incentives and negative penalties to generate a multi-objective system. The multi-objective system includes task objectives, safety objectives, efficiency objectives, and collaboration objectives. Using the set of intelligent agents as the decision-making body, the set of perception state spaces as the decision-making basis, and the multi-objective system as the constraint, a decision deduction based on the set of action spaces is performed to construct the multi-objective reward mechanism.

4. The method for dynamic collaborative allocation of spatial resources based on multi-agent reinforcement learning as described in claim 1, characterized in that, The set of intelligent agents is subjected to policy optimization analysis based on the multi-objective reward mechanism to obtain a set of regulatory action spaces and a set of regulatory perception state spaces, including: The spatial domain is divided into grids, and a distributed grid group is formed with a preset number of grids. The set of intelligent agents is distributed in the distributed grid group according to the set of perception state space to obtain a distributed set of intelligent agents. Based on the distributed intelligent agent set, the distributed adjustment action space set and the distributed adjustment perception state space set under the multi-objective reward mechanism are analyzed within the distributed grid group. Based on the central controller, and combining the distributed adjustment action space set and the distributed adjustment perception state space set, the adjustment action space set and adjustment perception state space set of the agent set under the multi-objective reward mechanism are analyzed.

5. The method for dynamic collaborative allocation of spatial resources based on multi-agent reinforcement learning as described in claim 4, characterized in that, The analysis of the distributed adjustment action space set and the distributed adjustment perception state space set under the multi-objective reward mechanism includes: Based on the aforementioned set of perception state spaces, a distributed topology graph structure is constructed; By combining the distributed topology graph structure and the action space set, temporal feature extraction is performed to obtain a distributed temporal dependency feature vector; Based on the distributed temporal dependency feature vector, the distributed adjustment action space set and the distributed adjustment perception state space set corresponding to the distributed optimal target value are obtained according to the multi-objective reward mechanism.

6. The method for dynamic collaborative allocation of spatial resources based on multi-agent reinforcement learning as described in claim 5, characterized in that, By combining the distributed topology graph structure and the action space set, temporal feature extraction is performed to obtain a distributed temporal dependency feature vector, including: A graph neural network layer is constructed to encode the spatial topological relationships of the agent set based on the distributed topological graph structure, thereby obtaining a spatial feature vector; A temporal neural network layer is constructed, and temporal dependency extraction is performed on the spatial feature vector based on the action sequence of the action space set to obtain a spatiotemporal fusion feature vector. The spatiotemporal fusion feature vector is mapped through a fully connected layer to generate the distributed temporal dependency feature vector.

7. The method for dynamic collaborative allocation of spatial resources based on multi-agent reinforcement learning as described in claim 5, characterized in that, Based on the distributed temporal dependency feature vector, and according to the multi-objective reward mechanism, the distributed adjustment action space set and the distributed adjustment perception state space set corresponding to the distributed optimal objective value are obtained, including: The distributed time-dependent feature vector is evaluated based on the multi-objective reward mechanism to obtain the distributed multi-objective value. An ideal target value is introduced, and a distributed multi-target value to be adjusted is generated based on the value deviation between the distributed multi-target value and the ideal target value. This is then mapped to obtain a distributed time-dependent feature vector to be adjusted. Based on the distributed time-dependent feature vectors to be adjusted, a set of distributed action spaces to be adjusted and a set of distributed perception states to be adjusted are generated. Based on the distributed set of actions to be adjusted and the distributed set of perception states to be adjusted, the distributed time-dependent feature vectors to be adjusted are iteratively adjusted to obtain the distributed optimal target value. Based on the distributed optimal target value, obtain the distributed adjustment action space set and the distributed adjustment perception state space set corresponding to the distributed adjustment time-dependent feature vector.

8. The method for dynamic collaborative allocation of spatial resources based on multi-agent reinforcement learning as described in claim 4, characterized in that, In distributed regulation, the multi-objective reward mechanism takes the primary objective as the main objective, while in cluster-based regulation, the multi-objective reward mechanism takes the secondary objective as the main objective.

9. The method for dynamic collaborative allocation of spatial resources based on multi-agent reinforcement learning as described in claim 1, characterized in that, Based on the safety interval constraint and the set of adjustment perception states, conflict detection and resolution are performed on the set of adjustment action spaces to obtain the target action space set, including: By combining the set of adjustment perception states, the predicted trajectory of the set of agents after executing the action corresponding to the set of adjustment action spaces is obtained; The predicted trajectory is subjected to conflict detection based on the safety interval rule. If a conflict is detected, a conflict resolution mechanism is triggered. Emergency maneuvers are generated based on the conflict resolution mechanism, and the corresponding actions in the adjustment action space set are covered by the emergency maneuvers to obtain the target action space set after safety optimization.

10. The method for dynamic collaborative allocation of spatial resources based on multi-agent reinforcement learning as described in claim 1, characterized in that, The target action space set is transformed into control command execution, and the environmental state space is updated by acquiring actual state feedback data. The process then returns to construct the perception state space set, achieving dynamic closed-loop allocation, including: The target action space set is converted into control commands and sent to the corresponding intelligent agent for execution; After a preset time step, the actual state feedback data of the intelligent agent set after executing the control command is obtained; The environmental state space of the spatial simulation environment is updated based on the actual state feedback data, and the set of perception state spaces for constructing the intelligent agent set is returned to achieve dynamic closed-loop allocation.

Images