A Sensor-Integrated Collaborative Decision-Making Method for Unmanned Aerial Vehicles Based on an Improved MADDPG Algorithm

Abstract

This invention discloses a collaborative decision-making method for unmanned aerial vehicles (UAVs) based on an improved MADDPG algorithm, belonging to the fields of UAV communication and perception integration and multi-agent reinforcement learning. The method constructs a collaborative decision-making environment with maximum UAV capacity. It identifies currently active UAVs through an activity mask, while masking the states, actions, and training samples of inactive UAVs. A multi-relationship graph is constructed based on the spatial distance relationships, communication connectivity relationships, coverage overlap relationships, and collaborative service relationships of active UAVs. A fixed-dimensional global state representation is generated through state compression, weighted fusion of relationships, and masked average pooling in a multi-relationship graph neural network. This global state representation and the masked effective joint actions are input into a centralized commentator network to train the collaborative control strategy. This method maintains a stable input dimension of the decision network even when the number of UAVs dynamically increases, and is suitable for communication coverage, perception coverage, and collaborative trajectory control tasks.

Description

Technical Field

[0001] This invention belongs to the fields of UAV swarm collaborative control, wireless communication and sensing integration, and artificial intelligence decision-making technology, and particularly relates to a UAV integrated sensing and collaborative decision-making method based on an improved MADDPG algorithm. Technical Background

[0002] With the development of applications such as low-altitude economy, emergency communication, intelligent inspection, and target monitoring, drone swarms have gradually become an important component of integrated air-space-ground networks due to their flexible deployment, high mobility, and adjustable coverage. Sensor-communication integration technology, which combines communication services and environmental perception on a single platform, effectively improves spectrum resource utilization and task collaboration efficiency. Therefore, sensor-communication integration systems with multiple drones have application value in scenarios such as regional coverage, user services, target perception, and emergency support.

[0003] In real-world missions, the size of a drone swarm often varies due to changes in the mission area, the number of users served, communication quality, or temporary mission requirements. For example, when the mission area expands or local service capabilities become insufficient, the system may need to dynamically add drone members to improve communication coverage and perception service capabilities. Such scenarios, where the number of active drones increases, cause multi-drone systems to exhibit characteristics such as variable member numbers, changing state space dimensions, and complex collaborative relationships.

[0004] Existing multi-agent reinforcement learning methods typically assume a fixed number of agents. In a centralized training, distributed execution framework, the critic network often receives fixed-dimensional global state and joint action inputs. Therefore, when the number of drones changes, traditional methods require redesigning the network input structure or retraining the model. If simple pooling or similar methods are used to handle variable-scale states, problems may arise such as difficulty in adapting the input dimension, increased redundant information, or insufficient representation of spatial relationships and cooperative dependencies between drones.

[0005] For multi-UAV sensing integration, factors such as the relative positions of UAVs, coverage areas, communication links, sensing ranges, and service user distribution all collectively affect system performance. Therefore, how to map the variable-scale state of multiple UAVs into a fixed-dimensional global representation while preserving key spatial relationships and task-related information as much as possible, given the dynamic change in the number of UAVs, is a technical problem that needs to be solved in this field. Summary of the Invention

[0006] The purpose of this invention is to provide a UAV sensory integration collaborative decision-making method based on an improved MADDPG algorithm, in order to solve the problems of existing multi-agent reinforcement learning methods, such as difficulty in maintaining fixed-dimensional state input, difficulty in fully expressing spatial relationships and collaborative dependencies between UAVs, and insufficient stability of policy training and execution after the number of UAVs increases dynamically.

[0007] To achieve the above objectives, this invention provides a UAV sensor-integrated collaborative decision-making method based on an improved MADDPG algorithm, comprising the following steps:

[0008] Step 1: Construct an integrated collaborative decision-making model for drones, setting the maximum number of drones, drone set, user set, task area, drone motion model, communication model, perception model, reward function, and constraints such as flight boundary, collision distance, speed, energy, communication resources, and task area, where users are the communication service objects and perception coverage objects.

[0009] Step 2: Build an activity member management mechanism for the dynamic increase of drone numbers, set the maximum number of drones, the initial number of active drones, and the drone joining event sequence, and use an activity mask to identify whether each drone is active at the current moment.

[0010] Step 3: Determine the current active UAV set based on the activity mask, use only active UAVs as graph nodes, obtain the local observation status of each active UAV, and construct a multi-relationship graph based on the spatial distance relationship, communication connectivity relationship, coverage overlap relationship and cooperative service relationship between active UAVs.

[0011] Step 4: Input the multi-relationship graph into the multi-relationship graph neural network state compression module, perform node feature encoding, relationship-specific message passing, and relationship weighted fusion to obtain the node representation of each active UAV.

[0012] Step 5: Perform mask pooling on the node representation of each active drone according to the activity mask to generate a global state representation with fixed dimensions, so as to mask the dynamic changes in the number of active drones.

[0013] Step 6: Under the multi-agent reinforcement learning framework of centralized training and decentralized execution, the agent network of each active UAV outputs actions according to the corresponding local observation state, and aggregates the action features of the active UAVs according to the activity mask to generate a fixed-dimensional effective joint action representation after masking.

[0014] Step 7: Input the fixed-dimensional global state representation and the masked fixed-dimensional effective joint action representation into the centralized critic network to obtain the action value function, and mask the invalid states, invalid actions and invalid samples corresponding to inactive drones during the critic network update and executor network update process according to the activity mask.

[0015] Step 8: Update the system status based on the actions output by each active drone, calculate the system reward, store the training data in the experience replay pool, and output a drone cooperative control strategy applicable to scenarios where the number of drones expands from the initial active number to the maximum number, based on the training termination condition.

[0016] In a specific implementation, the activity mask is used to determine the set of currently active drones; inactive drones do not participate in the calculation of valid samples in action selection, communication rate calculation, perception coverage calculation, reward calculation, motion state update and experience replay, while active drones participate in state update, communication calculation, perception calculation, reward calculation and policy learning.

[0017] In a specific implementation, the multi-relationship graph includes spatial distance relationships, communication connectivity relationships, coverage overlap relationships, and collaborative service relationships. For each relationship type, a corresponding edge feature matrix is ​​constructed, and relationship-specific parameters are used to perform message passing and neighbor aggregation on information between adjacent active UAV nodes. Then, the node update results obtained under different relationship types are fused by weighted summation to obtain a UAV node representation containing multiple types of collaborative dependency information.

[0018] In a specific implementation, the activity mask pooling filters the representations of active drone nodes based on the activity mask, and performs average pooling normalized to the number of active drones on the active node representations to obtain a fixed-dimensional global state representation; wherein inactive drone nodes do not participate in the pooling summation and pooling denominator calculation.

[0019] In other alternative implementations, the relationship graph can be constructed using an adjacency matrix, the relationship fusion can be achieved through splicing fusion, attention fusion, or gating fusion, and the activity mask pooling can be achieved through weighted pooling or attention pooling. The above alternative methods do not change the core technical concept of this invention: to filter active drones through activity masks, to obtain a fixed-dimensional global state representation through a multi-relationship graph neural network, and to access a centralized commentator network.

[0020] In specific implementations, system rewards include communication rate, perception coverage, user distance gains, and penalties; weighting coefficients are set according to task preferences, which include communication priority, perception priority, and security constraint priority.

[0021] Compared with the prior art, the present invention has at least the following beneficial effects:

[0022] First, by using an activity mask to distinguish between active and inactive drones in a dynamically sized drone swarm, the invention avoids interference from inactive members in action selection, state representation, reward calculation, and sample training. Second, by using a multi-relationship graph neural network and masked pooling to map the states of a variable number of drones to a fixed-dimensional global state representation, the centralized commentator network can maintain stable input when the number of drones changes dynamically. Third, by jointly modeling inter-drone collaborative dependencies through spatial distance relationships, communication connectivity relationships, coverage overlap relationships, and collaborative service relationships, the invention reduces the loss of key collaborative information compared to simple concatenation, zero-padding, or ordinary mean pooling. Fourth, this invention is applicable to multi-drone sensing scenarios and can improve communication coverage, sensing service capabilities, and collaborative decision-making stability under scalable conditions. Attached Figure Description

[0023] Figure 1 This is a schematic diagram of a dynamic expansion multi-UAV sensor integration application scenario according to an embodiment of the present invention.

[0024] Figure 2 This is an algorithm framework diagram based on active masking and multi-relationship graph neural network state compression according to an embodiment of the present invention.

[0025] Figure 3 This is a schematic diagram of the dynamic member management and activity mask update mechanism in an embodiment of the present invention.

[0026] Figure 4 This is a schematic diagram of the multi-relationship graph neural network state compression model structure according to an embodiment of the present invention.

[0027] Figure 5 This is a flowchart of a collaborative decision-making method based on activity mask and multi-relationship graph neural network according to an embodiment of the present invention.

[0028] Figure 6 This is a schematic diagram of the dynamic expansion simulation experiment area and event configuration according to an embodiment of the present invention.

[0029] Figure 7 This is a graph showing the changes in training rewards and coverage rates using multiple methods according to an embodiment of the present invention.

[0030] Figure 8 This is a graph showing the comparison between the average reward and average coverage rate for the final 100 rounds in this embodiment of the invention. Detailed Implementation

[0031] To provide a clearer understanding of the technical features, objectives, and beneficial effects of the present invention, the technical solution of the present invention will now be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, embodiments of the present invention, and should not be construed as limiting the scope of implementation of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the protection scope of the present invention.

[0032] Example 1: This example proposes a UAV sensory integration collaborative decision-making method based on an improved MADDPG algorithm, and applies it to a sensory integration simulation experiment scenario with a scalable number of UAVs. It solves the problems of inconsistent input dimensions of the centralized commentator network, interference of training samples by inactive members, and the difficulty in preserving spatial relationships and collaborative dependencies between UAVs when the number of UAVs increases.

[0033] A schematic diagram of the research problem scenario is shown below. Figure 1 As shown, the algorithm framework diagram is as follows: Figure 2 As shown, the dynamic member management and activity mask update mechanism is as follows: Figure 3 As shown, the structure of the multi-relationship graph neural network state compression model is as follows: Figure 4 As shown, the collaborative decision-making method process is as follows: Figure 5 As shown, the dynamic expansion simulation experiment area and event configuration are as follows: Figure 6 As shown, the simulation results are as follows: Figure 7 and Figure 8 As shown. Figure 1 This invention is used to illustrate the application background of air-space-ground collaborative technology and the issue of dynamic expansion. Figure 6 This is used to illustrate the region scale, dynamically added event sequences, and activity mask change methods used to verify the algorithm in this embodiment.

[0034] Part 1: Description of the simulation experiment scenario and parameter settings.

[0035] Step 1: Establish as follows Figure 6 The simulation experimental area is shown in the figure. In this embodiment, the simulation area is set as a square area of ​​200 m × 200 m, and x ( / m) and y ( / m) are used as plane coordinate axes to describe the relative positions of the UAV and the ground user in the two-dimensional plane. Figure 6 This is used to illustrate the region scale, dynamically added event sequence, and activity mask change method used in the algorithm verification of this embodiment.

[0036] Step 2: Set the number of drones and the conditions for increasing the number. Let the maximum number of drones be... The initial number of active drones is The drone's flight altitude is The communication coverage radius is The sensing coverage radius is The maximum number of simulation steps per round is 100. Initial time... to In an active state, to For drones to be added; trigger the addition event at simulation steps 10, 20, 30, 40, 50, 60, and 70 respectively, so that... to They are then added to the collaborative decision-making process in sequence.

[0037] Step 3: Set the simulation node locations. The locations of the UAV and ground users are randomly initialized within the simulation area, and the number of ground users is M=20. Figure 6 The node positions in the diagram are only used to visually illustrate the spatial relationship between the drone and the user within the simulation area, and do not indicate that all experimental runs will use fixed coordinates. This embodiment uses a fixed maximum number of drones, an initial number of active drones, dynamically added event sequences, and activity mask update rules to ensure that algorithm training and result comparison have clear and reproducible experimental boundaries.

[0038] Step 4: Define the drone set, user set, and activity mask. Let the drone set be... The user set is In any simulation step The activity mask is represented as ,in Indicates the first The drone is an active drone. Indicates the first The drone is an inactive drone.

[0039] For example, initially After the expansion event is completed, the activity mask is updated to...

[0040]

[0041] In this embodiment, the activity mask and the set of active drones satisfy the following:

[0042]

[0043]

[0044] In the formula, This represents the set of active drones in the k-th simulation step. This represents the maximum number of drones.

[0045] Step 5: Establish motion, communication, and perception sub-models. The UAV state includes at least one or more of position, velocity, heading, and remaining energy; UAV actions include one or more of flight direction, velocity adjustment, communication service selection, perception task selection, or power allocation. For communication services, the system estimates the communication link quality or communication rate based on the distance between the UAV and the user or base station, channel gain, bandwidth resources, and transmit power; for target perception, the system determines whether the target is effectively perceived and covered based on the distance between the UAV and the target, perception radius, field of view, and coverage status.

[0046] Let the i-th drone be... The planar position in the k-th simulation step is Covering objects The planar position is The planar distance between the drone and the covered object is expressed as:

[0047]

[0048] In the formula, Let be the planar position vector of the i-th UAV in the i-th simulation step. This is the position vector of the covered object; the distance is calculated using Euclidean distance.

[0049] When the overriding object Located within the communication coverage radius Within this timeframe, the communication signal-to-noise ratio, link rate, and total system rate are calculated as follows:

[0050]

[0051]

[0052]

[0053] The range of summation is

[0054] In the formula, For transmission power, For noise power, For link bandwidth, This is the path loss index. As a smoothing factor; when the distance Exceeding the communication coverage radius At that time, the corresponding link is not included in the effective communication rate.

[0055] Override object The perceived coverage status and coverage rate are calculated as follows:

[0056]

[0057] The range of summation is

[0058] In the formula, To sense the coverage radius, To sense the signal-to-noise ratio threshold. Indicates whether the object being overridden has been effectively overridden.

[0059] Step Six: Establish the reward function and constraints. The reward function includes communication rate rewards, perception coverage rewards, user service distance gains, and safety constraint penalties. Constraints include UAV flight boundary constraints, collision constraints, speed constraints, energy constraints, communication resource constraints, and mission area constraints. By adjusting the reward weights, task preferences such as communication priority, perception priority, or safety constraint priority can be achieved.

[0060] The global reward function used in this embodiment Represented as:

[0061]

[0062]

[0063] The range of summation is

[0064]

[0065] In the formula, The user distance benefit increases as the average distance from the user to the nearest active drone decreases; To normalize communication revenue; It is a comprehensive penalty for violations of flight boundaries, collision risks, speed limits, energy consumption, and communication resource constraints; L is the mission area reference scale.

[0066] Part Two: The MADDPG Collaborative Decision-Making Training Process under Scalable Conditions.

[0067] Step 7: Initialize the environment state, activity mask, executor network, critic network, target network, and experience replay pool. When And when the initial number of active drones is 3, the activity mask is initialized to... ,in to To activate drones, to This refers to inactive drones. Inactive drones do not participate in action selection, state updates, reward calculation, or training with valid samples.

[0068] Step 8: Check for drone joining events at each time step. These joining events can be triggered by a preset task script, insufficient communication coverage, increased target perception requirements, local service quality degradation, or expansion commands. When the event scheduler detects a join event at simulation steps 10, 20, 30, 40, 50, 60, and 70 respectively and confirms that the number of currently active drones is less than [a certain threshold], [the event scheduler will proceed]. At that time, the mask bit corresponding to the drone is updated from 0 to 1, so that the active mask changes from... Gradually updated to .

[0069] Step 9: Generate the set of currently active drones based on the updated activity mask. And obtain the local observation status of active UAVs. For For drones, their local observations are not involved in the construction of the multi-relationship graph, their actions are set to zero or ignored, their reward contributions are not included in the effective reward, and their state transitions are not used as effective training samples; for The observations, actions, rewards, and state transitions of the drones are all used as valid information in training and execution.

[0070] Step 10: Construct a multi-relationship graph based on the active drone set .in For an active collection of drone nodes, Let be the set of edges. This is a set of relationship types. The multi-relationship graph includes spatial distance relationships, communication connectivity relationships, coverage overlap relationships, and cooperative service relationships. Spatial distance relationships are used to describe the geometric proximity between UAVs, communication connectivity relationships are used to describe cooperative communication reachability, coverage overlap relationships are used to describe the degree of overlap of communication or sensing coverage areas, and cooperative service relationships are used to describe the task associations formed when multiple UAVs jointly serve the same user, the same area, or the same target.

[0071] Step 11: Fixed-dimensional state compression is performed using a multi-relationship graph neural network state compression module. First, the local observations of each active UAV node are encoded to obtain an initial node representation. Second, relation-specific message passing is performed for spatial distance relations, communication connectivity relations, coverage overlap relations, and cooperative service relations, with each relation type corresponding to a set of trainable parameters. Third, the update results obtained by the same node under different relation types are weighted and fused to obtain a UAV node representation that includes spatial, communication, coverage, and task cooperation information.

[0072] Specifically, let the local observation of the i-th active UAV at time k be... The initial node representation is obtained through the node encoder:

[0073]

[0074] For relation types ,remember For nodes In relationship The next set of neighbors, the first Layer-specific message passing is as follows:

[0075]

[0076]

[0077] in, Let r be the trainable parameters corresponding to relation r. These are the node's own transformation parameters. For relationship Next node For nodes edge weights or attention weights, It is a non-linear activation function. For relationship The candidate node representation below.

[0078] Merge candidate node representations obtained for the same node under different relation types:

[0079]

[0080]

[0081] in, Representing relation type For nodes The fusion weight, for the relationship The corresponding relational attention parameters.

[0082] go through After multi-layered message passing, the compressed representation of the UAV node is obtained:

[0083]

[0084] Step 12: Further obtain a fixed-dimensional global state representation through masked average pooling:

[0085] The summation range of both the numerator and denominator is [missing information].

[0086] In the formula, Let be the node representation of the i-th drone. For a fixed-dimensional global state representation, Let d be the node embedding dimension. In the main implementation, d=128, so regardless of whether there are 3 or 10 active drones, the pooled output is a 128-dimensional state vector; when the denominator is 0, the output is a zero vector.

[0087] Step 13: Perform collaborative decision-making training within the improved MADDPG framework, which features centralized training and distributed execution. Each active UAV outputs continuous actions from its corresponding actor network based on its local observations; inactive UAVs have their actions set to zero and are masked using an activity mask. The centralized commentator network receives a fixed-dimensional global state representation. Activity mask Effectively combine action features with fixed dimensions and output them. The value is used to evaluate the current joint actions.

[0088] Step Fourteen: Store the local observations, fixed-dimensional global state representation, effective joint actions, reward, next-time local observations, next-time fixed-dimensional global state representation, and activity mask into the experience replay pool. When sampling training data from the experience replay pool, determine the effective agents and effective action dimensions based on the activity mask; during the commentator network update process, the target commentator network receives the next-time fixed-dimensional global state representation, the next-time activity mask, and the masked target joint actions; during the executor network update process, calculate the policy gradient only for the policy output corresponding to the active drone.

[0089] Specifically, the experience replay samples with activity masks are represented as follows:

[0090]

[0091]

[0092]

[0093]

[0094] In the formula, and Based on the maximum number of drones Aligned observation groups and joint action groups; inactive UAV locations are filled with zero vectors and... Masking during sampling and network updates.

[0095] The continuous action encoding, target value, loss function, and target network soft update of the MADDPG commentator network and actor network are as follows:

[0096]

[0097]

[0098]

[0099]

[0100]

[0101]

[0102]

[0103]

[0104]

[0105]

[0106]

[0107]

[0108] In the formula, and Let represent the executor network and the target executor network corresponding to the j-th UAV, respectively, both of which output continuous actions; This represents the action coding layer or feature projection layer, used to map continuous actions aligned by the maximum number of drones and masked by an activity mask into fixed-dimensional action features. and Let these represent the critic network and the target critic network corresponding to the i-th drone, respectively; Indicates the discount factor; This indicates the end-of-round flag; it is set to 1 when the round ends and 0 otherwise. This represents the activity mask of the i-th drone at time k; This represents the target network soft update coefficient.

[0109] Part Three: Verification of Technical Effectiveness

[0110] Step 15: Run the training process based on the above simulation experimental area and parameter settings, and record one or more of the following indicators: training reward, coverage, loss, number of service users, and total system speed.

[0111] Step Sixteen: As Figure 7 As shown, the training reward curves and training coverage curves are displayed under different state representation configurations. These validation results illustrate the impact of activity masking, multi-relationship graph state compression, and fixed-dimensional critic input on training stability and collaborative coverage.

[0112] Step 17: As Figure 8 As shown, the average reward and average coverage of different methods in the final training phase are compared. These results are used to verify the performance differences between the activity mask and the multi-relationship graph neural network state compression mechanism described in this invention, relative to the mask mean and distance graph GNN baseline.

[0113] Step 18: Combining Figure 7 and Figure 8 It is known that in scenarios where the number of active drones increases, relying solely on input from a fixed number of agents or unmasked state representations is insufficient to stably adapt to changes in the number of drones. This invention improves training stability and collaborative decision-making performance in dynamically expanding scenarios by using an activity mask to shield inactive drones, extracting spatial, communication, coverage, and collaborative service relationships through a multi-relationship graph neural network, and accessing a centralized commentator network through a fixed-dimensional state representation.

[0114] Step 19: The simulation results above demonstrate that the activity mask and multi-relationship graph state compression mechanism proposed in this invention can maintain the stability of the training input structure when the number of UAVs dynamically increases, and provides a scalable state representation method for multi-UAV integrated sensory collaborative decision-making. Since different experimental parameters, reward weights, random seeds, and task preferences will affect the training curve and final statistical results, the number of UAVs, communication radius, perception radius, joining time, reward weights, and network structure parameters can be adjusted according to task requirements during actual deployment.

[0115] This embodiment uses an active mask throughout the entire process of dynamic expansion of multiple drones, expresses spatial, communication, coverage, and collaborative service relationships through a multi-relationship graph neural network, and outputs a fixed-dimensional global state representation through mask pooling, so that the centralized commentator network maintains stable input when the number of drones changes; at the same time, inactive drones do not participate in actions, rewards, and training, thereby reducing the interference of invalid members on collaborative decision-making training.

[0116] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments, and the embodiments and descriptions in the specification are merely illustrative of the principles of the invention. Various changes and modifications can be made to the present invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of the present invention as claimed. The scope of protection of the present invention is defined by the appended claims and their equivalents.

Claims

1. A sensor-integrated collaborative decision-making method for unmanned aerial vehicles (UAVs) based on an improved MADDPG algorithm, characterized in that, Includes the following steps: Step 1: Construct an integrated collaborative decision-making model for drones, setting the maximum number of drones, drone set, user set, task area, drone motion model, communication model, perception model, reward function, and constraints on flight boundary, collision distance, speed, energy, communication resources, and task area, wherein the users are the communication service objects and perception coverage objects; Step 2: Construct an activity member management mechanism for the dynamic increase of drone numbers, set the maximum number of drones, the initial number of active drones, and the drone joining event sequence, and use an activity mask to identify whether each drone is active at the current moment; Step 3: Determine the current active UAV set based on the activity mask, use only active UAVs as graph nodes, obtain the local observation status of each active UAV, and construct a multi-relationship graph based on the spatial distance relationship, communication connectivity relationship, coverage overlap relationship and cooperative service relationship between active UAVs; Step 4: Input the multi-relationship graph into the multi-relationship graph neural network state compression module, perform node feature encoding, relationship-specific message passing, and relationship weighted fusion to obtain the node representation of each active UAV; Step 5: Perform mask pooling on the node representation of each active drone according to the activity mask to generate a global state representation with fixed dimensions, so as to mask the dynamic changes in the number of active drones. Step 6: Under the multi-agent reinforcement learning framework of centralized training and decentralized execution, the agent network of each active UAV outputs actions according to the corresponding local observation state, and aggregates the action features of the active UAVs according to the activity mask to generate a fixed-dimensional effective joint action representation after masking. Step 7: Input the fixed-dimensional global state representation and the masked fixed-dimensional effective joint action representation into the centralized critic network to obtain the action value function, and according to the activity mask, mask the invalid states, invalid actions and invalid samples corresponding to inactive drones during the critic network update and executor network update process; Step 8: Update the system status based on the actions output by each active drone, calculate the system reward, store the training data in the experience replay pool, and output a drone cooperative control strategy applicable to scenarios where the number of drones expands from the initial active number to the maximum number, based on the training termination condition.

2. The UAV sensor-integrated collaborative decision-making method based on the improved MADDPG algorithm according to claim 1, characterized in that, Step 1 specifically includes the following sub-steps: Step 11: Set the maximum number of drones Initial number of active drones Number of users Task Area Communication range Perception range and task duration ; Step 12: Construct the drone motion model, based on the... The drone at time t ,speed and actions To determine the position and velocity at the next moment; Step 13: Construct a communication model and calculate the communication signal-to-noise ratio and system communication rate based on the distance between the drone and the user; Step 14: Construct a perception model and calculate the perception signal-to-noise ratio and perception coverage based on the distance between the drone and the user; Step 15: Construct a reward function to evaluate the effectiveness of collaborative decision-making based on communication rate, perception coverage, user distance gains, and penalties for flight boundaries, collision distance, speed, energy, and communication resources.

3. The UAV sensor-integrated collaborative decision-making method based on the improved MADDPG algorithm according to claim 2, characterized in that, The reward function in step 15 is a weighted combination as follows, with further limitations on each evaluation item and weight coefficient: (1) in, Indicates perceived coverage. Indicates user distance revenue, and , This represents the average distance from each user to the nearest active drone, where L represents the reference scale of the mission area. Increases as the distance between the drone and the user decreases; Indicates the communication rate or normalized communication rate. Indicates a penalty item. , , and These are non-negative weighting coefficients corresponding to perceived coverage, user distance gain, communication rate or normalized communication rate, and penalty term, respectively; the weighting coefficients are set according to task preferences, which include communication priority, perception priority, and security constraint priority.

4. The UAV sensor-integrated collaborative decision-making method based on the improved MADDPG algorithm according to claim 1, characterized in that, Step 2 specifically includes the following sub-steps: Step 21: Set the maximum number of drones and the initial number of active drones ,in ; Step 22: Define the activity mask: (2) in, This indicates that the i-th drone is active at time t. This indicates that the i-th drone is inactive at time t; Step 23: Determine the set of currently active drones based on the activity mask: (3) Step 24: Set up the drone to join the event sequence: (4) in, This indicates the time when the k-th joining event occurs. Indicates the drone number added to the system. Indicates that the drone has been added to the event type; Step 25: When the drone joins the event at time t, the number will be... The activity mask value of the drone is updated to 1, and it is added to the currently active drone cluster.

5. The UAV sensor-integrated collaborative decision-making method based on the improved MADDPG algorithm according to claim 4, characterized in that, Inactive drones do not participate in action selection, communication rate calculation, perception coverage calculation, reward calculation, motion state update in environmental state update, and effective sample calculation in experience replay. Active drones participate in action selection, state update, communication calculation, perception calculation, reward calculation, and policy learning.

6. The UAV sensor-integrated collaborative decision-making method based on the improved MADDPG algorithm according to claim 1, characterized in that, Step 3 specifically includes the following sub-steps: Step 31: Determine the current active drone set based on the activity mask, while blocking inactive drones; Step 32: Obtain the local observation status of the i-th UAV: (5) in, Indicates the position of the i-th drone. Indicates speed, Indicates battery level. Indicates characteristics related to boundary states, collision states, or safety constraints. This represents the perceived signal-to-noise ratio characteristic. Indicates user connection state characteristics; Step 33: Treat active UAVs as graph nodes and encode the local observation state into node features through a fully connected coding layer; Step 34: Construct a multi-relationship graph based on the various relationships between active drones: (6) in, Represents the set of active drone nodes. Represents the set of relation edges. This represents a set of relationship types, including spatial distance relationships, communication connectivity relationships, coverage overlap relationships, and collaborative service relationships. Among them, spatial distance relationships are used to characterize geometric proximity, communication connectivity relationships are used to characterize link reachability, coverage overlap relationships are used to characterize overlapping coverage areas, and collaborative service relationships are used to characterize associations that jointly serve the same user or task object.

7. A UAV sensor-integrated collaborative decision-making method based on an improved MADDPG algorithm according to claim 1, characterized in that, Step 4 specifically includes the following sub-steps: Step 41: Construct a multi-relation graph neural network including a node initialization encoding layer, a multi-relation message passing layer, and a masked pooling layer. Perform node initialization encoding on the local observation states of each active UAV to obtain the node representation of layer 0. ,set up This indicates the number of message passing layers in a multi-relationship graph neural network; Step 42: For the set of relation types Each relationship in Set the corresponding relationship message encoding parameters and node update parameters respectively; for relationships Based on the node representations of the i-th and j-th nodes under this relationship and the characteristics of the relationship edges... Generate relationship messages and aggregate neighbor node messages to obtain the relationship. The corresponding node update result; Step 43: Concatenate the node update results corresponding to spatial distance relationship, communication connectivity relationship, coverage overlap relationship and cooperative service relationship along the feature dimension to obtain a multi-relationship concatenation representation, and input the multi-relationship concatenation representation into the relationship fusion layer, and obtain the fused UAV node representation through linear mapping and nonlinear activation function; Step 44: Perform masked average pooling on the fused UAV node representation based on the activity mask, or in an optional implementation, perform weighted masked pooling by combining preset weights or importance weights to obtain a fixed-dimensional global state representation. The pooling denominator only counts active drones or their effective weights, ensuring that the output dimension does not dynamically increase or decrease with the number of active drones.

8. A UAV sensor-integrated collaborative decision-making method based on an improved MADDPG algorithm according to claim 7, characterized in that, The mask pooling process in step 44 includes: Step 441: Filter the active drone node representations based on the activity mask; Step 442: In the main embodiment, masked average pooling is performed on the representation of each active drone node; in the optional embodiment, weighted summation is performed according to preset weights or importance weights, and normalization is performed according to the number of effective weights to obtain a fixed-dimensional global state representation. Step 443: When no importance weight is set or all importance weights are 1, the weighted mask pooling degenerates into mask average pooling normalized according to the number of active drones.

9. A UAV sensor-integrated collaborative decision-making method based on an improved MADDPG algorithm according to claim 1, characterized in that, The distributed execution action output process in step 6 includes the following sub-steps: Step 61: Build an enforcer network for each drone; Step 62: During the distributed execution phase, currently active drones The executor network is based on the corresponding local observation state Output Action : (7) in, This represents the action output by the i-th drone at time t. This represents the local observation state of the i-th UAV. Denotes the executor network of the i-th drone. This represents the network parameters of the executor.

10. A UAV sensor-integrated collaborative decision-making method based on an improved MADDPG algorithm according to claim 1, characterized in that, Steps 7 and 8 specifically include the following sub-steps: Step 71: Update the position, speed, and mission status of the active drones based on the joint actions; Step 72: Calculate the communication rate, sensing coverage, user distance gain, and penalty based on the updated drone and user locations; Step 73: Calculate the system reward based on the reward function; Step 74: Store the current local observations aligned to the maximum number of drones N, the current activity mask, the fixed-dimensional global state representation, the joint actions aligned to the maximum number of drones N, the reward, the next time-instance local observations aligned to the maximum number of drones N, and the next time-instance activity mask into the experience replay pool; among them, the observation and action positions corresponding to inactive drones are filled with zero vectors and masked in sampling and network updates using the activity mask: (8) Step 75: Sample training data from the experience replay pool, filter effective observations and effective actions according to the activity mask during the intensive training phase, construct and update the centralized critic network, and update the actor network; The centralized critic network represents a fixed-dimensional global state. Effective joint action representation of fixed dimensions after convergence of activity masks As input, we obtain the action value function corresponding to the i-th drone: (9) in, This represents the commentator network corresponding to the i-th drone. This represents a fixed-dimensional global state representation. This represents a fixed-dimensional effective joint action feature; the fixed-dimensional effective joint action feature is formed by aligning the continuous actions of each active UAV according to the maximum number of UAVs N. Instead of directly averaging the physical motion quantities, the inactive drone motion positions are zero-filled and masked according to the activity mask, and then input into the motion encoding layer or feature projection layer. Indicates the network parameters of the critics; Step 76: Update the target network using a soft update method: (10) in, Indicates the current network parameters. This indicates the target network parameters before this soft update. This represents the updated target network parameters. Indicates the soft update coefficient; Step 77: Determine whether the training termination condition has been met. If it has, output the UAV cooperative control strategy; otherwise, return to step 3 to continue training.

Images