A heavy-load unmanned helicopter cooperative hoisting method based on multi-agent learning

Abstract

The application discloses a kind of heavy load unmanned helicopter collaborative hoisting methods based on multi-agent learning, comprising the following steps: constructing multi-agent environment state vector;For each unmanned helicopter, establish agent model;Using improved CPPN network, the connection structure and initialization parameter of multi-agent strategy network are obtained, and the agent model is initialized;Multi-agent learning training is carried out to the initialized agent model, and multi-agent strategy model is formed;Collaborative control command is generated through preset index rule;Continuously collect updated multi-agent environment state vector, generate updated collaborative control command;Form collaborative hoisting process data;Improved CPPN network and multi-agent strategy model are updated, and stable collaborative control and adaptive optimization of multiple unmanned helicopters in heavy load hoisting scene are realized.

Description

Technical Field

[0001] This invention relates to the field of cooperative control of unmanned helicopters, and in particular to a cooperative lifting method for heavy-duty unmanned helicopters based on multi-agent learning. Background Technology

[0002] With the development of unmanned helicopter technology, its application in high-altitude operations, transportation in complex terrains, emergency rescue, and hoisting in special environments is becoming increasingly widespread. Especially in heavy-load transportation scenarios, the load capacity of a single unmanned helicopter is limited by structural strength, power performance, and its own stability. Therefore, the coordinated hoisting of the same target load by multiple unmanned helicopters has gradually become an effective way to improve transportation capacity and operational efficiency. However, during coordinated hoisting, multiple unmanned helicopters need to maintain attitude synchronization, position coordination, traction balance, and load stability simultaneously. The control process exhibits characteristics such as multivariable coupling, strong time-varying nature, and high dynamic complexity, making it difficult for traditional coordinated hoisting control methods to meet the comprehensive requirements of real-time performance, safety, and stability.

[0003] In existing technologies, collaborative lifting by multiple unmanned helicopters typically relies on manually designed control rules or centralized control strategies. For example, by pre-setting formation structures or trajectory control formulas, the unmanned helicopters maintain a relatively fixed positional relationship during collaboration, and a simple traction force distribution strategy is used to control the load attitude. However, such methods are difficult to adjust in real time according to environmental changes, load swing characteristics, or dynamic differences between unmanned helicopters. In the presence of wind disturbances, changes in load swing amplitude, or deviations in the execution of a particular unmanned helicopter, the overall collaborative control can easily become unstable, reducing the safety and reliability of the system. Summary of the Invention

[0004] One objective of this invention is to propose a collaborative lifting method for heavy-load unmanned helicopters based on multi-agent learning. This invention fully utilizes multi-agent learning technology, an improved CPPN network generation mechanism, and unmanned helicopter collaborative control technology. By constructing multi-agent environmental state vectors, establishing agent models, generating the connection structure and initialization parameters of the multi-agent policy network, executing multi-agent learning training, and generating collaborative control commands, it achieves autonomous collaboration and dynamic adaptation of multiple unmanned helicopters in heavy-load lifting tasks.

[0005] A collaborative lifting method for heavy-duty unmanned helicopters based on multi-agent learning according to an embodiment of the present invention includes the following steps:

[0006] The flight status information, position and attitude information, sling force information, and target load mass and position parameters of multiple unmanned helicopters are collected to construct a multi-agent environmental state vector.

[0007] Based on the multi-agent environmental state vector, an agent model is established for each unmanned helicopter;

[0008] The multi-agent environment state vector is input into the improved CPPN network to obtain the connection structure and initialization parameters of the multi-agent policy network. The initialization parameters are then allocated to each agent model to form the initial agent model.

[0009] Based on the connection structure of the multi-agent policy network, the initial agent model is trained by multi-agent learning to form a multi-agent policy model.

[0010] Based on a multi-agent policy model, collaborative control instructions are generated through preset indexing rules.

[0011] The system executes flight control on multiple unmanned helicopters according to the cooperative control commands. During the cooperative lifting process, it continuously collects and updates the multi-agent environmental state vectors and inputs them into the multi-agent policy model to generate updated cooperative control commands.

[0012] Based on the updated collaborative control commands, the attitude, traction output, and relative position of multiple unmanned helicopters are adjusted in real time to form collaborative lifting process data;

[0013] The improved CPPN network and multi-agent policy model are updated based on the data from the collaborative hoisting process.

[0014] Optionally, the multi-agent environment state vector is obtained by mapping flight state information, position and attitude information, sling force information, and the mass and position parameters of the target load into corresponding state sub-vectors and concatenating them in a preset order.

[0015] Optionally, the establishment of the intelligent agent model specifically includes:

[0016] The state sub-vectors in the multi-agent environment state vector are defined as the state space of the unmanned helicopter.

[0017] The motion of the unmanned helicopter is defined as a motion vector, which consists of attitude adjustment commands, position correction commands, and traction output commands. The attitude adjustment commands refer to adjusting the attitude of the unmanned helicopter by controlling its pitch, roll, and yaw angles. The position correction commands refer to adjusting the forward, backward, left, right, and up / down positions of the unmanned helicopter in three-dimensional space based on the deviation between the current position and the target position. The traction output commands refer to calculating the applied traction force based on the weight of the load, the stress on the sling, and the state of the unmanned helicopter. The calculation process involves calculating the magnitude of the traction force required for each unmanned helicopter using a preset mechanical model. The magnitude of the traction force includes the magnitude of gravity, the magnitude of inertial force, and the magnitude of sling tension. The traction force applied by the unmanned helicopter is controlled by adjusting the hovering height and attitude angle.

[0018] Based on the requirements of the collaborative hoisting mission and the performance parameters of multiple unmanned helicopters, value ranges are set for attitude adjustment commands, position correction commands, and traction output commands, and the set of all motion vectors is defined as the motion space of the unmanned helicopter.

[0019] The state space and action space of each unmanned helicopter are mapped and associated to form a corresponding intelligent agent model.

[0020] Optionally, the formation of the initial intelligent agent model specifically includes:

[0021] The multi-agent environment state vector is input into the improved CPPN network, which includes a task feature and structural constraint encoding module, a policy network topology generation module, and a policy network parameter initialization and structure evaluation module. The task feature and structural constraint encoding module performs time series aggregation and importance weighted encoding on the multi-agent environment state vector to obtain task feature sub-vectors. The policy network topology generation module performs threshold screening and structural pruning on the candidate connection strength matrix according to preset sparsity constraints and structural symmetry constraints to obtain the connection structure of the multi-agent policy network. The policy network parameter initialization and structure evaluation module generates the initialization parameters of the multi-agent policy network using a meta-parameter mapping initialization mechanism.

[0022] In the task feature and structural constraint encoding module, time series aggregation and importance weighted encoding are performed on the multi-agent environment state vector to obtain task feature sub-vectors. The task feature sub-vectors are obtained by aggregating the multi-agent environment state vectors at different time steps according to a preset time window and calculating the weighted representation of each state component based on a preset weight allocation rule.

[0023] In the policy network topology generation module, a candidate connection strength matrix is ​​generated based on the task feature sub-vectors. The candidate connection strength matrix is ​​obtained by calculating the connection strength of each node pair in the policy network one by one according to the task feature sub-vectors and the preset connection strength calculation rules. The candidate connection strength matrix is ​​then subjected to threshold screening and structural pruning according to the preset sparsity constraints and structural symmetry constraints to obtain the connection structure of the multi-agent policy network.

[0024] In the policy network parameter initialization and structure evaluation module, based on the connection structure of the multi-agent policy network as a constraint, the initialization parameters of the multi-agent policy network are generated using a meta-parameter mapping initialization mechanism. The generation process involves identifying connection clusters with the same topology according to the connection structure of the multi-agent policy network, assigning a set of meta-parameters to each type of connection cluster, and expanding each set of meta-parameters into the initial values ​​of each connection weight and each bias in the multi-agent policy network according to a preset mapping rule. The parameter set composed of the initial values ​​of the connection weights and the initial values ​​of the biases is combined to form the initialization parameters of the multi-agent policy network.

[0025] The initialization parameters of the multi-agent policy network are configured into the corresponding multi-agent policy networks in each agent model to form the initial agent model.

[0026] Optionally, the formation of the multi-agent policy model specifically includes:

[0027] The current state sub-vector in the multi-agent environment state vector is input into the initial agent model. The initial agent model generates the action vector at the current moment, and the state sub-vector at the next moment is obtained based on the action vector at the current moment. The current state sub-vector, the action vector at the current moment, and the state sub-vector at the next moment constitute the state transition data.

[0028] The reward value at the current moment is calculated based on the state transition data, and the reward values ​​at different time steps are arranged in chronological order to form a reward value sequence. The calculation process involves evaluating the state sub-vectors and action vectors of the state transition data, and combining them with the state sub-vectors of the next moment to obtain the state change amount. The attitude deviation change, position deviation change, and sling force change in the state change amount are weighted to obtain the reward value at the current moment.

[0029] The state transition data sequence and the reward value sequence are combined into a training sample set. Under the connection structure constraints of the multi-agent policy network, the parameter vector in the initialized agent model is updated according to the training sample set. The updated initialized agent model is then used as the multi-agent policy model.

[0030] Optionally, the generation of the cooperative control command specifically includes:

[0031] The joint action vector at the current moment is generated based on the multi-agent policy model. The joint action vector is composed of the action vectors of each unmanned helicopter participating in the coordinated hoisting at the current moment, in the order of the unmanned helicopter numbers.

[0032] The joint action vector is split according to a preset indexing rule, and each sub-vector obtained is corresponding to each unmanned helicopter being hoisted in collaboration. Each sub-vector consists of a relative position holding control quantity, an attitude synchronization control quantity, a traction force distribution control quantity, and a load stabilization control quantity.

[0033] The sub-vectors are aggregated according to the preset indexing rules to form the collaborative control command for the current moment.

[0034] Optionally, the generation of the updated cooperative control instructions specifically includes:

[0035] The current collaborative control command is sent to the flight control systems of multiple unmanned helicopters to control the multiple unmanned helicopters to perform flight control.

[0036] During the flight control process at the current time step, flight status information, position and attitude information, sling force information, and mass and position parameters of the target load of multiple unmanned helicopters are collected in real time to form an updated multi-agent environmental state vector.

[0037] The updated multi-agent environment state vector is input into the multi-agent policy model, which then generates updated cooperative control instructions.

[0038] Optionally, the formation of the collaborative hoisting process data specifically includes:

[0039] Based on the updated cooperative control instructions, the relative position, attitude, traction output, and load stability control states of the unmanned helicopter at the current time step are adjusted to obtain the adjusted unmanned helicopter state data.

[0040] The adjusted unmanned helicopter status data is combined according to the unmanned helicopter number sequence and combined with the updated collaborative control command at the current time step to form collaborative hoisting process data.

[0041] Optionally, the updates to the improved CPPN network and the multi-agent policy model specifically include:

[0042] The collaborative hoisting process data is organized in chronological order to form a collaborative hoisting process data sequence;

[0043] Based on the collaborative hoisting process data sequence, the structural evaluation value is calculated and arranged in chronological order to form a structural evaluation index sequence. The calculation process involves extracting the execution results from the collaborative hoisting process data sequence, obtaining evaluation index items based on the multi-agent strategy network connection structure, and weighting and combining the deviations of each evaluation index item to obtain the structural evaluation value.

[0044] Based on the structural evaluation index sequence, the meta-parameter vector in the improved CPPN network is adjusted. The changes in the structural evaluation values ​​in the structural evaluation index sequence are mapped to the adjustment amounts of each component of the meta-parameter vector. The meta-parameter vector is then updated to obtain the updated meta-parameter vector.

[0045] Using the updated meta-parameter vector, the improved CPPN network is invoked to update the connection structure and initialization parameters of the multi-agent policy network;

[0046] Under the constraints of the updated connection structure and initialization parameters of the multi-agent policy network, the collaborative hoisting process data sequence is used as training data to update the parameter vector in the multi-agent policy model. The current parameter vector of the multi-agent policy model is combined with the state information and collaborative control commands corresponding to the collaborative hoisting process data sequence, and each component of the parameter vector is iteratively optimized to obtain the updated multi-agent policy model.

[0047] The beneficial effects of this invention are:

[0048] This invention achieves a systemic improvement unattainable by existing technologies by constructing a multi-agent environmental state vector, establishing an agent model, introducing an improved CPPN network to generate the connection structure and initialization parameters of the multi-agent policy network, and then performing multi-agent learning and training and real-time collaborative control. First, this invention breaks through the traditional collaborative hoisting method that relies on manual rules or fixed control structures, enabling multiple unmanned helicopters to autonomously generate collaborative behaviors based on environmental changes in complex and dynamic heavy-load hoisting scenarios, thus significantly improving the autonomy and real-time adaptability of collaborative control. Second, this invention achieves adaptive generation between the policy network structure and task characteristics through an improved CPPN network, enabling the policy model to deeply match the task requirements at the structural level, enhancing the learning efficiency and generalization ability of the multi-agent policy model, and solving the problem that traditional fixed-structure policy models are difficult to adapt to task differences. Furthermore, this invention constructs collaborative hoisting process data during the execution of collaborative hoisting tasks and uses this data to iteratively update the improved CPPN network and the multi-agent policy model, giving the policy model continuous optimization capabilities and increasingly superior performance in repetitive tasks, further improving the system's stability, security, and task completion quality.

[0049] Based on the aforementioned multi-module linkage mechanism, this invention not only achieves high-precision collaborative control of multiple unmanned helicopters in terms of attitude synchronization, traction coordination, and load stability, but also significantly improves the robustness of the lifting process to external disturbances, load swaying, and individual performance differences, making heavy-load lifting tasks more stable, safe, and efficient. By replacing traditional manual adjustment and centralized control with intelligent collaborative decision-making, this invention effectively reduces the system's computational burden and the need for manual intervention, enabling multiple unmanned helicopters to achieve a higher level of collaborative performance in complex lifting tasks. This provides a more practical technical approach and performance advantages for the application of unmanned aerial vehicle swarms in heavy-load transportation, emergency rescue, and special environment operations. Attached Figure Description

[0050] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings:

[0051] Figure 1 This is an overall flowchart of a collaborative hoisting method for heavy-duty unmanned helicopters based on multi-agent learning proposed in this invention.

[0052] Figure 2 This is a schematic diagram of the module structure of the improved CPPN network for a heavy-load unmanned helicopter cooperative hoisting method based on multi-agent learning proposed in this invention.

[0053] Figure 3 This is a schematic diagram illustrating the construction of a multi-agent strategy model for a heavy-load unmanned helicopter collaborative hoisting method based on multi-agent learning proposed in this invention. Detailed Implementation

[0054] The present invention will now be described in further detail with reference to the accompanying drawings. These drawings are simplified schematic diagrams, illustrating only the basic structure of the invention, and therefore only show the components relevant to the invention.

[0055] refer to Figures 1-3 A collaborative lifting method for heavy-duty unmanned helicopters based on multi-agent learning includes the following steps:

[0056] The flight status information, position and attitude information, sling force information, and target load mass and position parameters of multiple unmanned helicopters are collected to construct a multi-agent environmental state vector.

[0057] Based on the multi-agent environmental state vector, an agent model is established for each unmanned helicopter;

[0058] The multi-agent environment state vector is input into the improved CPPN network to obtain the connection structure and initialization parameters of the multi-agent policy network. The initialization parameters are then allocated to each agent model to form the initial agent model.

[0059] Based on the connection structure of the multi-agent policy network, the initial agent model is trained by multi-agent learning to form a multi-agent policy model.

[0060] Based on a multi-agent policy model, collaborative control instructions are generated through preset indexing rules.

[0061] The system executes flight control on multiple unmanned helicopters according to the cooperative control commands. During the cooperative lifting process, it continuously collects and updates the multi-agent environmental state vectors and inputs them into the multi-agent policy model to generate updated cooperative control commands.

[0062] Based on the updated collaborative control commands, the attitude, traction output, and relative position of multiple unmanned helicopters are adjusted in real time to form collaborative lifting process data;

[0063] The improved CPPN network and multi-agent policy model are updated based on the data from the collaborative hoisting process.

[0064] In this embodiment, the multi-agent environment state vector is obtained by mapping flight state information, position and attitude information, sling force information, and the mass and position parameters of the target load into corresponding state sub-vectors and concatenating them in a preset order.

[0065] In this embodiment, the establishment of the intelligent agent model specifically includes:

[0066] The state sub-vectors in the multi-agent environment state vector are defined as the state space of the unmanned helicopter.

[0067] The motion of the unmanned helicopter is defined as a motion vector, which consists of attitude adjustment commands, position correction commands, and traction output commands. The attitude adjustment commands refer to adjusting the attitude of the unmanned helicopter by controlling its pitch, roll, and yaw angles. The position correction commands refer to adjusting the forward, backward, left, right, and up / down positions of the unmanned helicopter in three-dimensional space based on the deviation between the current position and the target position. The traction output commands refer to calculating the applied traction force based on the weight of the load, the stress on the sling, and the state of the unmanned helicopter. The calculation process involves calculating the magnitude of the traction force required for each unmanned helicopter using a preset mechanical model. The magnitude of the traction force includes the magnitude of gravity, the magnitude of inertial force, and the magnitude of sling tension. The traction force applied by the unmanned helicopter is controlled by adjusting the hovering height and attitude angle.

[0068] Based on the requirements of the collaborative hoisting mission and the performance parameters of multiple unmanned helicopters, value ranges are set for attitude adjustment commands, position correction commands, and traction output commands, and the set of all motion vectors is defined as the motion space of the unmanned helicopter.

[0069] The state space and action space of each unmanned helicopter are mapped and associated to form a corresponding intelligent agent model.

[0070] In this embodiment, the formation of the initialization agent model specifically includes:

[0071] The multi-agent environment state vector is input into the improved CPPN network, which includes a task feature and structural constraint encoding module, a policy network topology generation module, and a policy network parameter initialization and structure evaluation module. The task feature and structural constraint encoding module performs time series aggregation and importance weighted encoding on the multi-agent environment state vector to obtain task feature sub-vectors. The policy network topology generation module performs threshold screening and structural pruning on the candidate connection strength matrix according to preset sparsity constraints and structural symmetry constraints to obtain the connection structure of the multi-agent policy network. The policy network parameter initialization and structure evaluation module generates the initialization parameters of the multi-agent policy network using a meta-parameter mapping initialization mechanism.

[0072] In the task feature and structural constraint encoding module, time series aggregation and importance weighted encoding are performed on the multi-agent environment state vector to obtain task feature sub-vectors. The task feature sub-vectors are obtained by aggregating the multi-agent environment state vectors at different time steps according to a preset time window and calculating the weighted representation of each state component based on a preset weight allocation rule.

[0073] In the policy network topology generation module, a candidate connection strength matrix is ​​generated based on the task feature sub-vectors. The candidate connection strength matrix is ​​obtained by calculating the connection strength of each node pair in the policy network one by one according to the task feature sub-vectors and the preset connection strength calculation rules. The candidate connection strength matrix is ​​then subjected to threshold screening and structural pruning according to the preset sparsity constraints and structural symmetry constraints to obtain the connection structure of the multi-agent policy network.

[0074] In the policy network parameter initialization and structure evaluation module, based on the connection structure of the multi-agent policy network as a constraint, the initialization parameters of the multi-agent policy network are generated using a meta-parameter mapping initialization mechanism. The generation process involves identifying connection clusters with the same topology according to the connection structure of the multi-agent policy network, assigning a set of meta-parameters to each type of connection cluster, and expanding each set of meta-parameters into the initial values ​​of each connection weight and each bias in the multi-agent policy network according to a preset mapping rule. The parameter set composed of the initial values ​​of the connection weights and the initial values ​​of the biases is combined to form the initialization parameters of the multi-agent policy network.

[0075] The initialization parameters of the multi-agent policy network are configured into the corresponding multi-agent policy networks in each agent model to form the initial agent model.

[0076] In this embodiment, the formation of the multi-agent policy model specifically includes:

[0077] The current state sub-vector in the multi-agent environment state vector is input into the initial agent model. The initial agent model generates the action vector at the current moment, and the state sub-vector at the next moment is obtained based on the action vector at the current moment. The current state sub-vector, the action vector at the current moment, and the state sub-vector at the next moment constitute the state transition data.

[0078] The reward value at the current moment is calculated based on the state transition data, and the reward values ​​at different time steps are arranged in chronological order to form a reward value sequence. The calculation process involves evaluating the state sub-vectors and action vectors of the state transition data, and combining them with the state sub-vectors of the next moment to obtain the state change amount. The attitude deviation change, position deviation change, and sling force change in the state change amount are weighted to obtain the reward value at the current moment.

[0079] The state transition data sequence and the reward value sequence are combined into a training sample set. Under the connection structure constraints of the multi-agent policy network, the parameter vector in the initialized agent model is updated according to the training sample set. The updated initialized agent model is then used as the multi-agent policy model.

[0080] In this embodiment, the generation of the cooperative control command specifically includes:

[0081] The joint action vector at the current moment is generated based on the multi-agent policy model. The joint action vector is composed of the action vectors of each unmanned helicopter participating in the coordinated hoisting at the current moment, in the order of the unmanned helicopter numbers.

[0082] The joint action vector is split according to a preset indexing rule, and each sub-vector obtained is corresponding to each unmanned helicopter being hoisted in collaboration. Each sub-vector consists of a relative position holding control quantity, an attitude synchronization control quantity, a traction force distribution control quantity, and a load stabilization control quantity.

[0083] The sub-vectors are aggregated according to the preset indexing rules to form the collaborative control command for the current moment.

[0084] In this embodiment, the generation of the updated cooperative control command specifically includes:

[0085] The current collaborative control command is sent to the flight control systems of multiple unmanned helicopters to control the multiple unmanned helicopters to perform flight control.

[0086] During the flight control process at the current time step, flight status information, position and attitude information, sling force information, and mass and position parameters of the target load of multiple unmanned helicopters are collected in real time to form an updated multi-agent environmental state vector.

[0087] The updated multi-agent environment state vector is input into the multi-agent policy model, which then generates updated cooperative control instructions.

[0088] In this embodiment, the formation of the collaborative hoisting process data specifically includes:

[0089] Based on the updated cooperative control instructions, the relative position, attitude, traction output, and load stability control states of the unmanned helicopter at the current time step are adjusted to obtain the adjusted unmanned helicopter state data.

[0090] The adjusted unmanned helicopter status data is combined according to the unmanned helicopter number sequence and combined with the updated collaborative control command at the current time step to form collaborative hoisting process data.

[0091] In this embodiment, the updating of the improved CPPN network and the multi-agent policy model specifically includes:

[0092] The collaborative hoisting process data is organized in chronological order to form a collaborative hoisting process data sequence;

[0093] Based on the collaborative hoisting process data sequence, the structural evaluation value is calculated and arranged in chronological order to form a structural evaluation index sequence. The calculation process involves extracting the execution results from the collaborative hoisting process data sequence, obtaining evaluation index items based on the multi-agent strategy network connection structure, and weighting and combining the deviations of each evaluation index item to obtain the structural evaluation value.

[0094] Based on the structural evaluation index sequence, the meta-parameter vector in the improved CPPN network is adjusted. The changes in the structural evaluation values ​​in the structural evaluation index sequence are mapped to the adjustment amounts of each component of the meta-parameter vector. The meta-parameter vector is then updated to obtain the updated meta-parameter vector.

[0095] Using the updated meta-parameter vector, the improved CPPN network is invoked to update the connection structure and initialization parameters of the multi-agent policy network;

[0096] Under the constraints of the updated connection structure and initialization parameters of the multi-agent policy network, the collaborative hoisting process data sequence is used as training data to update the parameter vector in the multi-agent policy model. The current parameter vector of the multi-agent policy model is combined with the state information and collaborative control commands corresponding to the collaborative hoisting process data sequence, and each component of the parameter vector is iteratively optimized to obtain the updated multi-agent policy model.

[0097] Example 1:

[0098] In an emergency supplies transport mission in a high-altitude mountainous area, the rescue team needed to hoist a set of heavy emergency power supply equipment from the bottom of the valley to a platform on the mountainside. Due to the large weight of the equipment, a single unmanned helicopter could not complete the hoisting. Furthermore, the valley experienced significant wind disturbances, and traditional collaborative hoisting systems relying on manual rules or centralized control often failed to complete the task smoothly in such scenarios due to unstable control or excessive load fluctuations. Therefore, this embodiment employs the heavy-load unmanned helicopter collaborative hoisting method based on multi-agent learning proposed in this invention, which coordinates the control of four heavy-load unmanned helicopters to achieve a stable, safe, and efficient hoisting mission.

[0099] In this embodiment, four unmanned helicopters first collect their own flight status information, position and attitude information, sling force information, and the mass and position parameters of the target load in real time, and combine this information to form a multi-agent environmental state vector. The flight control system establishes an agent model for each of the four unmanned helicopters based on this environmental state vector. Subsequently, the environmental state vector is input into an improved CPPN network, which generates a multi-agent policy network connection structure and initialization parameters adapted to the current lifting task, and distributes them to each agent model to form an initialized agent model. Through multi-agent learning and training, these four unmanned helicopters can gradually master at the policy level how to maintain overall attitude consistency, how to evenly distribute traction force, how to suppress load sway, and how to automatically correct attitude under wind interference. As training progresses, the multi-agent policy model gradually converges, the action vector output of each agent tends to stabilize, and finally a cooperative strategy for actual lifting tasks is formed.

[0100] In the actual execution scenario, four unmanned helicopters initiate a collaborative lifting mode after the load is connected. The system inputs the current environmental state vector into the multi-agent policy model, which then outputs a set of collaborative control commands. Each unmanned helicopter can obtain its own attitude adjustment control, position correction control, traction output control, and load stabilization control from the collaborative control command set, and adjust its attitude, position, and traction accordingly. During flight, the system continuously monitors the state changes of each unmanned helicopter and updates the environmental state vector in real time, inputting it again into the policy model to generate the collaborative control commands for the next moment, thereby ensuring that the overall lifting structure remains under control at all times.

[0101] Without the method of this invention, the four unmanned helicopters exhibited significant instability in similar missions. Due to wind speed fluctuations and complex flight paths, it was difficult for the four unmanned helicopters to maintain synchronization, resulting in individual helicopters having excessively large or small traction forces, causing increased load sway amplitude. In the past three simulated missions, the peak peak lateral sway angle of the load reached an average of 12.4°, and the traction force difference between the four unmanned helicopters reached a maximum of 28%. This difference is sufficient to cause system instability or even lifting failure in complex dynamic scenarios.

[0102] After applying the method of this invention, the four unmanned helicopters exhibited significantly higher consistency and stability in dynamic environments through the cooperative control strategy formed by the multi-agent policy model. In eight consecutive sets of task data, the peak value of the load lateral swing angle remained consistently below 4°, and the difference in traction output among the four unmanned helicopters did not exceed 8%. Furthermore, in two sets of tasks with significant wind speed variations, the system was still able to stably maintain hovering coordination and load directional control, without exhibiting the load swing torque imbalance problem commonly seen in traditional methods. The comparison demonstrates that the multi-agent learning method can effectively achieve complex dynamic cooperative control and maintain strong system robustness even in windy environments.

[0103] To further verify the beneficial effects of the present invention, this embodiment compares the performance of key indicators of the traditional centralized cooperative control method and the method of the present invention in multiple hoisting tasks, including peak load swing angle, unmanned helicopter traction force balance, task time, hoisting path deviation, and system coordination under wind disturbances. The data are shown in the table below.

[0104] Table 1 Comparison of Collaborative Lifting Performance of Heavy-Duty Unmanned Helicopters

[0105]

[0106] As shown in Table 1, the method of this invention significantly outperforms traditional cooperative control methods in several key performance indicators. First, by constructing a multi-agent strategy model, this invention enables the four unmanned helicopters to learn and form effective cooperative behaviors, thereby significantly reducing the peak value of the load's lateral sway angle, with an average reduction of over 70%, indicating a significant improvement in load stability. Second, in the multi-agent strategy model, the traction output control quantity can be generated in real time based on the multi-agent environmental state vector, reducing the traction difference among the four unmanned helicopters from 28% in the traditional method to 7.6%, demonstrating more balanced force control and effectively avoiding the risk of load sway or breakage.

[0107] In terms of path control, the average deviation of the hoisting path is significantly reduced because the strategy model can continuously update the action vectors according to environmental changes, indicating a substantial improvement in cooperative navigation capabilities. A reduction of approximately 34% in task time demonstrates improved overall system decision-making efficiency, while the task failure rate in windy environments decreased from 23.1% to 2.8%, further proving the strong environmental adaptability and anti-interference capabilities of the method presented in this invention. In multiple training iterations, this invention significantly accelerates the convergence speed of the joint action vectors and reduces the number of iterations by more than 50% by improving the CPPN network to generate a better connection structure and initialization parameters. This also verifies the effectiveness of optimizing the strategy model at the structural level.

[0108] In summary, the data from this embodiment fully demonstrates that the multi-agent learning method, agent model construction mechanism, and improved CPPN network structure generation mechanism of the present invention can significantly improve the stability, coordination, and efficiency of collaborative lifting missions by heavy-load unmanned helicopters, and effectively solve problems such as insufficient collaborative capabilities, fixed strategies unable to adapt to complex dynamic environments, and poor load stability in existing technologies. The method of the present invention not only performs excellently in simulated environments but also has high practical value and engineering promotion potential in near-real-world tasks.

Claims

1. A method for heavy-load unmanned helicopter cooperative hoisting based on multi-agent learning, characterized in that, The method comprises the following steps: Collecting flight state information, position and attitude information, sling force information of multiple unmanned helicopters, and mass parameters and position parameters of target loads to construct a multi-agent environment state vector; Establishing an agent model for each unmanned helicopter according to the multi-agent environment state vector; Inputting the multi-agent environment state vector into an improved CPPN network to obtain the connection structure and initialization parameters of the multi-agent policy network, and distributing the initialization parameters to each agent model to form an initialized agent model; Based on the connection structure of the multi-agent policy network, performing multi-agent learning and training on the initialized agent model to form a multi-agent policy model; Generating a cooperative control instruction through a preset index rule based on the multi-agent policy model; Performing flight control on multiple unmanned helicopters according to the cooperative control instruction, continuously collecting updated multi-agent environment state vectors during the cooperative hoisting process, and inputting the updated multi-agent environment state vectors into the multi-agent policy model to generate an updated cooperative control instruction; Real-time adjusting the attitude, traction output and relative position relationship of multiple unmanned helicopters according to the updated cooperative control instruction to form cooperative hoisting process data; Updating the improved CPPN network and the multi-agent policy model according to the cooperative hoisting process data; The formation of the initialized agent model specifically comprises: Inputting the multi-agent environment state vector into the improved CPPN network, wherein the improved CPPN network comprises a task feature and structure constraint encoding module, a policy network topology generation module, and a policy network parameter initialization and structure evaluation module, the task feature and structure constraint encoding module refers to time series aggregation and importance weighted encoding of the multi-agent environment state vector to obtain a task feature subvector, the policy network topology generation module refers to threshold filtering and structure pruning of a candidate connection strength matrix according to preset sparsity constraints and structure symmetry constraints to obtain the connection structure of the multi-agent policy network, and the policy network parameter initialization and structure evaluation module refers to generating the initialization parameters of the multi-agent policy network by using a meta-parameter mapping initialization mechanism; In the task feature and structure constraint encoding module, the multi-agent environment state vector is time series aggregated and importance weighted encoded to obtain a task feature subvector, wherein the task feature subvector is obtained by aggregating the multi-agent environment state vectors at different time steps according to a preset time window and calculating the weighted representation of each state component based on a preset weight distribution rule; In the policy network topology generation module, a candidate connection strength matrix is generated based on the task feature subvector, wherein the candidate connection strength matrix is obtained by calculating the connection strength of each node pair of the policy network according to the task feature subvector and a preset connection strength calculation rule, and the candidate connection strength matrix is threshold filtered and structure pruned according to the preset sparsity constraints and structure symmetry constraints to obtain the connection structure of the multi-agent policy network. In the strategy network parameter initialization and structure evaluation module, based on the connection structure of the multi-agent strategy network as a constraint condition, an initialization parameter of the multi-agent strategy network is generated by using a meta-parameter mapping initialization mechanism, the generation process is that a connection cluster with the same topological structure is identified according to the connection structure of the multi-agent strategy network, a group of meta-parameters is allocated to each type of connection cluster, and each group of meta-parameters is expanded into initial values of each connection weight and initial values of each bias in the multi-agent strategy network according to a preset mapping rule, and a parameter set composed of the initial values of the connection weights and the initial values of the biases is combined to form the initialization parameter of the multi-agent strategy network; The initialization parameter of the multi-agent strategy network is configured into the corresponding multi-agent strategy network of each agent model to form an initialized agent model.

2. The multi-agent learning-based heavy-load unmanned helicopter cooperative hoisting method according to claim 1, characterized in that, The multi-agent environment state vector is obtained by mapping flight state information, position and attitude information, sling force information, and mass parameters and position parameters of the target load into corresponding state sub-vectors and splicing them in a preset order.

3. The multi-agent learning-based heavy-load unmanned helicopter cooperative hoisting method according to claim 1, characterized in that, The establishment of the agent model specifically includes: The state sub-vector in the multi-agent environment state vector is defined as the state space of the unmanned helicopter. The action of the unmanned helicopter is defined as an action vector, which is composed of an attitude adjustment instruction, a position correction instruction, and a traction force output instruction, the attitude adjustment instruction is to adjust the attitude of the unmanned helicopter by controlling the pitch, roll, and yaw angles of the unmanned helicopter, the position correction instruction is to adjust the front, rear, left, right, and up-down positions of the unmanned helicopter in three-dimensional space according to the deviation between the current position and the target position, and the traction force output instruction is to calculate the traction force to be applied according to the weight of the load, the force of the sling, and the state of the unmanned aerial vehicle, the calculation process is to calculate the size of the traction force to be applied by each unmanned helicopter through a preset mechanical model, the size of the traction force including the size of the gravity, the size of the inertia force, and the size of the sling tension, and the size of the traction force is controlled by adjusting the hovering height and the attitude angle of the unmanned helicopter. According to the cooperative hoisting task requirements and the performance parameters of the multiple unmanned helicopters, the value ranges of the attitude adjustment instruction, the position correction instruction, and the traction force output instruction are set, and a set composed of all the action vectors is defined as the action space of the unmanned helicopter. The state space and the action space of each unmanned helicopter are mapped and associated to form a corresponding agent model.

4. The multi-agent learning-based heavy-load unmanned helicopter cooperative hoisting method according to claim 1, characterized in that, The formation of the multi-agent strategy model specifically includes: The state sub-vector at the current time in the multi-agent environment state vector is input into the initialized agent model, the action vector at the current time is generated by the initialized agent model, and the state sub-vector at the next time is obtained based on the action vector at the current time, the state transition data is composed of the state sub-vector at the current time, the action vector at the current time, and the state sub-vector at the next time. The reward value at the current moment is calculated according to the state transition data, and the reward values at different time steps are arranged in time sequence to form a reward value sequence, the calculation process is to evaluate the state sub-vector and the action vector of the state transition data, and the state change amount is obtained by combining the next moment state sub-vector, the attitude deviation change, the position deviation change and the sling force change in the state change amount are weighted to obtain the reward value at the current moment; The state transition data sequence and the reward value sequence are combined into a training sample set, and under the constraint of the connection structure of the multi-agent policy network, the parameter vector in the initialized agent model is updated according to the training sample set, and the updated initialized agent model is used as the multi-agent policy model.

5. The multi-agent learning-based heavy-load unmanned helicopter cooperative hoisting method according to claim 1, characterized in that, The generation of the cooperative control instruction specifically includes: A joint action vector at the current moment is generated based on the multi-agent policy model, and the joint action vector is combined by the action vectors of each unmanned helicopter participating in cooperative hoisting at the current moment in the order of unmanned helicopter numbers; Each sub-vector obtained by splitting the joint action vector according to a preset index rule corresponds to each unmanned helicopter for cooperative hoisting, and each sub-vector is composed of relative position maintaining control, attitude synchronization control, traction force distribution control and load stability control; Each sub-vector is summarized according to the preset index rule to form a cooperative control instruction at the current moment.

6. The multi-agent learning-based heavy-load unmanned helicopter cooperative hoisting method according to claim 1, characterized in that, The generation of the updated cooperative control instruction specifically includes: The cooperative control instruction at the current moment is respectively sent to the flight control system of the plurality of unmanned helicopters to control the plurality of unmanned helicopters to execute flight control; During the flight control process at the current time step, the flight state information, the position and attitude information, the sling force information, the mass parameter and the position parameter of the target load of the plurality of unmanned helicopters are collected in real time to form an updated multi-agent environment state vector; The updated multi-agent environment state vector is input into the multi-agent policy model to generate an updated cooperative control instruction.

7. The multi-agent learning based heavy-load unmanned helicopter cooperative hoisting method according to claim 1, characterized in that, The formation of the cooperative hoisting process data specifically includes: According to the updated cooperative control instruction, the relative position state, the attitude state, the traction force output state and the load stability control state of the unmanned helicopter at the current time step are adjusted to obtain adjusted unmanned helicopter state data; The adjusted unmanned helicopter state data is combined in the order of unmanned helicopter numbers, and the updated cooperative control instruction at the current time step is combined to form the cooperative hoisting process data.

8. The multi-agent learning-based heavy-load unmanned helicopter cooperative hoisting method according to claim 1, characterized in that, The updating of the improved CPPN network and the multi-agent policy model specifically includes: The cooperative hoisting process data is arranged in time sequence to form a cooperative hoisting process data sequence; According to the cooperative hoisting process data sequence, a structure evaluation value is calculated, and a structure evaluation index sequence is arranged in time sequence, the calculation process is to extract the execution result from the cooperative hoisting process data sequence, to obtain the evaluation index item according to the multi-agent policy network connection structure, to weight and combine the deviation amount of each evaluation index item to obtain the structure evaluation value; According to the structural evaluation index sequence, the meta-parameter vector in the improved CPPN network is adjusted, the change result of the structural evaluation value in the structural evaluation index sequence is mapped to the adjustment amount of each component of the meta-parameter vector, the meta-parameter vector is updated, and an updated meta-parameter vector is obtained; By using the updated meta-parameter vector, the improved CPPN network is called to update the connection structure and the initialization parameter of the multi-agent strategy network; Under the constraint of the updated connection structure and the initialization parameter of the multi-agent strategy network, the collaborative hoisting process data sequence is used as training data to update the parameter vector in the multi-agent strategy model, the current parameter vector of the multi-agent strategy model is combined with the state information corresponding to the collaborative hoisting process data sequence and the collaborative control instruction, each component of the parameter vector is iteratively optimized, and an updated multi-agent strategy model is obtained.

Images