Air-drop system intelligent control method and device based on GRPO reinforcement learning

Abstract

The invention provides an air-drop system intelligent control method and device based on GRPO reinforcement learning, and belongs to the technical field of air-drop system intelligent control. In the method, a reinforcement learning controller optimized by a group relative strategy is introduced, the autonomous decision-making ability, the dynamic environment adaptability and the high-precision landing performance in the air-drop process are remarkably improved, and the reliability of the air-drop system is improved. Manual intervention is not needed, a control strategy can be dynamically adjusted according to environment changes, control precision is high, and safety is good.

Description

Technical Field

[0001] This invention relates to the technical field of intelligent control of airdrop systems, and in particular to an intelligent control method and device for airdrop systems based on GRPO reinforcement learning. Background Technology

[0002] Traditional airdrop systems typically rely on preset trajectories and fixed parameters (such as parachute opening time and attitude adjustment commands), making it difficult to cope with the needs of complex weather environments and dynamic target points.

[0003] The shortcomings of traditional technologies are mainly reflected in the following aspects: (1) Lack of autonomous adaptability. Traditional airdrop systems cannot perceive environmental changes (such as wind speed, wind direction, and terrain) in real time and dynamically adjust control strategies, resulting in a decrease in airdrop accuracy; (2) Reliance on manual intervention. They need to rely on ground guidance or manual operation and cannot complete tasks independently under the "three no's" conditions (no ground guidance, no ground control, and no meteorological data); (3) Limited control accuracy. Traditional control systems (such as PID control) have poor adaptability to nonlinear dynamic systems and are difficult to achieve optimal control under complex working conditions; (4) Insufficient safety. In scenarios such as landing against the wind and obstacle avoidance, they lack intelligent decision-making capabilities, which can easily lead to cargo damage or mission failure.

[0004] In summary, traditional airdrop systems suffer from technical problems such as poor autonomous adaptability, the need for manual intervention, limited control precision, and insufficient safety. Summary of the Invention

[0005] In view of this, the purpose of this invention is to provide an intelligent control method and device for airdrop systems based on GRPO reinforcement learning, so as to alleviate the technical problems of poor autonomous adaptability, need for manual intervention, limited control accuracy and insufficient safety in the traditional intelligent control process of airdrop systems.

[0006] In a first aspect, embodiments of the present invention provide an intelligent control method for an airdrop system based on GRPO reinforcement learning. The airdrop system includes: a perception module, a reinforcement learning controller for swarm relative policy optimization, a PID controller, and an execution module. The method includes: The system acquires the real-time status and environmental information of the airdrop system collected by the sensing module, and obtains the future environmental changes predicted by the dynamic environment model based on the status and environmental information. The first airdrop control command generated by the reinforcement learning controller with the group relative policy optimization based on the state information, the environment information, and the future environment changes is obtained, and the second airdrop control command generated by the PID controller based on the state information and the environment information is obtained. The reinforcement learning controller with the group relative policy optimization is obtained by training the original reinforcement learning controller with the group relative policy optimization in a simulation environment by introducing adversarial examples to obtain an intermediate reinforcement learning controller with the group relative policy optimization, and then deploying the intermediate reinforcement learning controller with the group relative policy optimization to a real hardware system and fine-tuning it online using actual flight data. The adversarial examples are used to improve the generalization ability of the reinforcement learning controller with the group relative policy optimization. The first airdrop control command is used to modify the second airdrop control command to obtain the target airdrop control command; The execution module is controlled to perform the airdrop action according to the target airdrop control command, and then returns to the step of obtaining the status information and environmental information of the airdrop system collected in real time by the sensing module, until the airdrop mission is completed.

[0007] Furthermore, the reinforcement learning controller for group relative policy optimization includes: multiple agents; each agent represents a control policy; The training process for the multiple agents includes: Initialize multiple initial agents; Run each of the initial agents and record its behavioral trajectory and calculate the corresponding reward value; Calculate the relative reward for each initial agent based on the reward value corresponding to each initial agent; The parameters of the initial agents are adjusted according to the relative rewards of each initial agent, thereby obtaining the agent.

[0008] Furthermore, the corresponding reward value is calculated, including: Calculation formula based on reward value Calculate the reward value, where, This represents the reward value. This represents the Euclidean distance between the current position and the target position at time t. Indicates the attenuation distance. Indicates the instantaneous velocity upon landing. Indicates the safe landing speed. Indicates cumulative energy consumption. Indicates maximum energy consumption. , , This represents the weighting coefficient.

[0009] Furthermore, the relative reward for each initial agent is calculated based on the reward value corresponding to each initial agent, including: According to the relative reward calculation formula Calculate the relative reward for each of the initial agents, where, This represents the relative reward for the i-th initial agent. This represents the reward value corresponding to the i-th initial agent. This represents the average reward value for the group. This represents the standard deviation of the group reward value.

[0010] Furthermore, the parameters of the initial agents are adjusted based on the relative rewards of each initial agent, including: Adjust the formula according to the parameters Adjust the parameters of the initial agent, wherein, This indicates the adjusted parameters. This indicates the parameters before adjustment. Indicates the master learning rate. Represents the entropy regularization coefficient. This refers to the initial intelligent agent. The parameters representing the initial agent, The information entropy function is represented by... , This represents the action vector of the i-th initial agent. Let represent the state vector of the i-th initial agent. This represents the relative reward for the i-th initial agent. This indicates the number of initial intelligent agents.

[0011] Furthermore, the first airdrop control command generated by the reinforcement learning controller for relative policy optimization of the population based on the state information, the environment information, and the future environment changes includes: When the height is greater than the first height, the first airdrop control command is used to quickly stabilize the cargo's attitude; When the height is less than the first height but greater than the second height, the first airdrop control command is used to counteract the effects of crosswinds and turbulence, ensuring that the cargo approaches the target area along a predetermined trajectory. When the height is less than the second height, the first airdrop control command is used to slow the descent speed and avoid potential obstacles.

[0012] Furthermore, the second airdrop control command is modified using the first airdrop control command to obtain the target airdrop control command, including: According to the revised formula The second airdrop control command is modified to obtain the target airdrop control command, wherein, This indicates the target airdrop control command. This represents the proportional gain matrix in the second airdrop control command. This represents the differential gain matrix in the second airdrop control command. Indicates state error, This indicates the first airdrop control command.

[0013] Secondly, embodiments of the present invention also provide an intelligent control device for an airdrop system based on GRPO reinforcement learning. The airdrop system includes: a perception module, a reinforcement learning controller for swarm relative policy optimization, a PID controller, and an execution module. The device includes: The first acquisition unit is used to acquire the status information and environmental information of the airdrop system collected in real time by the sensing module, and to acquire the future environmental changes predicted by the dynamic environment model based on the status information and the environmental information. The second acquisition unit is used to acquire the first airdrop control command generated by the reinforcement learning controller for group relative policy optimization based on the state information, the environment information, and the future environment changes, and to acquire the second airdrop control command generated by the PID controller based on the state information and the environment information. The reinforcement learning controller for group relative policy optimization is obtained by training the original reinforcement learning controller for group relative policy optimization in a simulation environment using adversarial examples to obtain an intermediate reinforcement learning controller for group relative policy optimization. This intermediate reinforcement learning controller is then deployed to a real hardware system and fine-tuned online using actual flight data. The adversarial examples are used to improve the generalization ability of the reinforcement learning controller for group relative policy optimization. The correction unit is used to correct the second airdrop control command using the first airdrop control command to obtain the target airdrop control command; The control unit is used to control the execution module to perform airdrop actions according to the target airdrop control command, and return to the step of obtaining the status information and environmental information of the airdrop system collected in real time by the sensing module, until the airdrop mission is completed.

[0014] Thirdly, embodiments of the present invention also provide an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the method described in any of the first aspects above.

[0015] Fourthly, embodiments of the present invention also provide a computer-readable storage medium storing machine-executable instructions, which, when invoked and executed by a processor, cause the processor to perform the method described in any of the first aspects above.

[0016] In this embodiment of the invention, an intelligent control method for an airdrop system based on GRPO reinforcement learning is provided. The airdrop system includes: a perception module, a swarm relative policy optimization reinforcement learning controller, a PID controller, and an execution module. The method includes: acquiring the state information and environmental information of the airdrop system collected in real time by the perception module, and acquiring future environmental changes predicted by a dynamic environment model based on the state information and environmental information; acquiring a first airdrop control command generated by the swarm relative policy optimization reinforcement learning controller based on the state information, environmental information, and future environmental changes, and acquiring a second airdrop control command generated by the PID controller based on the state information and environmental information, wherein the swarm relative policy optimization reinforcement learning controller... The device trains an intermediate group relative policy optimization reinforcement learning controller by introducing adversarial examples in a simulation environment. This intermediate group relative policy optimization reinforcement learning controller is then deployed to a real hardware system and fine-tuned online using actual flight data. The adversarial examples are used to improve the generalization ability of the group relative policy optimization reinforcement learning controller. A first airdrop control command is used to modify a second airdrop control command to obtain a target airdrop control command. The control execution module executes the airdrop action according to the target airdrop control command and returns to the step of acquiring the real-time state and environmental information of the airdrop system collected by the perception module, until the airdrop mission is completed. As described above, the intelligent control method for airdrop systems based on GRPO reinforcement learning of this invention introduces a group relative policy optimization reinforcement learning controller, which significantly improves the autonomous decision-making ability, dynamic environmental adaptability, and high-precision landing performance of the airdrop process. It requires no manual intervention, can dynamically adjust the control strategy according to environmental changes, and has high control accuracy and good safety. This alleviates the technical problems of poor autonomous adaptability, the need for manual intervention, limited control accuracy, and insufficient safety in traditional intelligent control processes for airdrop systems. Attached Figure Description

[0017] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0018] Figure 1A flowchart of an intelligent control method for an airdrop system based on GRPO reinforcement learning, provided as an embodiment of the present invention; Figure 2 A schematic diagram of the intelligent control process of an airdrop system based on GRPO reinforcement learning provided in an embodiment of the present invention; Figure 3 A flowchart of a multi-stage hybrid control strategy provided in an embodiment of the present invention; Figure 4 A schematic diagram of an intelligent control device for an airdrop system based on GRPO reinforcement learning, provided for an embodiment of the present invention; Figure 5 This is a schematic diagram of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0019] The technical solution of the present invention will be clearly and completely described below with reference to the embodiments. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0020] Traditional airdrop systems suffer from poor autonomous adaptability in their intelligent control processes, require manual intervention, have limited control precision, and are not safe enough.

[0021] Based on this, the intelligent control method for airdrop systems based on GRPO reinforcement learning of the present invention introduces a reinforcement learning controller for group relative policy optimization, which significantly improves the autonomous decision-making ability, dynamic environment adaptability and high-precision landing performance of the airdrop process. It does not require manual intervention, can dynamically adjust the control strategy according to environmental changes, and has high control accuracy and good safety.

[0022] To facilitate understanding of this embodiment, a detailed description of an intelligent control method for an airdrop system based on GRPO reinforcement learning, as disclosed in this embodiment of the invention, will be provided first.

[0023] Example 1: According to an embodiment of the present invention, an embodiment of an intelligent control method for an airdrop system based on GRPO reinforcement learning is provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0024] Figure 1 This is a flowchart of an intelligent control method for an airdrop system based on GRPO reinforcement learning according to an embodiment of the present invention, as shown below. Figure 1 As shown, the method includes the following steps: Step S102: Obtain the status information and environmental information of the airdrop system collected in real time by the sensing module, and obtain the future environmental changes predicted by the dynamic environment model based on the status information and environmental information; In this embodiment of the invention, the airdrop system includes: a perception module, a swarm relative strategy optimization reinforcement learning controller, a PID controller, and an execution module. The perception module integrates BeiDou navigation satellites, an inertial measurement unit (IMU), and a wind speed sensor to collect real-time status and environmental information of the airdrop system, including key parameters such as altitude, speed, attitude angle, wind speed, and wind direction. The swarm relative strategy optimization reinforcement learning controller, based on the GRPO framework, generates a first airdrop control command. The PID controller generates a second airdrop control command based on the status and environmental information. The first and second airdrop control commands are then fused to obtain a target airdrop control command. The execution module executes the target airdrop control command through hardware such as servo mechanisms and motor drivers, adjusting the parachute attitude or cargo release timing.

[0025] Specifically, the perception module collects the status and environmental information of the airdrop system in real time through high-precision sensors (such as GPS, inertial measurement unit IMU, and anemometer). At the same time, it can combine historical airdrop data to build and train a dynamic environment model. The dynamic environment model can then predict future environmental changes (such as wind field changes) based on the status and environmental information.

[0026] Step S104: Obtain the first airdrop control command generated by the reinforcement learning controller for group relative policy optimization based on state information, environmental information, and future environmental changes; and obtain the second airdrop control command generated by the PID controller based on state information and environmental information. The reinforcement learning controller for group relative policy optimization is obtained by training the original reinforcement learning controller for group relative policy optimization in a simulation environment with adversarial examples, obtaining an intermediate reinforcement learning controller for group relative policy optimization, and then deploying the intermediate reinforcement learning controller for group relative policy optimization to a real hardware system and fine-tuning it online using actual flight data. The adversarial examples are used to improve the generalization ability of the reinforcement learning controller for group relative policy optimization. Specifically, extreme conditions (i.e., adversarial examples, such as extremely high wind speeds and sudden airflow disturbances) are introduced into a simulation environment to test the coping capabilities of the reinforcement learning controller with the original population's relative policy optimization (i.e., adversarial training). By continuously adjusting the distribution of training samples, its generalization performance in complex scenarios is improved, and its robustness is enhanced. Then, the agents trained on a large number of simulations (i.e., the intermediate population relative policy optimization reinforcement learning controller) are deployed to a real hardware system (i.e., transfer learning), and fine-tuned online using a small amount of actual flight data to obtain the population relative policy optimization reinforcement learning controller. This method not only reduces the cost of field testing but also significantly improves the system's adaptability and reliability.

[0027] Step S106: The second airdrop control command is modified using the first airdrop control command to obtain the target airdrop control command; Step S108: The control execution module executes the airdrop action according to the target airdrop control command, and returns to the step of obtaining the status information and environmental information of the airdrop system collected in real time by the perception module, until the airdrop mission is completed.

[0028] Specifically, the intelligent control process of the airdrop system based on GRPO reinforcement learning is as follows: Figure 2 As shown.

[0029] In this embodiment of the invention, an intelligent control method for an airdrop system based on GRPO reinforcement learning is provided. The airdrop system includes: a perception module, a swarm relative policy optimization reinforcement learning controller, a PID controller, and an execution module. The method includes: acquiring the state information and environmental information of the airdrop system collected in real time by the perception module, and acquiring future environmental changes predicted by a dynamic environment model based on the state information and environmental information; acquiring a first airdrop control command generated by the swarm relative policy optimization reinforcement learning controller based on the state information, environmental information, and future environmental changes, and acquiring a second airdrop control command generated by the PID controller based on the state information and environmental information, wherein the swarm relative policy optimization reinforcement learning controller... The device trains an intermediate group relative policy optimization reinforcement learning controller by introducing adversarial examples in a simulation environment. This intermediate group relative policy optimization reinforcement learning controller is then deployed to a real hardware system and fine-tuned online using actual flight data. The adversarial examples are used to improve the generalization ability of the group relative policy optimization reinforcement learning controller. A first airdrop control command is used to modify a second airdrop control command to obtain a target airdrop control command. The control execution module executes the airdrop action according to the target airdrop control command and returns to the step of acquiring the real-time state and environmental information of the airdrop system collected by the perception module, until the airdrop mission is completed. As described above, the intelligent control method for airdrop systems based on GRPO reinforcement learning of this invention introduces a group relative policy optimization reinforcement learning controller, which significantly improves the autonomous decision-making ability, dynamic environmental adaptability, and high-precision landing performance of the airdrop process. It requires no manual intervention, can dynamically adjust the control strategy according to environmental changes, and has high control accuracy and good safety. This alleviates the technical problems of poor autonomous adaptability, the need for manual intervention, limited control accuracy, and insufficient safety in traditional intelligent control processes for airdrop systems.

[0030] The above provides a brief overview of the intelligent control method for airdrop systems based on GRPO reinforcement learning of the present invention. The specific details involved are described in detail below.

[0031] In an optional embodiment of the present invention, the reinforcement learning controller for group relative policy optimization includes: multiple agents; each agent represents a control policy; The training process for multiple agents includes: (1) Initialize multiple initial agents; Specifically, define the state vector of the i-th initial agent. s (i)This includes: the location of the airdrop platform (longitude, latitude, altitude); cargo speed and attitude angles (pitch angle, roll angle, yaw angle); real-time wind speed and direction; target location and surrounding environmental information (such as terrain height, obstacle distance); and historical action sequences (used for memory effect modeling). Define the action vector of the i-th initial agent. a (i) This includes: parachute control commands (such as opening angle and closing rate); and motor drive signals (such as the speed difference between the left and right motors, used for attitude adjustment). For a wing-parachute precision airdrop system, the motion vectors are the left and right control lines' manipulation amount and speed.

[0032] (2) Run each initial agent and record its behavior trajectory and calculate the corresponding reward value; Specifically, the calculation of the corresponding reward value includes: Calculation formula based on reward value Calculate the reward value, where, Indicates the reward value. This represents the Euclidean distance between the current position and the target position at time t. This represents the attenuation distance, which is a pre-set constant (unit: m). This represents the instantaneous landing velocity (unit: m / s). Indicates the safe landing speed (unit: m / s). This indicates cumulative energy consumption (unit: J). This indicates the maximum energy consumption (unit: J). , , Indicates the weighting coefficient. and .

[0033] The above reward function design (i.e., the reward value calculation formula) comprehensively considers factors such as target point deviation, landing speed, and energy consumption. To position the reward, For safety reasons, Due to energy consumption constraints, , represents the Euclidean distance (in meters) between the position at time t and the target position. (3) Calculate the relative reward of each initial agent based on the reward value corresponding to each initial agent; Specifically, based on the relative reward calculation formula Calculate the relative reward for each initial agent, where, This represents the relative reward (i.e., the standardized advantage value) of the i-th initial agent. This represents the reward value corresponding to the i-th initial agent. This represents the average reward value for the group. This represents the standard deviation of the group reward value.

[0034] In the above process, by comparing the performance of different initial agents, the relative reward (i.e., the advantage function value) of each initial agent relative to the group average performance is calculated according to the above formula.

[0035] (4) Adjust the parameters of the initial agents according to the relative rewards of each initial agent to obtain the agents.

[0036] Specifically, adjust the formula according to the parameters. Adjust the parameters of the initial agent, where, This indicates the adjusted parameters. This indicates the parameters before adjustment. Indicates the master learning rate. Represents the entropy regularization coefficient. Represents the initial intelligent agent. The parameters represent the initial agent. The information entropy function is represented by... , This represents the action vector of the i-th initial agent. Let represent the state vector of the i-th initial agent. This represents the relative reward for the i-th initial agent. This indicates the initial number of agents.

[0037] By adjusting the above parameters, the decision-making ability of the initial agent is gradually optimized.

[0038] In an optional embodiment of the present invention, the first airdrop control command generated by the reinforcement learning controller for obtaining the relative policy optimization of the population based on state information, environmental information, and future environmental changes includes: (1) When the height is greater than the first height, the first airdrop control command is used to quickly stabilize the cargo attitude; (2) When the altitude is less than the first altitude but greater than the second altitude, the first airdrop control command is used to counteract the effects of crosswinds and turbulence, ensuring that the cargo approaches the target area along the predetermined trajectory; (3) When the altitude is less than the second altitude, the first airdrop control command is used to slow down the descent speed and avoid potential obstacles.

[0039] Specifically, the control process is divided into three phases based on the characteristics of the airdrop process: the initial phase, the mid-flight phase, and the terminal landing phase. Figure 3 As shown: Initial Phase (i.e., when the altitude is greater than the first altitude): After cargo deployment, the agent (i.e., the reinforcement learning controller for group relative policy optimization) quickly assesses the current environmental conditions and plans an initial descent path. The main objective of this phase is to rapidly stabilize the cargo's attitude and enter mid-course flight; Mid-course flight (i.e., when the altitude is less than the first altitude but greater than the second altitude): A reinforcement learning controller optimized by group relative strategy adjusts the parachute opening angle and attitude in real time to counteract the effects of crosswinds and turbulence, ensuring that the cargo gradually approaches the target area along the predetermined trajectory. This stage requires a focus on addressing nonlinear dynamic characteristics to improve control accuracy. Terminal landing (when the altitude is less than the second altitude): As the cargo approaches the ground, the agent (i.e., the reinforcement learning controller for group relative policy optimization) switches to a headwind landing mode, optimizing the parachute deployment to slow the descent. Simultaneously, it uses terrain information to avoid potential obstacles, ensuring a safe landing for the cargo.

[0040] In an optional embodiment of the present invention, a first airdrop control command is used to modify a second airdrop control command to obtain a target airdrop control command, including: According to the revised formula The second airdrop control command is modified to obtain the target airdrop control command, wherein, Indicates the target airdrop control command. This represents the proportional gain matrix in the second airdrop control command. This represents the differential gain matrix in the second airdrop control command. Indicates state error, This indicates the first airdrop control command.

[0041] Specifically, a hybrid control architecture of "traditional PID control (i.e., PID controller) + intelligent correction compensation (i.e., correction compensation of the GRPO reinforcement learning controller)" is adopted. Traditional PID control provides basic stability and response speed, while GRPO reinforcement learning (i.e., GRPO reinforcement learning controller) is responsible for handling complex nonlinear problems and environmental disturbances, ultimately achieving high-precision and high-robust control of the airdrop system in complex dynamic environments.

[0042] The method of this invention is applicable to precision airdrop missions of precision airdrop systems, drones, and other equipment under complex weather conditions. This method has the following inventive features: Group Relative Policy Optimization (GRPO) framework: (1) Abandoning the traditional Critic model: No independent value network is required; the advantage function is calculated directly through the relative reward output of the group, thus reducing the computational cost.

[0043] (2) Dynamic environment modeling: Combine sensor data (such as wind speed, altitude, attitude angle) and historical airdrop data to construct a dynamic environment model, and use the GRPO algorithm to compensate for environmental disturbances (such as crosswinds and turbulence) in real time.

[0044] (3) End-to-end autonomous decision-making: By training the agent through reinforcement learning, it can autonomously complete tasks such as trajectory planning, parachute control, and landing strategy selection during the airdrop process without human intervention.

[0045] Multi-stage airdrop control strategy: (1) Initial stage: Rapidly descend to the predetermined altitude, while sensing the environment and planning a preliminary path.

[0046] (2) Mid-course flight: The parachute attitude is adjusted in real time by the GRPO controller (i.e., the reinforcement learning controller for group relative policy optimization) to counteract wind speed disturbances and gradually approach the target point.

[0047] (3) Terminal landing: Implement a headwind landing strategy to reduce landing speed and ensure cargo safety.

[0048] Robustness enhancement mechanisms: Adversarial training: Introduce adversarial examples (such as extreme wind speeds and sudden obstacles) to improve the generalization ability of the agent.

[0049] Transfer learning: Transferring simulation training results to real hardware systems to improve actual performance through minor online fine-tuning.

[0050] Example 2: This invention also provides an intelligent control device for an airdrop system based on GRPO reinforcement learning. This intelligent control device for an airdrop system based on GRPO reinforcement learning is mainly used to execute the intelligent control method for an airdrop system based on GRPO reinforcement learning provided in Embodiment 1 of this invention. The following is a detailed description of the intelligent control device for an airdrop system based on GRPO reinforcement learning provided in this invention.

[0051] Figure 4 This is a schematic diagram of an intelligent control device for an airdrop system based on GRPO reinforcement learning according to an embodiment of the present invention, as shown below. Figure 4 As shown, the device mainly includes: a first acquisition unit 10, a second acquisition unit 20, a correction unit 30, and a control unit 40, wherein: The first acquisition unit is used to acquire the status information and environmental information of the airdrop system collected in real time by the perception module, and to acquire the future environmental changes predicted by the dynamic environment model based on the status information and environmental information. The second acquisition unit is used to acquire the first airdrop control command generated by the reinforcement learning controller for group relative policy optimization based on state information, environmental information and future environmental changes, and to acquire the second airdrop control command generated by the PID controller based on state information and environmental information. The reinforcement learning controller for group relative policy optimization is obtained by training the original reinforcement learning controller for group relative policy optimization in a simulation environment with adversarial examples to obtain an intermediate reinforcement learning controller for group relative policy optimization. The intermediate reinforcement learning controller for group relative policy optimization is then deployed to a real hardware system and fine-tuned online using actual flight data. The adversarial examples are used to improve the generalization ability of the reinforcement learning controller for group relative policy optimization. The correction unit is used to correct the second airdrop control command using the first airdrop control command to obtain the target airdrop control command; The control unit is used to control the execution module to perform airdrop actions according to the target airdrop control command, and return to the execution steps of acquiring the status information and environmental information of the airdrop system collected in real time by the sensing module, until the airdrop mission is completed.

[0052] In this embodiment of the invention, an intelligent control device for an airdrop system based on GRPO reinforcement learning is provided. The airdrop system includes: a perception module, a swarm relative policy optimization reinforcement learning controller, a PID controller, and an execution module. The device includes: acquiring the state information and environmental information of the airdrop system collected in real time by the perception module, and acquiring future environmental changes predicted by a dynamic environment model based on the state information and environmental information; acquiring a first airdrop control command generated by the swarm relative policy optimization reinforcement learning controller based on the state information, environmental information, and future environmental changes, and acquiring a second airdrop control command generated by the PID controller based on the state information and environmental information, wherein the swarm relative policy optimization reinforcement learning controller... The device trains an intermediate group relative policy optimization reinforcement learning controller by introducing adversarial examples in a simulation environment. This intermediate group relative policy optimization reinforcement learning controller is then deployed to a real hardware system and fine-tuned online using actual flight data. The adversarial examples are used to improve the generalization ability of the group relative policy optimization reinforcement learning controller. A first airdrop control command is used to modify a second airdrop control command to obtain a target airdrop control command. The control execution module executes the airdrop action according to the target airdrop control command and returns to the step of acquiring the real-time state and environmental information of the airdrop system collected by the perception module, until the airdrop mission is completed. As described above, the GRPO-based intelligent control device for airdrop systems of this invention introduces a group relative policy optimization reinforcement learning controller, significantly improving the autonomous decision-making ability, dynamic environmental adaptability, and high-precision landing performance of the airdrop process. It requires no manual intervention, can dynamically adjust the control strategy according to environmental changes, and offers high control accuracy and safety. This alleviates the technical problems of poor autonomous adaptability, the need for manual intervention, limited control accuracy, and insufficient safety in traditional intelligent control processes for airdrop systems.

[0053] Optionally, the reinforcement learning controller for group relative policy optimization includes: multiple agents; each agent represents a control policy; the device is also used to: initialize multiple initial agents; run each initial agent and record its behavior trajectory and calculate the corresponding reward value; calculate the relative reward of each initial agent based on the reward value corresponding to each initial agent; adjust the parameters of the initial agents based on the relative rewards of each initial agent, thereby obtaining the agents.

[0054] Optionally, the device is also used to: calculate a formula based on the reward value. Calculate the reward value, where, Indicates the reward value. This represents the Euclidean distance between the current position and the target position at time t. Indicates the attenuation distance. Indicates the instantaneous velocity upon landing. Indicates the safe landing speed. Indicates cumulative energy consumption. Indicates maximum energy consumption. , , This represents the weighting coefficient.

[0055] Optionally, the device is also used to: calculate a formula based on relative rewards. Calculate the relative reward for each initial agent, where, This represents the relative reward for the i-th initial agent. This represents the reward value corresponding to the i-th initial agent. This represents the average reward value for the group. This represents the standard deviation of the group reward value.

[0056] Optionally, the device is also used to: adjust the formula according to parameters. Adjust the parameters of the initial agent, where, This indicates the adjusted parameters. This indicates the parameters before adjustment. Indicates the master learning rate. Represents the entropy regularization coefficient. Represents the initial intelligent agent. The parameters represent the initial agent. The information entropy function is represented by... , This represents the action vector of the i-th initial agent. Let represent the state vector of the i-th initial agent. This represents the relative reward for the i-th initial agent. This indicates the initial number of agents.

[0057] Optionally, the first acquisition unit is further configured to: when the altitude is greater than a first altitude, use a first airdrop control command to quickly stabilize the cargo attitude; when the altitude is less than the first altitude but greater than a second altitude, use the first airdrop control command to counteract the effects of crosswinds and turbulence, ensuring that the cargo approaches the target area along a predetermined trajectory; when the altitude is less than the second altitude, use the first airdrop control command to slow down the descent speed and avoid potential obstacles.

[0058] Optionally, the correction unit is also used to: according to the correction formula The second airdrop control command is modified to obtain the target airdrop control command, wherein, Indicates the target airdrop control command. This represents the proportional gain matrix in the second airdrop control command. This represents the differential gain matrix in the second airdrop control command. Indicates state error, This indicates the first airdrop control command.

[0059] The device provided in this embodiment of the invention has the same implementation principle and technical effect as the aforementioned method embodiment. For the sake of brevity, any parts not mentioned in the device embodiment can be referred to the corresponding content in the aforementioned method embodiment.

[0060] like Figure 5 As shown in the embodiment of this application, an electronic device 600 includes a processor 601, a memory 602, and a bus. The memory 602 stores machine-readable instructions that can be executed by the processor 601. When the electronic device is running, the processor 601 communicates with the memory 602 through the bus. The processor 601 executes the machine-readable instructions to perform the steps of the intelligent control method for the airdrop system based on GRPO reinforcement learning described above.

[0061] Specifically, the memory 602 and processor 601 mentioned above can be general-purpose memory and processor, without any specific limitations. When the processor 601 runs the computer program stored in the memory 602, it can execute the above-mentioned intelligent control method for the airdrop system based on GRPO reinforcement learning.

[0062] The processor 601 may be an integrated circuit chip with signal processing capabilities. In implementation, each step of the above method can be completed by the integrated logic circuitry in the hardware of the processor 601 or by instructions in software form. The processor 601 may be a general-purpose processor, including a Central Processing Unit (CPU), a Network Processor (NP), etc.; it may also be a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field-Programmable Gate Array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor may be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this application can be directly manifested as execution by a hardware decoding processor, or execution by a combination of hardware and software modules in the decoding processor. The software module can reside in a mature storage medium in the art, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, or registers. This storage medium is located in memory 602, and processor 601 reads the information from memory 602 and, in conjunction with its hardware, completes the steps of the above method.

[0063] Corresponding to the above-mentioned intelligent control method for airdrop systems based on GRPO reinforcement learning, this application embodiment also provides a computer-readable storage medium storing machine-executable instructions. When the machine-executable instructions are called and run by a processor, the machine-executable instructions cause the processor to perform the steps of the above-mentioned intelligent control method for airdrop systems based on GRPO reinforcement learning.

[0064] The intelligent control device for an airdrop system based on GRPO reinforcement learning provided in this application embodiment can be specific hardware on the device or software or firmware installed on the device. The implementation principle and technical effects of the device provided in this application embodiment are the same as those in the foregoing method embodiments. For the sake of brevity, any parts not mentioned in the device embodiment can be referred to the corresponding content in the foregoing method embodiments. Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can all be referred to the corresponding processes in the above method embodiments, and will not be repeated here.

[0065] In the embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. The apparatus embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. Furthermore, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Additionally, the displayed or discussed mutual couplings, direct couplings, or communication connections may be through some communication interfaces; indirect couplings or communication connections between devices or units may be electrical, mechanical, or other forms.

[0066] For example, the flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of apparatus, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram and / or flowchart, and combinations of blocks in block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0067] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0068] In addition, the functional units in the embodiments provided in this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0069] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause an electronic device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the intelligent control method for the airdrop system based on GRPO reinforcement learning described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0070] It should be noted that similar labels and letters in the following figures indicate similar items. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures. In addition, the terms "first", "second", "third", etc. are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0071] Finally, it should be noted that the above-described embodiments are merely specific implementations of this application, used to illustrate the technical solutions of this application, and not to limit them. The protection scope of this application is not limited thereto. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that any person skilled in the art can still modify or easily conceive of changes to the technical solutions described in the foregoing embodiments, or make equivalent substitutions for some of the technical features, within the scope of the technology disclosed in this application; and these modifications, changes, or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this application. All should be covered within the protection scope of this application. Therefore, the protection scope of this application should be determined by the protection scope of the claims.

Claims

1. A method for intelligent control of an airdrop system based on GRPO reinforcement learning, characterized in that, The airdrop system includes: a perception module, a reinforcement learning controller for swarm relative policy optimization, a PID controller, and an execution module; the method includes: The system acquires the real-time status and environmental information of the airdrop system collected by the sensing module, and obtains the future environmental changes predicted by the dynamic environment model based on the status and environmental information. The first airdrop control command generated by the reinforcement learning controller with the group relative policy optimization based on the state information, the environment information, and the future environment changes is obtained, and the second airdrop control command generated by the PID controller based on the state information and the environment information is obtained. The reinforcement learning controller with the group relative policy optimization is obtained by training the original reinforcement learning controller with the group relative policy optimization in a simulation environment by introducing adversarial examples to obtain an intermediate reinforcement learning controller with the group relative policy optimization, and then deploying the intermediate reinforcement learning controller with the group relative policy optimization to a real hardware system and fine-tuning it online using actual flight data. The adversarial examples are used to improve the generalization ability of the reinforcement learning controller with the group relative policy optimization. The first airdrop control command is used to modify the second airdrop control command to obtain the target airdrop control command; The execution module is controlled to perform the airdrop action according to the target airdrop control command, and then returns to the step of obtaining the status information and environmental information of the airdrop system collected in real time by the sensing module, until the airdrop mission is completed.

2. The method according to claim 1, characterized in that, The reinforcement learning controller for group relative policy optimization includes: multiple agents; each agent represents a control policy; The training process for the multiple agents includes: Initialize multiple initial agents; Run each of the initial agents and record its behavioral trajectory and calculate the corresponding reward value; Calculate the relative reward for each initial agent based on the reward value corresponding to each initial agent; The parameters of the initial agents are adjusted according to the relative rewards of each initial agent, thereby obtaining the agent.

3. The method according to claim 2, characterized in that, Calculate the corresponding reward value, including: Calculation formula based on reward value Calculate the reward value, where, This represents the reward value. This represents the Euclidean distance between the current position and the target position at time t. Indicates the attenuation distance. Indicates the instantaneous velocity upon landing. Indicates the safe landing speed. Indicates cumulative energy consumption. Indicates maximum energy consumption. , , This represents the weighting coefficient.

4. The method according to claim 2, characterized in that, Calculate the relative reward for each initial agent based on the reward value corresponding to each initial agent, including: According to the relative reward calculation formula Calculate the relative reward for each of the initial agents, where, This represents the relative reward for the i-th initial agent. This represents the reward value corresponding to the i-th initial agent. This represents the average reward value for the group. This represents the standard deviation of the group reward value.

5. The method according to claim 2, characterized in that, Adjusting the parameters of the initial agents based on their relative rewards includes: Adjust the formula according to the parameters Adjust the parameters of the initial agent, wherein, This indicates the adjusted parameters. This indicates the parameters before adjustment. Indicates the master learning rate. Represents the entropy regularization coefficient. This refers to the initial intelligent agent. The parameters represent the initial agent. Represents the information entropy function. , This represents the action vector of the i-th initial agent. Let represent the state vector of the i-th initial agent. This represents the relative reward for the i-th initial agent. This indicates the number of initial intelligent agents.

6. The method according to claim 1, characterized in that, The reinforcement learning controller for obtaining the relative policy optimization of the population generates a first airdrop control command based on the state information, the environment information, and the future environment changes, including: When the height is greater than the first height, the first airdrop control command is used to quickly stabilize the cargo's attitude; When the height is less than the first height but greater than the second height, the first airdrop control command is used to counteract the effects of crosswinds and turbulence, ensuring that the cargo approaches the target area along a predetermined trajectory. When the height is less than the second height, the first airdrop control command is used to slow the descent speed and avoid potential obstacles.

7. The method according to claim 1, characterized in that, The first airdrop control command is used to modify the second airdrop control command to obtain the target airdrop control command, including: According to the revised formula The second airdrop control command is modified to obtain the target airdrop control command, wherein, This indicates the target airdrop control command. This represents the proportional gain matrix in the second airdrop control command. This represents the differential gain matrix in the second airdrop control command. Indicates state error, This indicates the first airdrop control command.

8. An intelligent control device for an airdrop system based on GRPO reinforcement learning, characterized in that, The airdrop system includes: a perception module, a reinforcement learning controller for swarm relative policy optimization, a PID controller, and an execution module; the device includes: The first acquisition unit is used to acquire the status information and environmental information of the airdrop system collected in real time by the sensing module, and to acquire the future environmental changes predicted by the dynamic environment model based on the status information and the environmental information. The second acquisition unit is used to acquire the first airdrop control command generated by the reinforcement learning controller for group relative policy optimization based on the state information, the environment information, and the future environment changes, and to acquire the second airdrop control command generated by the PID controller based on the state information and the environment information. The reinforcement learning controller for group relative policy optimization is obtained by training the original reinforcement learning controller for group relative policy optimization in a simulation environment using adversarial examples to obtain an intermediate reinforcement learning controller for group relative policy optimization. This intermediate reinforcement learning controller is then deployed to a real hardware system and fine-tuned online using actual flight data. The adversarial examples are used to improve the generalization ability of the reinforcement learning controller for group relative policy optimization. The correction unit is used to correct the second airdrop control command using the first airdrop control command to obtain the target airdrop control command; The control unit is used to control the execution module to perform airdrop actions according to the target airdrop control command, and return to the step of obtaining the status information and environmental information of the airdrop system collected in real time by the sensing module, until the airdrop mission is completed.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores machine-executable instructions that, when invoked and executed by a processor, cause the processor to perform the method according to any one of claims 1 to 7.

Images