Water-air cross-medium unmanned aerial vehicle control modeling method based on reinforcement learning and related device

Abstract

The invention discloses a reinforcement learning-based control modeling method for a water-air cross-medium unmanned aerial vehicle and a related device, and the method comprises the steps: carrying out the interaction of a strategy network and a simulation environment, and generating a series of trajectory data; predicting a state value according to the trajectory data through a value network, and calculating a corresponding dominant function and a target value function based on the state value; calculating a value loss item, a strategy loss item and a strategy regular item through a strategy optimization algorithm so as to construct a total loss function, and performing joint training update on the strategy network and the value network by taking minimization of the total loss function as a target to obtain the updated strategy network and the updated value network; and judging whether the updated strategy network and the updated value network meet a preset dual convergence condition, if so, taking the updated strategy network and the updated value network as a water-air cross-medium unmanned aerial vehicle control model, and guiding the unmanned aerial vehicle to realize smooth and stable autonomous control in air, underwater and transition stages. And the intelligent autonomous control level of the cross-medium unmanned aerial vehicle is improved.

Description

Technical Field

[0001] This invention relates to the field of unmanned aerial vehicle (UAV) control technology, and in particular to a control modeling method and related apparatus for a water-air cross-medium UAV based on reinforcement learning. Background Technology

[0002] In recent years, cross-medium unmanned aerial vehicles (UAVs) have attracted widespread attention due to their ability to operate both in the air and underwater in missions such as marine exploration, environmental monitoring, maritime rescue, and military reconnaissance. A key challenge in the motion of cross-medium UAV platforms lies in the drastic abrupt changes in physical quantities at the medium boundary: when the platform crosses the water surface, drag, buoyancy, added mass, and available effective thrust change by orders of magnitude in a short period, resulting in highly time-varying and nonlinear dynamics and transport characteristics. This directly affects sensor readings (e.g., underwater sensors are disturbed the moment they emerge from the water), actuator effectiveness (propeller efficiency differs significantly between air and water), and control robustness. Traditional PID (Proportional-Integral-Derivative Controller) or MPC (Model Predictive Control) algorithms struggle to handle such highly nonlinear and time-varying systems. Currently, cross-medium UAVs are typically controlled using precise dynamic segmented control technology, requiring the establishment of accurate dynamic models for both air and water media, and controller switching during medium transitions. This approach heavily relies on model accuracy, but the hydrodynamic characteristics of the water-to-air transition process are complex, highly nonlinear, and difficult to model, leading to severe model mismatch. Furthermore, the numerous control parameters and cumbersome manual adjustments require extensive cross-disciplinary experience to ensure stable control. At the moment of medium switching, control commands are prone to abrupt changes, causing system oscillations or even instability, making smooth and continuous cross-medium transition control difficult. Therefore, existing water-to-air cross-medium UAV control technologies are highly dependent on dynamic models, making it difficult to achieve stable control of water-to-air cross-medium UAVs. Summary of the Invention

[0003] This invention provides a control modeling method and related apparatus for a water-air cross-medium unmanned aerial vehicle (UAV) based on reinforcement learning, which solves the technical problem that existing water-air cross-medium UAV control technologies are highly dependent on dynamic models and difficult to achieve stable control of water-air cross-medium UAVs.

[0004] This invention provides a control modeling method for a water-air cross-medium unmanned aerial vehicle based on reinforcement learning, the method comprising:

[0005] Initialize the policy network and value network, and construct a simulation environment; interact the policy network with the simulation environment to generate a series of trajectory data;

[0006] The value network predicts the state value based on the trajectory data, calculates the corresponding advantage function and target value function based on the state value, generates a series of experience data and stores it in the experience replay cache pool;

[0007] Random batch sampling is performed in the experience replay cache pool. Based on the sampled experience data, the value loss term, policy loss term, and policy regularization term are calculated using a policy optimization algorithm. The value loss term, policy loss term, and policy regularization term are combined to construct a total loss function. With the goal of minimizing the total loss function, the policy network and value network are jointly trained and updated to obtain the updated policy network and value network.

[0008] Determine whether the updated policy network and value network satisfy the preset dual convergence condition. If they do, use the updated policy network and value network as the control model for the water-air cross-medium UAV and output it.

[0009] Optionally, the step of interacting the policy network with the simulation environment to generate a series of trajectory data includes:

[0010] The initial multimodal observation state of the UAV is set, and the policy network interacts with the simulation environment. For each interaction time step, the current multimodal observation state is input into the policy network to generate the corresponding action command, the reward function generated by the action command in the simulation environment is calculated, and the multimodal observation state corresponding to the next interaction time step is collected. This process continues until the interaction reaches a preset number of time steps, generating a series of trajectory data.

[0011] The series of trajectory data are represented as follows:

[0012]

[0013] In the formula: Represents a series of trajectory data; This indicates the multimodal observation state at time step t. Issue an action command Afterwards, the environment transitions to the multimodal observation state at time step t+1. and obtain reward function ; Indicates the termination flag; This represents the cache capacity.

[0014]

[0015] In the formula: , , and All are reward weighting coefficients; Indicates navigation reward, Indicates energy penalty. Indicates a stability reward. This indicates a cross-media reward.

[0016] Optionally, the step of predicting state value based on the trajectory data through a value network, calculating the corresponding advantage function and target value function based on the state value, generating a series of empirical data and storing them in an experience replay cache pool includes:

[0017] The value network predicts the state value at different time steps based on the trajectory data.

[0018] Calculate the corresponding temporal difference error based on the state value at adjacent time steps;

[0019] The generalized dominance estimation method is used to calculate the corresponding dominance function based on the time-series difference error;

[0020] The target value function is calculated using the state value and advantage function at the same time step;

[0021] A series of empirical data are generated based on the trajectory data, the advantage function, and the target value function, and stored in the empirical replay cache pool.

[0022] Optionally, the step of randomly sampling in the experience replay cache pool and calculating the value loss term, policy loss term, and policy regularization term based on the sampled experience data using a policy optimization algorithm includes:

[0023] Randomly sample in batches from the experience replay cache pool to obtain small batches of experience data;

[0024] The value loss term is calculated based on the target value function in the small batch of empirical data using a strategy optimization algorithm; the value loss term is used to guide the state value predicted by the value network to approximate the target value function.

[0025] The policy ratio is calculated based on the multimodal observation states and action commands in the small batch of empirical data; the policy loss term is calculated based on the policy ratio and the corresponding dominance function through a pruning mechanism; the policy loss term is used to constrain the update magnitude of the policy network;

[0026] The policy regularization term is determined based on the action probability distribution of the current policy network under multimodal observation states corresponding to small batches of empirical data; the policy regularization term is used to encourage the randomness of the policy network.

[0027] Optionally, the total loss function is calculated as follows:

[0028]

[0029] In the formula: Represents the total loss function; Indicates in the parameter New Strategy Network The strategy loss term, Indicates a value loss item. This represents the policy entropy regularization term; and All are loss weighting coefficients;

[0030] in,

[0031]

[0032] In the formula: This indicates the use of small batches of empirical data. Find the expected value; The time step t parameter represents the time step parameter. New Strategy Network Strategy ratio; This represents the dominance function at time step t; Indicates the strategy ratio Values ​​are clipped to [ ]Inside; This represents the clipping hyperparameter;

[0033] in,

[0034]

[0035] In the formula: This represents the average expectation at time step t; Indicates multimodal observation state State value, Indicates multimodal observation state The target value function is as follows.

[0036] Optionally, the step of determining whether the updated policy network and value network satisfy the preset dual convergence condition, and if so, using the updated policy network and value network as the control model for the water-air cross-medium UAV and outputting it, includes:

[0037] Determine whether the updated policy network and value network satisfy the internal convergence condition. If they do not satisfy the internal convergence condition, proceed to the step of randomly sampling in the experience replay cache pool, and calculating the value loss term, policy loss term, and policy regularization term based on the sampled experience data using the policy optimization algorithm.

[0038] If the internal convergence condition is met, then determine whether the updated policy network and value network meet the external convergence condition. If the external convergence condition is not met, proceed to the step of interacting the policy network with the simulation environment to generate a series of trajectory data. If the external convergence condition is met, then use the updated policy network and value network as the control model for the water-air cross-medium UAV and output it.

[0039] This invention also provides a control modeling system for a water-air cross-medium unmanned aerial vehicle based on reinforcement learning, the system comprising:

[0040] The simulation interaction module is used to initialize the policy network and value network, and build a simulation environment; the policy network interacts with the simulation environment to generate a series of trajectory data;

[0041] The experience data acquisition module is used to predict state value based on the trajectory data through the value network, calculate the corresponding advantage function and target value function based on the state value, generate a series of experience data and store them in the experience playback cache pool;

[0042] The policy optimization module is used to randomly sample in batches from the experience replay cache pool, and calculate the value loss term, policy loss term, and policy regularization term based on the sampled experience data through the policy optimization algorithm; construct a total loss function by combining the value loss term, the policy loss term, and the policy regularization term, and perform joint training and updating of the policy network and the value network with the goal of minimizing the total loss function, so as to obtain the updated policy network and value network.

[0043] The control model generation and output module is used to determine whether the updated policy network and value network meet the preset dual convergence conditions. If they do, the updated policy network and value network are used as the control model for the water-air cross-medium UAV and output.

[0044] This invention also provides a computer device, including a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the control modeling method for a transmedium-water unmanned aerial vehicle as described above.

[0045] This invention also provides a computer-readable storage medium storing a computer program or instructions thereon, which, when executed by a processor, implements the steps of the control modeling method for a transmedium-water unmanned aerial vehicle as described above.

[0046] This invention also provides a computer program product, including a computer program or instructions, which, when executed by a processor, implement the steps of the control modeling method for a transmedium-water unmanned aerial vehicle as described above.

[0047] As can be seen from the above technical solutions, the present invention has the following advantages:

[0048] This invention provides a control modeling method and related apparatus for a water-air cross-medium unmanned aerial vehicle (UAV) based on reinforcement learning. The method includes: initializing a policy network and a value network, and constructing a simulation environment; interacting the policy network with the simulation environment to generate a series of trajectory data; predicting state values ​​based on the trajectory data using the value network, calculating the corresponding advantage function and target value function based on the state values, generating a series of empirical data and storing it in an experience replay cache; randomly sampling in batches from the experience replay cache, and calculating the value loss term, policy loss term, and policy regularization term based on the sampled empirical data using a policy optimization algorithm; constructing a total loss function by jointly selecting the value loss term, policy loss term, and policy regularization term, and jointly training and updating the policy network and value network with the goal of minimizing the total loss function, obtaining the updated policy network and value network; determining whether the updated policy network and value network satisfy a preset dual convergence condition, and if so, using the updated policy network and value network as the control model for the water-air cross-medium UAV and outputting it. In this invention, the PPO algorithm is used to learn cross-medium control strategies from high-fidelity simulation interaction, effectively guiding UAVs to achieve smooth and stable autonomous control in air, underwater and transition phases. This avoids dependence on complex fluid dynamics precise mathematical models, significantly improves the intelligent autonomous control level of cross-medium UAVs, and solves the technical problem that existing water-air cross-medium UAV control technologies have a high dependence on dynamic models and are difficult to achieve stable control of water-air cross-medium UAVs. Attached Figure Description

[0049] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0050] Figure 1 A flowchart illustrating the steps of a control modeling method for a transmedium unmanned aerial vehicle (UAV) based on reinforcement learning, provided in an embodiment of the present invention.

[0051] Figure 2 This is a schematic diagram illustrating the motion process of the unmanned aerial vehicle (UAV) body in a simulation experiment, provided by an embodiment of the present invention.

[0052] Figure 3 This invention provides an analysis chart showing the trend of average round reward changing with iteration rounds during training.

[0053] Figure 4This is a diagram showing the change in Gaussian policy noise during training, provided as an embodiment of the present invention.

[0054] Figure 5 This is a diagram showing the average survival time of a drone during training, provided by an embodiment of the present invention.

[0055] Figure 6 The present invention provides a structural block diagram of a control modeling system for a transmedium unmanned aerial vehicle based on reinforcement learning. Detailed Implementation

[0056] This invention provides a control modeling method and related apparatus for a water-air cross-medium unmanned aerial vehicle (UAV) based on reinforcement learning, which solves the technical problem that existing water-air cross-medium UAV control technologies are highly dependent on dynamic models and difficult to achieve stable control of water-air cross-medium UAVs.

[0057] To make the objectives, features, and advantages of this invention more apparent and understandable, the technical solutions of the embodiments of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the embodiments described below are only some embodiments of this invention, and not all embodiments. Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this invention.

[0058] It should be noted that, in the optional embodiments of the present invention, the data related to object information, etc., requires the permission or consent of the object when the embodiments of the present invention are applied to specific products or technologies. Furthermore, the collection, use, and processing of the relevant data must comply with the relevant laws, regulations, and standards of the relevant countries and regions. In other words, if the embodiments of the present invention involve data related to an object, it needs to be obtained with the object's authorization and consent, the authorization and consent of relevant departments, and in accordance with the relevant laws, regulations, and standards of the country and region. If the embodiments involve personal information, the acquisition of all personal information requires the individual's consent. If sensitive information is involved, the separate consent of the information subject is required. The embodiments also need to be implemented with the object's authorization and consent.

[0059] Intelligent Agent: A complete reinforcement learning model including a policy network and a value network; the policy network is the "brain" of the intelligent agent, directly responsible for generating action instructions based on the current state observation, and is the entity that interacts with the environment in real time; the value network is the "evaluation system" of the intelligent agent. It does not directly participate in the interaction, but provides a crucial value benchmark and advantage estimate for the optimization of the policy network, and is an indispensable part of the training process.

[0060] Please see Figure 1This invention provides a control modeling method for a water-air cross-medium unmanned aerial vehicle based on reinforcement learning. The method includes:

[0061] Step 101: Initialize the policy network and value network, and build a simulation environment; interact the policy network with the simulation environment to generate a series of trajectory data.

[0062] It should be noted that both the policy network and the value network are neural network models. The policy network receives the state of the environment as input and then outputs the corresponding action selection; the value network receives the state and action of the environment as input and then outputs the state value of executing the action in that state. In this embodiment, the initialization of the policy network and the value network can be performed by setting preset parameters or random parameters. For example, these parameters can be network weights and biases, which can be set by preset values ​​or randomly generated initial values.

[0063] Simultaneously, a simulation environment is built using the NVIDIA Isaac Lab simulation platform. The NVIDIA Isaac Lab platform provides a high-fidelity physical simulation environment capable of simulating the complex physical effects of water and air media and their transition processes. Within the NVIDIA Isaac Lab platform, the simulation environment needs to consider a UAV model, which involves rigid body dynamics models, propeller thrust models, and key hydrodynamic effects. This establishes a high-fidelity physical simulation environment encompassing air, water, and their transition boundaries, enabling accurate simulation of the UAV's attitude, velocity, thrust, and other physical characteristics.

[0064] 1) Rigid body dynamics model: The six-degree-of-freedom motion of the UAV is described using the Newton-Euler equations, and its state vector x is:

[0065] (1)

[0066] In the formula: This refers to the position in the world coordinate system. A quaternion representing attitude, and These are the linear velocity and angular velocity in the body coordinate system, respectively.

[0067] 2) Propeller Thrust Model: The propeller thrust model is a key actuator model connecting the agent's control commands and physical simulation. This model receives normalized motor commands from the agent and, based on the current fluid medium density (air or water), converts them into actual thrust and torque acting on the UAV body, thereby driving the UAV's six-degree-of-freedom motion in a cross-medium environment. This model is one of the fundamental physical modules for realizing hydro-air dynamic coupling and smooth transition control.

[0068] 3) Hydrodynamic Effects: When a UAV moves through water and air across media, the fluid exerts various forces and torques on the UAV. The multi-media hydrodynamics resulting from these effects is a crucial part of the simulation; specifically, different physical materials are set up in the simulation environment, and their densities are defined. With viscosity coefficient This allows us to distinguish between the two media: air and water.

[0069] Multi-media hydrodynamics acting on the body of the UAV It can be calculated using the following formula:

[0070] (2)

[0071] In the formula: For fluid resistance, The buoyancy force can be expressed as: ,in For drainage volume, It is the acceleration due to gravity; The effects of added mass on the organism; For wave resistance.

[0072] Fluid resistance The calculation is based on the drone's object characteristics and frontal area. With dimensionless drag coefficient The expression is as follows:

[0073] (3)

[0074] in, The velocity-dependent form, which considers the coupling effect of linear and quadratic damping, can be written as:

[0075] (4)

[0076] In the formula, The velocity scalar of the object relative to the fluid; and These are the linear damping coefficient and the quadratic damping coefficient, respectively, reflecting the contributions of viscous drag and inertial drag of the fluid. Equation (4) can smoothly describe the drag characteristics in different speed ranges: when the speed is low, the linear damping term dominates the drag; while in the high-speed range, the influence of the quadratic damping term is more significant, thus more accurately reflecting the hydrodynamic changes of an object moving in water.

[0077] Among them, the effect of added mass on the organism The calculation formula is:

[0078] (5)

[0079] In the formula: Add a mass matrix to the six degrees of freedom. The acceleration vector of the organism. The Coriolis and centripetal force matrices are added. Since the designed UAV model is axisymmetric, the added mass matrix can be simplified to a diagonal form, as shown below:

[0080] (6)

[0081] In the formula: , , , , and These are all additional mass coefficients corresponding to the degrees of freedom, and can be obtained through empirical parameters by CFD modeling to complete the design.

[0082] When a cross-medium unmanned aerial vehicle (UAV) navigates on or near the water surface, its motion generates surface gravity waves, producing wave-generating drag that cannot be ignored. According to the model recommended by the International Towed Pool Conference (ITTC) (i.e., the ITTC wave-generating resistance model), wave-generating resistance... It can be represented as:

[0083] (7)

[0084] In the formula: For the density of water, It is the acceleration due to gravity. For drainage volume, For sailing speed; The wave-generating drag coefficient, whose value strongly depends on the Froude number. :

[0085] (8)

[0086] In the formula: The characteristic length of the organism (usually the waterline length);

[0087] Wave drag coefficient It exhibits non-monotonic changes: when At that time, the wave-generating resistance is relatively small; A peak appears in the interval, corresponding to the resonant interference of the wave systems at the bow and stern of the ship; as... Enter gliding mode. The drag gradually decreases. For cross-medium mission scenarios, the ITTC wave-generating drag model provided in this embodiment can accurately capture the drag abrupt change characteristics of the UAV during the water entry / exit transition phase, providing key dynamic load information for controller design.

[0088] The agent interacts in parallel within the Isaac Lab simulation environment constructed above, generating a series of trajectory data and storing them in an experience replay cache. This trajectory data includes multimodal state observation data (multimodal state observation states), action data (action commands), and reward data (reward function). The multimodal state observation data provides a comprehensive environmental perception foundation for the agent's decision-making. The action data consists of normalized motor thrust commands generated by the policy network based on the current state, directly driving the actuator model to produce actual thrust and torque, thereby achieving six-degree-of-freedom motion control of the UAV. The reward data provides immediate policy optimization guidance for the agent's decision-making; it is a scalar feedback signal calculated by the simulation environment based on a hierarchical reward function, used to guide the policy network to continuously improve in the direction of maximizing long-term cumulative rewards.

[0089] In one specific implementation, the process of interacting the policy network with the simulation environment to generate a series of trajectory data may include the following steps:

[0090] S11. Set the initial multimodal observation state of the UAV and interact the policy network with the simulation environment;

[0091] S12. For each interaction time step, input the current multimodal observation state into the policy network to generate the corresponding action command, calculate the reward function generated by the action command in the simulation environment, and collect the multimodal observation state corresponding to the next interaction time step; until the interaction reaches the preset number of time steps, a series of trajectory data are generated.

[0092] Specifically, at the interaction time step t, the multimodal observation state It is a feature vector that integrates information from multiple sensors (such as aerodynamic sensors, underwater sensors, and attitude sensors), and it is represented as:

[0093] (9)

[0094] In the formula: Let t be the position coordinates of the UAV's center of mass in the world coordinate system. , The unit quaternion for the body attitude at time step t , The linear velocity vector in the body coordinate system at time step t. , Let be the angular velocity vector in the body coordinate system at time step t. , The three-axis acceleration of the body measured by the inertial measurement unit at time step t. ; The hydrostatic pressure of the fluid, measured by a pressure sensor, is used in conjunction with the ambient fluid density to estimate the depth information of the UAV. This is a medium identifier indicating the degree of submersion of the propeller blades.

[0095] Multimodal observation status It integrates multi-source information such as the drone's pose, motion, and water contact status, providing a comprehensive environmental perception foundation for intelligent agent decision-making.

[0096] At interaction time step t, the agent outputs action commands. The normalized values ​​for the four propeller thrust commands are: ,in , for The motor command value, in the range of Negative values ​​represent reverse thrust. The thrust and rotational speed of each blade are then calculated to convert the thrust into a propeller thrust model, as shown in the following formula:

[0097] (10)

[0098] In the formula, Let be the thrust of the i-th propeller; The thrust coefficient is a dimensionless constant related to the propeller's geometry. The density of the current medium (air or water). This is the maximum speed of the motor. The diameter is the propeller diameter. The spatial design of this motion command allows the agent to generate the required roll, pitch, and yaw moments through differential thrust, thereby achieving six-degree-of-freedom omnidirectional motion control.

[0099] In this specific embodiment, the reward function adopts a hierarchical reward structure design, aiming to guide the agent to learn complex skills of cross-media flight and ensure the global optimization of the strategy across multimodal tasks; at the interaction time step t, the reward function... It consists of a weighted sum of multiple objectives:

[0100] (11)

[0101] In the formula, For navigation rewards, including underwater navigation rewards and air navigation rewards The underwater phase employs a Gaussian function design. Encourage agents to move towards the target point Motion, its value is proportional to the reduction in distance from the agent to the target point, and is not directly directed towards an aerial target; during the aerial phase, combining distance and velocity direction, the aerial navigation reward is... ,in Let the target direction be the unit vector. These are the weighting coefficients of the Gaussian term. is the scaling parameter (standard deviation) of the Gaussian function. Weighting coefficients for the velocity direction term; navigation reward The design encourages movement toward the target and rewards effective movement along the target direction, avoiding back-and-forth oscillations.

[0102] As an energy penalty, The energy penalty term introduces underwater and air-gated variables, where... and These are gated variables in water and gated variables in air, respectively. This is the control input vector for the underwater phase. This is the control input vector for the air phase; As energy penalty weight, For control input changes that are not adjacent to each other, ,in ; To compensate for excessive consumption, a nonlinear penalty is introduced to correspond to the response speed coefficient. When the control input is large, the penalty will increase significantly, forcing the agent to prioritize "small amplitude, high efficiency" power output.

[0103] As a stability reward, ;in This is the roll angle. The pitch angle, The penalty weight for attitude angle deviation is... This is the angular velocity vector in body coordinates. The penalty weight for angular velocity, and the stability reward. Nonlinear penalties are used to address unstable states such as excessive angular velocity and excessive tilt in attitude, thereby enhancing flight robustness.

[0104] For cross-media rewards, ,in A reward will be given for successful water extraction. The condition for successful water discharge is defined as follows: its value is... , The lowest point height of the aircraft. The height of the water surface. To ensure a safe margin for water discharge and avoid misjudgments of incomplete water discharge, To establish a safe threshold for the water exit attitude, rewards are only given when the height and attitude meet the standard and it is the first time the event is triggered, thus avoiding duplicate rewards. As a punishment for accidental water entry, among which The hefty penalty for re-entry into the water. ,in The average immersion degree of the blades. To prevent duplicate penalties, once triggered, the marker will not be penalized again until the mission resets. Cross-media rewards include both successful water exit rewards and accidental re-entry penalties, used to reinforce the key objective of "successfully exiting the water and maintaining aerial status" through sparse, high-value feedback. By combining sparse rewards with large feedback, if an agent that has successfully exited the water re-enters the water (such as falling out of control in the air), a large reward will be immediately deducted to reinforce the goal of maintaining the aerial state.

[0105] in, , , and All of these are reward weights, which can be determined through extensive experimental debugging to balance the importance of different objectives. This hybrid reward system guides the agent to learn how to break through underwater obstacles, achieve aerial ascents, and hover precisely. Furthermore, the reward function design provides dense learning signals to enhance the agent's learning efficiency, while sparse rewards clarify the final task objective.

[0106] After collecting trajectory data, the trajectory data is stored in the Experience Replay Buffer. The Experience Replay Buffer is a key component in reinforcement learning, used to store and manage trajectory data generated by the agent's interaction with the environment. A series of trajectory data stored in the Experience Replay Buffer are represented as follows:

[0107] (12)

[0108] In the formula: A series of trajectory data cached in the experience replay cache pool; This indicates the multimodal observation state at time step t. Issue an action command Afterwards, the environment transitions to the multimodal observation state at time step t+1. and obtain reward function ; The termination flag is a Boolean variable that represents the state. Whether it is the end of the round (e.g., the drone crashed or successfully reached the target); This is the cache capacity (i.e., the preset time steps).

[0109] Understandably, the cache in the experience replay cache pool adopts a first-in-first-out queue management strategy. When the cache is full, new trajectory data will overwrite the oldest trajectory data. This makes it convenient for the algorithm to randomly sample a small batch of experience data from the cache during subsequent training. This random sampling breaks the temporal correlation between data, greatly improving the stability of training and the efficiency of sample utilization.

[0110] Step 102: Predict state value based on trajectory data through value network, calculate the corresponding advantage function and target value function based on state value, generate a series of experience data and store them in experience replay cache pool.

[0111] It should be noted that after the amount of trajectory data cached in the experience replay cache pool reaches the preset number of time steps, the value network can be used to predict the state value under each observation state using the collected trajectory data. Subsequently, based on the state value, the corresponding advantage function and target value function are calculated, providing an accurate and low-variance value benchmark and advantage signal for subsequent policy optimization. The experience data including state, action, reward, advantage, and target value and advantage are stored in the cache pool to support subsequent efficient and stable small-batch sampling and network updates.

[0112] In one specific implementation, step 102 may include the following steps:

[0113] S21. Predict the state value at different time steps based on trajectory data using a value network;

[0114] S22. Calculate the corresponding temporal difference error based on the state value at adjacent time steps;

[0115] S23. The generalized dominance estimation method is used to calculate the corresponding dominance function based on the time series difference error.

[0116] S24. Calculate the corresponding target value function using the state value and advantage function at the same time step;

[0117] S25. Generate a series of empirical data based on trajectory data, advantage function, and target value function, and store them in the experience replay cache pool.

[0118] In this specific embodiment, when the amount of data in the experience replay buffer reaches a preset number of steps, the value network is used to predict the state value under different time steps of multimodal observation using the collected trajectory data. The corresponding temporal difference error is calculated based on the state value, and then the dominance function is calculated using the generalized dominance estimation (GAE) algorithm. With the target value function .

[0119] Among them, the timing difference error at time step t The calculation method is expressed as follows:

[0120] (13)

[0121] In the formula: This is a discount factor used to weigh the importance of current rewards against future rewards; the closer its value is to 1, the more farsighted the agent is. For the value network based on the next state The predicted state value represents the value derived from the state. The expected cumulative return that can be obtained at the beginning; For the value network to the current state Value prediction. Time series difference error. Used to measure and predict the current value state With real rewards in one step And the next prediction The differences between the constituent objectives.

[0122] Subsequently, the generalized advantage estimation (GAE) method is used to estimate the time series difference error. Perform generalized advantage estimation and calculate the corresponding advantage function. ,Right now:

[0123] (14)

[0124] In the formula: Let be the dominance function at time step t; For GAE smoothing parameters, This constitutes an exponentially decaying weighting coefficient, which decreases with the step size. As the value increases, its impact gradually decreases; Equation (14) mainly uses exponential decay to weight the multi-step time series difference error, and then constructs the dominance function. Then quantify in the state Perform specific actions The relative effectiveness of the strategy compared to the average strategy.

[0125] Next, the value is estimated using the current state. Add a dominance function to measure the quality of the action. Calculate the target value function ,Right now:

[0126] (15)

[0127] In the formula: Current state The objective value function; the objective value function This provides a supervisory signal for the subsequent learning of the value network. It is used as a constant term in the calculation to prevent the training objective of the value function from changing continuously as the network updates, thus stabilizing the training. Combined with the currently collected trajectory data, it corrects the state... To make more accurate estimates, thereby improving the predicted values as close as possible Thus, they learn to evaluate any state more accurately. Its long-term value.

[0128] Step 103: Randomly sample in batches from the experience replay cache pool. Based on the sampled experience data, calculate the value loss term, policy loss term, and policy regularization term using the policy optimization algorithm. Construct the total loss function by combining the value loss term, policy loss term, and policy regularization term. With the goal of minimizing the total loss function, jointly train and update the policy network and the value network to obtain the updated policy network and value network.

[0129] It should be noted that the core of the policy optimization algorithm used in this embodiment is the Proximal Policy Optimization (PPO) algorithm, which achieves effective policy improvement while ensuring training stability. It uses batch-collected interaction experience data to construct stochastic gradient estimates and jointly updates the policy network and value network parameters in a small-batch, multi-round iterative manner.

[0130] The joint training mechanism provided in this embodiment achieves closed-loop optimization of policy improvement and value assessment: the value loss term drives the value network to provide an accurate value baseline, reducing estimation variance; the policy loss term encourages policy improvement while strictly limiting the update magnitude to avoid training oscillations; and the policy regularization term maintains the policy's exploratory capability. These three elements work together to ensure efficient and stable closed-loop optimization of the cross-media control policy towards the global optimum.

[0131] In one specific implementation, the process of randomly sampling in batches from the experience replay cache pool and calculating the value loss term, policy loss term, and policy regularization term based on the sampled experience data using a policy optimization algorithm may include the following steps:

[0132] S31. Randomly sample in batches from the experience replay cache pool to obtain small batches of experience data;

[0133] S32. Using a strategy optimization algorithm, calculate the value loss term based on the target value function in a small batch of empirical data; the value loss term is used to guide the state value predicted by the value network to approximate the target value function.

[0134] S33. Calculate the policy ratio based on the multimodal observation states and action commands in the small batch of empirical data; calculate the policy loss term based on the policy ratio and the corresponding advantage function through a pruning mechanism; the policy loss term is used to constrain the update magnitude of the policy network;

[0135] S34. Determine the policy regularization term based on the action probability distribution of the current policy network under the multimodal observation state corresponding to the small batch of empirical data; the policy regularization term is used to encourage the randomness of the policy network.

[0136] In each strategy optimization iteration, the experience replay cache pool is used. A small batch of empirical data B is randomly and uniformly sampled, i.e.:

[0137] (16)

[0138] In the formula, The size of the mini-batch, used for the number of gradient updates in a single operation; , , and This refers to the i-th batch of trajectory data sampled randomly. The value of the dominant function in the i-th batch of random samples. Let i be the target value function of the randomly sampled batch; For the old policy network at the i-th batch sampling time, the state is... Next action The logarithmic probability;

[0139] It is worth noting that random sampling of small batches of data can break the temporal correlation between sequential data, so that the empirical data used for updating can satisfy independent distribution, effectively improving the training stability and efficiency of the model network.

[0140] During training, the value network uses the target value function. As a monitoring signal, with the objective of minimizing value loss, the value loss item... Represented as:

[0141] (17)

[0142] In the formula: represents the average expectation at time step t; the value loss term guides the value network's predicted state value to approximate the target value function; minimizing the value loss term allows the value network to approximate the true state value, providing a low-variance baseline for advantage estimation, thereby reducing the variance of policy gradient estimation and improving training stability. It is obtained by forward computation of the value network.

[0143] It should be noted that this specific embodiment involves two policy networks. The old policy network is a copy that remains unchanged during the optimization loop to provide a stable benchmark. The new policy network is a policy network trained in real time. This design allows PPO to update the policy multiple times using old data, while using ratio pruning to constrain each update from deviating too far from the old policy, thereby ensuring the stability of training.

[0144] Specifically, this embodiment measures the difference between the old and new policies by introducing an importance policy ratio and limits it within a preset trust interval; the new policy network Based on the advantage function To improve the policy by adjusting the policy ratio, and to prevent the training from becoming unstable due to excessively large single update magnitudes, the PPO clipping agent objective can be used to constrain the updates.

[0145] Among them, the importance strategy ratio under normal circumstances Represented as:

[0146] (18)

[0147] In the formula: The parameter is Used to characterize the state The conditional probability distribution of taking an action; Indicates in the parameter New Strategy Network In state Conditions for taking action Probability distribution; Indicates in the parameter Old policy network In state Conditions for taking action Probability distribution. It is worth noting that, to improve numerical stability and facilitate the reuse of sampled data, this specific embodiment records the logarithmic probability of the old policy for each action during the sampling phase. The strategy ratio is calculated in the following manner during the update phase:

[0148] (19)

[0149] In the strategy optimization loop, the importance strategy ratio The magnitude of policy updates is quantified, and the policy ratio is used to measure whether the policy network should be updated. The update method is as follows:

[0150] (20)

[0151] Therefore, the strategy loss term can be expressed as:

[0152] (twenty one)

[0153] In the formula: Indicates in the parameter New Strategy Network The strategy loss term; This indicates the use of small batches of empirical data. Find the expected value; Indicates the strategy ratio Values ​​are clipped to [ ]Inside; This represents the pruning hyperparameter, which defines the trust zone boundary for policy updates. The policy loss term is used to constrain the update magnitude of the policy network.

[0154] The calculation formula for the clip function is expressed as follows:

[0155] (twenty two)

[0156] The importance of strategy ratio Cut off in interval Inside: when When the value exceeds this range, it is replaced with a boundary value, thereby limiting the magnitude of the change in policy probability relative to the old policy network in a single update; and through... Mechanism in the advantage function When the value is positive (encouraging an increase in probability) or negative (encouraging a decrease in probability), a more "conservative" target value is selected to avoid excessive increase / decrease in action probability, thereby improving training stability. This limits policy deviation while increasing the probability of high-dominance actions, ultimately achieving closed-loop optimization of "policy evaluation - policy improvement".

[0157] The strategy loss term is the pruning proxy objective of PPO, which aims to increase the probability of high-dominance actions and decrease the probability of low-dominance actions while simultaneously... Limit the shift between old and new strategies to suppress instability caused by excessive updates.

[0158] In each policy optimization loop, PPO jointly updates the policy network and value network based on the same batch of sampled empirical data. To balance stable updates and effective exploration, this invention adopts a joint objective of "policy pruning loss (i.e., policy loss term) + value regression loss (i.e., value loss term) + entropy regularization term (i.e., policy regularization term)," defining the minimized total loss function as:

[0159] (twenty three)

[0160] In the formula: Let be the total loss function, representing the optimization objective used for backpropagation and gradient update in one loop of PPO; and These are all loss weight coefficients, used to adjust the proportion of value regression and exploration intensity in the overall objective. They are usually determined as hyperparameters through experimental tuning to balance the relative importance of "strategy improvement - value assessment - exploration". This represents the policy entropy regularization term. During training, it aims to minimize the total loss function. The goal is to achieve a balance between strategy improvement, value assessment, and exploration capabilities.

[0161] Among them, the policy entropy regularization term Entropy, representing the policy distribution, is used to encourage policies to maintain a certain degree of randomness.

[0162] (twenty four)

[0163] (25)

[0164] In the formula: Indicates the state New Strategy Network The entropy of the output action probability distribution; Let be the entropy of the policy distribution; where the policy entropy regularization term is used to prevent the policy from converging prematurely to a deterministic action, improve exploration capability, and reduce the risk of getting trapped in local optima. Entropy of the policy distribution Used to measure the policy in a given state Adding "how random / uncertain" can encourage strategies to maintain a certain degree of randomness, preventing them from becoming "only capable of one action" too early, improving exploration capabilities, and reducing the risk of getting trapped in local optima.

[0165] Step 104: Determine whether the updated policy network and value network satisfy the preset dual convergence condition. If they do, use the updated policy network and value network as the control model for the water-air cross-medium UAV and output it.

[0166] In this embodiment, the preset dual convergence conditions include internal convergence conditions and external convergence conditions. The internal convergence condition is used to control the magnitude of a single policy and value network update in the policy optimization inner loop, while the external convergence condition is used to determine the performance convergence of the overall training process of the agent, thereby realizing the automatic termination of training and model output. By using the preset dual convergence conditions to determine the performance convergence, it can be ensured that the model stops in time after the policy performance stabilizes, avoiding overfitting and waste of computational resources. The final output is a water-air cross-medium UAV control model with high stability and generalization ability, which significantly improves training efficiency and deployment reliability.

[0167] In one specific implementation, step 104 may include the following steps:

[0168] S41. Determine whether the updated policy network and value network meet the internal convergence condition. If they do not meet the internal convergence condition, proceed to the next step of randomly sampling in the experience replay cache pool, and calculating the value loss term, policy loss term, and policy regularization term based on the sampled experience data using the policy optimization algorithm.

[0169] S42. If the internal convergence condition is met, determine whether the updated policy network and value network meet the external convergence condition. If the external convergence condition is not met, proceed to the step of interacting the policy network with the simulation environment to generate a series of trajectory data. If the external convergence condition is met, use the updated policy network and value network as the control model for the water-air cross-medium UAV and output it.

[0170] In this specific embodiment, satisfying the internal convergence condition can be either KL divergence convergence or the number of internal training iterations reaching a preset maximum batch size. For example, if KL divergence converges, the internal training loop is stopped to limit policy shift. Even if KL divergence does not converge, the preset maximum batch size can ensure that the internal optimization loop is bounded.

[0171] After completing the internal training loop, the algorithm does not immediately begin a new round of data collection. Instead, it first systematically evaluates the overall performance of the agent. Specifically, it calculates the average performance metric of the most recent training rounds and its sliding variance, using this as an objective and quantitative basis for determining whether the external training has converged. This allows for the automatic termination of the training process and the output of a high-performance final inference model. The external convergence condition can be either that the sliding variance of the average reward across K rounds reaches performance convergence or that the number of external training iterations reaches a preset maximum batch size.

[0172] To accurately assess training trends, a fixed-size [value] can be maintained. A sliding window used to store the most recent Average round prize of external training, number of rounds Average round reward The calculation formula is:

[0173] (26)

[0174] In the formula, For the first The total number of time steps collected during each training round (i.e., the number of steps required for the experience replay buffer pool to be filled). ); For the first During external training at any time The reward function obtained.

[0175] Store the most recent items sequentially using a sliding window. Average round reward ,Right now:

[0176] (27)

[0177] Among them, average round reward Used to characterize the central trend of current strategy performance;

[0178] Then the sliding variance of the average reward It can be represented as:

[0179] (28)

[0180] Where: the sliding variance of the average reward Used to measure recent The average round reward is around the mean. The fluctuation range (stability) as the strategy gradually converges. Changes within the window tend to decrease, thus It will decrease and approach a smaller value. Therefore, when At that time, among them For a pre-set variance threshold (e.g., take...) This is used to measure the degree of reward fluctuation. When triggered, it can be considered that the performance fluctuation is small enough and the training process tends to be stable, thus determining that the training has converged and triggering the termination condition, and thus determining that the training has successfully converged.

[0181] Understandably, when external training rounds Exceeding the preset maximum external batch setting At 500 or 1000 rounds (for example), training is forcibly terminated to prevent infinite loops due to performance oscillations and lack of convergence. This is a safety parameter that ensures the training process will eventually stop. If it terminates due to reaching the maximum number of epochs, it typically outputs the model parameters that achieved the best performance up to that epoch, rather than the parameters from the last epoch, serving as a simple performance guarantee mechanism.

[0182] After the updated policy network and value network satisfy the preset dual convergence conditions, the network parameters of the finally updated policy network and value network are separated from the training computation and saved as independent model files. Simultaneously, the model format of the saved training model file (such as PyTorch) is converted to an efficient inference format suitable for deployment, such as ONNX (Open Neural Network Exchange) or the TensorRT engine, to maximize inference speed. Finally, the output includes the policy network model file. The inference model of the value network model file serves as the control model for the water-air cross-medium UAV. Through policy network inference, the rotational speeds of the UAV's four motors are output, enabling the UAV to achieve autonomous control and smooth transition in air and water media.

[0183] In this embodiment of the invention, based on the Proximal Policy Optimization (PPO) algorithm, multimodal observation states, action commands, and reward functions are collected through parallel interaction between the agent and the Isaac Lab high-fidelity simulation environment. The generalized advantage estimation algorithm is used to calculate the advantage function and the target value function. A policy loss term with a pruning mechanism is employed, combined with the cross-medium optimization value function term and the policy regularization term to calculate the total loss, ensuring training stability. The termination of the internal update loop is dynamically controlled by checking whether the KL divergence converges, fully utilizing the value of the data while avoiding training divergence. Finally, automated training termination is determined by evaluating the sliding variance of the average reward from the most recent K rounds, generating a high-performance intelligent decision-making water-air cross-medium UAV control model, enabling autonomous intelligent control of the UAV in air, water, and their transition processes.

[0184] To verify the effectiveness of this invention, corresponding simulation experiments are also provided in this embodiment. A water simulation environment was constructed in the NVIDIA Isaac Lab simulation platform, and the UAV was controlled to cross the water body and stabilize at the target position in the water simulation environment. The UAV model parameters are shown in Table 1; the PPO algorithm and corresponding training parameters are shown in Table 2; the experimental environment parameters are shown in Table 3; and the initial wave difficulty and takeoff rate statistics are shown in Table 4.

[0185] Table 1. Model parameters of the UAV

[0186]

[0187] Table 2. PPO Algorithm and Training Parameter Design

[0188]

[0189] Table 3 Experimental Environment Parameter Design

[0190]

[0191] Table 4. Statistical Results of Initial Wave Difficulty and Takeoff Rate

[0192]

[0193] In the analysis of simulation experiment results, Figure 2 The movement of the drone's body was demonstrated. Figure 3This shows the trend of the average reward per epoch during training as the number of iterations (steps) changes. This can be seen from... Figure 3 It was observed that the average round reward continuously increased in the early stage of training, indicating that the agent could gradually learn a better cross-medium control strategy under the joint constraints of hierarchical reward signals (navigation, energy consumption, stability and cross-medium motion). As training progressed, the reward curve entered a plateau period and the fluctuation range gradually decreased, indicating that the policy improvement tended to stabilize and met the convergence criterion of "sliding window variance is below the threshold". Figure 4 This demonstrates the change in noise intensity (mean_std_noise) of the Gaussian random policy during the training process. In the early stages of training, the noise remains at a relatively high level, which is beneficial for full exploration in complex water-air dynamic switching scenarios. In the later stages of training, the noise gradually decreases and tends to stabilize, indicating that the policy shifts from "exploration" to "utilization", and the action output becomes more deterministic and smooth. Figure 5 The changes in the mean survival time of the UAV with training rounds are presented: as the training iterations progress, the mean survival time gradually increases and eventually remains at a high level, indicating that the agent's instability / crash events are significantly reduced during underwater propulsion, water emergencies, and aerial flight, and it can maintain a "controllable state" for a longer period of time.

[0194] The following describes the control modeling system for a water-air-transmedia UAV based on reinforcement learning provided in the embodiments of this application. The control modeling system for a water-air-transmedia UAV based on reinforcement learning described below can be referred to in correspondence with the control modeling method for a water-air-transmedia UAV based on reinforcement learning described above.

[0195] Please see Figure 6 This invention also provides a control modeling system for a water-air cross-medium unmanned aerial vehicle based on reinforcement learning. The system includes:

[0196] The simulation interaction module 201 is used to initialize the policy network and value network, and build a simulation environment; it enables the policy network to interact with the simulation environment and generate a series of trajectory data.

[0197] The experience data acquisition module 202 is used to predict the state value based on trajectory data through the value network, calculate the corresponding advantage function and target value function based on the state value, generate a series of experience data and store them in the experience playback cache pool;

[0198] The policy optimization module 203 is used to randomly sample in batches from the experience replay buffer pool. Based on the sampled experience data, the value loss term, policy loss term, and policy regularization term are calculated through the policy optimization algorithm. The total loss function is constructed by jointly constructing the value loss term, policy loss term, and policy regularization term. With the goal of minimizing the total loss function, the policy network and the value network are jointly trained and updated to obtain the updated policy network and value network.

[0199] The control model generation and output module 204 is used to determine whether the updated policy network and value network meet the preset dual convergence conditions. If they do, the updated policy network and value network are used as the control model for the water-air cross-medium UAV and output.

[0200] This invention also provides a computer device, including a memory, a processor, and a computer program stored in the memory. The processor executes the computer program to implement the steps of any of the above-described control modeling methods for a transmedium-water unmanned aerial vehicle.

[0201] This invention also provides a computer-readable storage medium storing a computer program or instructions thereon, which, when executed by a processor, implements the steps of any of the above-described control modeling methods for water-air-transmedia unmanned aerial vehicles.

[0202] This invention also provides a computer program product, including a computer program or instructions, which, when executed by a processor, implement the steps of any of the above-mentioned control modeling methods for water-air-transmedia unmanned aerial vehicles.

[0203] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0204] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection between apparatuses or units through some interfaces, and may be electrical, mechanical, or other forms.

[0205] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0206] Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0207] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0208] The above-described embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for modeling control of a water-air cross-medium unmanned aerial vehicle based on reinforcement learning, characterized in that, The method includes: Initialize the policy network and value network, and construct a simulation environment; interact the policy network with the simulation environment to generate a series of trajectory data; The value network predicts the state value based on the trajectory data, calculates the corresponding advantage function and target value function based on the state value, generates a series of experience data and stores it in the experience replay cache pool; Random batch sampling is performed in the experience replay cache pool. Based on the sampled experience data, the value loss term, policy loss term, and policy regularization term are calculated using a policy optimization algorithm. The value loss term, policy loss term, and policy regularization term are combined to construct a total loss function. With the goal of minimizing the total loss function, the policy network and value network are jointly trained and updated to obtain the updated policy network and value network. Determine whether the updated policy network and value network satisfy the preset dual convergence condition. If they do, use the updated policy network and value network as the control model for the water-air cross-medium UAV and output it.

2. The control modeling method of the water-crossing unmanned aerial vehicle according to claim 1, wherein, The step of interacting the policy network with the simulation environment to generate a series of trajectory data includes: The initial multimodal observation state of the UAV is set, and the policy network interacts with the simulation environment. For each interaction time step, the current multimodal observation state is input into the policy network to generate the corresponding action command, the reward function generated by the action command in the simulation environment is calculated, and the multimodal observation state corresponding to the next interaction time step is collected. This process continues until the interaction reaches a preset number of time steps, generating a series of trajectory data. The series of trajectory data are represented as follows: In the formula: Represents a series of trajectory data; This indicates the multimodal observation state at time step t. Issue an action command Afterwards, the environment transitions to the multimodal observation state at time step t+1. and obtain reward function ; Indicates the termination flag; This represents the cache capacity. In the formula: , , and All are reward weighting coefficients; Indicates navigation reward, Indicates energy penalty. Indicates a stability reward. This indicates a cross-media reward.

3. The method of claim 1, wherein, The steps of predicting state value based on trajectory data through a value network, calculating the corresponding advantage function and target value function based on the state value, generating a series of empirical data and storing them in an experience replay cache pool include: The value network predicts the state value at different time steps based on the trajectory data. Calculate the corresponding temporal difference error based on the state value at adjacent time steps; The generalized dominance estimation method is used to calculate the corresponding dominance function based on the time-series difference error; The target value function is calculated using the state value and advantage function at the same time step; A series of empirical data are generated based on the trajectory data, the advantage function, and the target value function, and stored in the empirical replay cache pool.

4. The control modeling method of a water-crossing unmanned aerial vehicle according to claim 3, wherein, The step of randomly sampling in the experience replay cache pool and calculating the value loss term, policy loss term, and policy regularization term based on the sampled experience data using a policy optimization algorithm includes: Randomly sample in batches from the experience replay cache pool to obtain small batches of experience data; The value loss term is calculated based on the target value function in the small batch of empirical data using a strategy optimization algorithm; the value loss term is used to guide the state value predicted by the value network to approximate the target value function. The policy ratio is calculated based on the multimodal observation states and action commands in the small batch of empirical data; the policy loss term is calculated based on the policy ratio and the corresponding advantage function through a pruning mechanism; the policy loss term is used to constrain the update magnitude of the policy network; The policy regularization term is determined based on the action probability distribution of the current policy network under multimodal observation states corresponding to small batches of empirical data; the policy regularization term is used to encourage the randomness of the policy network.

5. The control modeling method for a water-air transmedium unmanned aerial vehicle according to claim 1, characterized in that, The total loss function is calculated as follows: In the formula: represents the total loss function; represents the policy loss term of the new policy network under the parameters , represents the value loss term, represents the policy entropy regularization term; and are loss weight coefficients. in, In the formula: This indicates the use of small batches of empirical data. Find the expected value; The time step t parameter represents the time step parameter. New Strategy Network Strategy ratio; This represents the dominance function at time step t; Indicates the strategy ratio Values ​​are clipped to [ ]Inside; This represents the clipping hyperparameter; in, In the formula: This represents the average expectation at time step t; Indicates multimodal observation state State value, Indicates multimodal observation state The target value function is as follows.

6. The method of claim 1, wherein, The step of determining whether the updated policy network and value network satisfy the preset dual convergence condition, and if so, using the updated policy network and value network as the control model for the water-air cross-medium UAV and outputting it, includes: Determine whether the updated policy network and value network satisfy the internal convergence condition. If they do not satisfy the internal convergence condition, proceed to the step of randomly sampling in the experience replay cache pool, and calculating the value loss term, policy loss term, and policy regularization term based on the sampled experience data using the policy optimization algorithm. If the internal convergence condition is met, then determine whether the updated policy network and value network meet the external convergence condition. If the external convergence condition is not met, proceed to the step of interacting the policy network with the simulation environment to generate a series of trajectory data. If the external convergence condition is met, then use the updated policy network and value network as the control model for the water-air cross-medium UAV and output it.

7. A control modeling system for a water-air cross-medium unmanned aerial vehicle based on reinforcement learning, characterized in that, The system includes: The simulation interaction module is used to initialize the policy network and value network, and build a simulation environment; the policy network interacts with the simulation environment to generate a series of trajectory data; The experience data acquisition module is used to predict state value based on the trajectory data through the value network, calculate the corresponding advantage function and target value function based on the state value, generate a series of experience data and store them in the experience playback cache pool; The policy optimization module is used to randomly sample in batches from the experience replay cache pool, and calculate the value loss term, policy loss term, and policy regularization term based on the sampled experience data through the policy optimization algorithm; construct a total loss function by combining the value loss term, the policy loss term, and the policy regularization term, and perform joint training and updating of the policy network and the value network with the goal of minimizing the total loss function, so as to obtain the updated policy network and value network. The control model generation and output module is used to determine whether the updated policy network and value network meet the preset dual convergence conditions. If they do, the updated policy network and value network are used as the control model for the water-air cross-medium UAV and output.

8. A computer device, comprising a memory, a processor, and a computer program stored in the memory, characterized in that, The processor executes the computer program to implement the steps of the control modeling method for a transmedium-water unmanned aerial vehicle as described in any one of claims 1-6.

9. A computer readable storage medium having stored thereon a computer program or instructions, characterized in that, When the computer program or instructions are executed by the processor, they implement the steps of the control modeling method for a transmedium-water unmanned aerial vehicle as described in any one of claims 1-6.

10. A computer program product comprising computer programs or instructions, characterized in that, When the computer program or instructions are executed by the processor, they implement the steps of the control modeling method for a transmedium-water unmanned aerial vehicle as described in any one of claims 1-6.

Images