A method and framework for policy training optimization combining off-line and on-line reinforcement learning

Abstract

The application discloses a strategy training optimization method and framework combining off-line and on-line reinforcement learning, and belongs to the technical field of intelligent control and reinforcement learning. In the method, an improved off-line reinforcement learning algorithm is used to perform strategy training through off-line data, learn an initial strategy, and perform evaluation in an on-line environment; when the success rate of the strategy is greater than or equal to a conversion threshold, the method is switched to on-line reinforcement learning training; then, an on-line reinforcement learning algorithm is used to perform real-time optimization on the initial strategy, so that the strategy is continuously adjusted in the execution process. The application combines the efficient data utilization capability of off-line reinforcement learning and the real-time optimization capability of on-line reinforcement learning, so that the strategy can be dynamically adjusted in the task execution process, can adapt to changes in different environmental conditions, improves the applicability and robustness of the strategy in complex tasks, can learn a better decision strategy in a more challenging environment, and thus helps to ensure the reliability of task execution.

Description

Technical Field

[0001] This invention relates to the fields of intelligent control and reinforcement learning technology, and more specifically to a policy training optimization method and framework that combines online and offline reinforcement learning. Background Technology

[0002] In the field of reinforcement learning (RL), online reinforcement learning and offline reinforcement learning are two main training paradigms. Online reinforcement learning dynamically collects data and optimizes policies through real-time interaction between the agent and the environment, enabling it to adapt to environmental changes and making it suitable for scenarios where task objectives are constantly changing or environmental characteristics are unknown. However, existing online reinforcement learning methods have the following problems: First, their sample utilization efficiency is low; in high-dimensional state spaces, a large amount of interaction data is often required to learn an effective policy, resulting in high training costs. Second, due to the randomness of online exploration, the agent may experience suboptimal or even incorrect policies, leading to performance degradation during training and even irreversible damage in practical applications. Furthermore, in high-risk, high-cost tasks (such as drone docking, robotics, autonomous driving, and medical decision-making), the unpredictability of online exploration may lead to safety hazards, making this method difficult to directly apply to real-world environments.

[0003] In contrast, offline reinforcement learning relies on pre-collected historical datasets for training, avoiding the instability of online exploration and enabling safe and efficient policy optimization in high-risk scenarios. However, existing offline reinforcement learning methods still face the following technical bottlenecks: First, because offline training cannot involve active exploration, the agent's policy optimization is severely limited by the quality of the dataset. If the dataset lacks sufficient exploration samples, the learned policy may not cover all potential situations, resulting in limited generalization ability. Second, offline reinforcement learning generally faces the distribution shift problem, where the distribution of policy behavior does not match the distribution of training data, potentially reducing the model's applicability in real-world environments and thus affecting the reliability of task execution.

[0004] In summary, existing reinforcement learning methods have certain limitations in different application scenarios. In high-risk, high-cost tasks, the security issues of online reinforcement learning limit its application, while the insufficient generalization ability and distribution shift problem of offline reinforcement learning affect the applicability of the policies.

[0005] Therefore, how to enable strategies to be dynamically adjusted during task execution to adapt to changes in different environmental conditions, achieve more stable and efficient optimization results, improve their applicability and robustness in complex tasks, and ensure the reliability of task execution are technical problems that need to be solved in this field. Summary of the Invention

[0006] In view of this, the present invention provides a policy training optimization method and framework that combines offline and online reinforcement learning. It integrates the data efficiency of offline reinforcement learning with the real-time optimization capability of online reinforcement learning, enabling the policy to be dynamically adjusted during task execution. This allows the policy to adapt to changes in different environmental conditions, improves the applicability and robustness of the policy in complex tasks, and enables it to learn better decision-making strategies in more challenging environments, thereby helping to ensure the reliability of task execution.

[0007] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0008] In a first aspect, the present invention provides a policy training optimization method combining offline and online reinforcement learning, the method comprising the following steps:

[0009] Step 1: Use offline reinforcement learning algorithms to train a policy on offline data, learn an initial policy, and evaluate it in an online environment. When the success rate of the policy is greater than or equal to the transition threshold, switch to online reinforcement learning training.

[0010] Step 2: Use an online reinforcement learning algorithm to optimize the initial policy in real time, so that the policy is continuously adjusted during execution.

[0011] Furthermore, in step 1, the offline reinforcement learning algorithm is an improved IQL algorithm; its improvement includes:

[0012] Reduce estimation bias through multi-Q network integration;

[0013] Importance sampling is used to adjust data weights and correct data distribution bias.

[0014] The learning rate is adjusted by using a cosine learning rate decay.

[0015] Furthermore, estimation bias is reduced through multi-Q network integration, specifically as follows:

[0016] Multiple independent Q-networks are defined to reduce the uncertainty of a single Q-value estimate. The parameters of each network are initialized independently, as follows:

[0017]

[0018] In the formula, Represents an independent Q-parameter network;

[0019] Then, by calculating the minimum or average of multiple Q values, the overestimation bias is reduced; where:

[0020] The formula for calculating the minimum value is:

[0021]

[0022] The formula for calculating the mean is:

[0023]

[0024] in, Let represent the Q-value objective, γ be the discount factor, r be the immediate reward, s and a represent the state and action respectively, s′ and a′ represent the next state and the next action respectively; N represents the number of Q-networks.

[0025] Furthermore, the formula for calculating the importance sampling weight is as follows:

[0026]

[0027] Where w(a|s) represents the importance sampling weight, π target (a|s) represents the target policy for the current optimization, π behavior (a|s) represents the behavioral strategy used during the data collection process.

[0028] Furthermore, the formula for calculating the cosine learning rate decay is as follows:

[0029]

[0030] Where, η t η represents the learning rate at the current step size t. max and η min These are the maximum and minimum values ​​of the learning rate, respectively, and T is the total training step size.

[0031] Furthermore, in step 2, the online reinforcement learning algorithm is the PPO algorithm.

[0032] Secondly, the present invention also provides a policy training optimization framework combining offline reinforcement learning, which applies the above-mentioned policy training optimization method combining offline reinforcement learning to perform policy training optimization. This framework includes:

[0033] The offline training module is used to train a policy using offline reinforcement learning algorithms on offline data, learn an initial policy, and evaluate it in an online environment. When the success rate of the policy is greater than or equal to the transition threshold, it switches to online reinforcement learning training. The offline reinforcement learning algorithm is an improved IQL algorithm.

[0034] The online training module is used to optimize the initial policy in real time using a linear reinforcement learning algorithm, so that the policy is continuously adjusted during execution.

[0035] Thirdly, embodiments of the present invention also provide an electronic device, including a processor and a memory, wherein the memory stores machine-executable instructions that can be executed by the processor, and the processor executes the machine-executable instructions to implement the above-described policy training optimization method combining offline and online reinforcement learning.

[0036] As can be seen from the above technical solution, the present invention provides a method and framework for policy training optimization that combines online and offline reinforcement learning. Compared with the prior art, the present invention has at least the following beneficial effects:

[0037] 1. This invention integrates the data-efficient utilization capability of offline reinforcement learning with the real-time optimization capability of online reinforcement learning, enabling the strategy to be dynamically adjusted during task execution. It can adapt to changes in different environmental conditions, improve the applicability and robustness of the strategy in complex tasks, and learn better decision-making strategies in more challenging environments, thus helping to ensure the reliability of task execution.

[0038] 2. This invention has made several optimizations and improvements to IQL, including integrating multiple Q networks to alleviate the overestimation problem, using importance sampling to improve data utilization efficiency, and using cosine learning rate decay to enhance the stability of policy training; it has improved the training efficiency and quality of the policy in complex tasks, not only accelerating the convergence speed of offline RL training, but also improving the performance of the policy in high-dimensional complex tasks (such as UAV docking tasks).

[0039] 3. This invention helps to promote the application and development of reinforcement learning in complex environments.

[0040] Other features and advantages of the invention will be set forth in the description which follows, and will be apparent in part from the description, or may be learned by practicing the invention. The objects and other advantages of the invention may be realized and obtained by means of the structures particularly pointed out in the written description and the accompanying drawings.

[0041] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description

[0042] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0043] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used together with the embodiments of the invention to explain the invention and do not constitute a limitation thereof.

[0044] Figure 1 This is a schematic diagram of a policy training optimization method combining offline and online reinforcement learning, provided by an embodiment of the present invention.

[0045] Figure 2 This is a schematic diagram of a policy training optimization framework combining offline and online reinforcement learning, provided as an embodiment of the present invention.

[0046] Figure 3 This is a schematic diagram of the electronic device structure provided in an embodiment of the present invention. Detailed Implementation

[0047] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are some embodiments of the present invention, but not all embodiments.

[0048] Therefore, the following detailed description of the embodiments of the invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely to illustrate selected embodiments of the invention. All other embodiments obtained by those skilled in the art based on the embodiments of the invention without inventive effort are within the scope of protection of the invention.

[0049] See Figure 1 and Figure 2 As shown, this embodiment of the invention provides a policy training optimization method combining offline and online reinforcement learning. The method mainly includes the following steps:

[0050] Step 1: Use an offline reinforcement learning algorithm to train a policy on offline data, learn an initial policy, and evaluate it in an online environment. When the success rate of the policy is greater than or equal to the transition threshold, switch to online reinforcement learning training. The offline reinforcement learning algorithm is an improved IQL algorithm.

[0051] Step 2: Use an online reinforcement learning algorithm to optimize the initial policy in real time, so that the policy is continuously adjusted during execution.

[0052] This invention employs an offline-online fusion approach to better leverage the advantages of reinforcement learning. Especially in the expensive processes of drone docking and robot intelligent control training, this method not only improves training efficiency but also enhances the overall performance of decision-making strategies.

[0053] I. The principles and specific implementation methods involved in this invention will be described in detail below:

[0054] Implicit Q-Learning (IQL) is a highly efficient offline reinforcement learning algorithm that has rapidly become a significant research advancement in the field since its inception. Its core innovation lies in the introduction of implicit policy optimization, which effectively mitigates the instability caused by out-of-distribution (OOD) actions during offline RL training by decoupling the behavioral policy from the objective policy.

[0055] Traditional offline reinforcement learning (RL) methods typically rely on explicit behavior constraints, such as behavior cloning (BC) loss or policy constraints, to ensure that the learned policy does not deviate from the training data distribution. However, these methods often introduce additional optimization difficulty and may cause the policy to converge to a suboptimal solution. IQL, through quantile regression Q-learning and advantage-weighted policy extraction, can effectively learn the optimal policy without explicit behavior constraints, thus significantly simplifying the algorithm structure while improving the stability and generalization ability of the policy.

[0056] Building upon this foundation, this invention incorporates several optimizations and improvements to IQL to further enhance its training efficiency and policy quality in complex tasks. Specifically, this invention introduces the following key optimization methods:

[0057] Multiple Q-Networks: Employing multiple Q-networks (similar to Ensemble Q-Learning) reduces the uncertainty of a single Q-value estimate and improves the stability of the value function. By calculating the mean or minimum of multiple Q-values, overestimation bias is effectively reduced, thereby improving policy performance.

[0058] Importance Sampling (IS): During training, importance sampling is used to adjust the weights of different data samples to enhance the policy's learning ability on out-of-distribution (OOD) samples. This method can effectively alleviate the distribution bias problem of the dataset, thereby improving the generalization ability of the policy.

[0059] Cosine Learning Rate Decay: This invention employs cosine annealing learning rate decay, which allows for rapid exploration using a larger learning rate in the early stages of training, while gradually reducing the learning rate in the later stages of training to improve convergence stability and avoid policy oscillations.

[0060] In this embodiment of the invention, this optimized IQL variant is referred to as MiQL (Modified IQL). This improved method not only accelerates the convergence speed of offline RL training but also enhances the performance of the policy in high-dimensional complex tasks, enabling it to learn better decision-making policies in more challenging environments. Specifically:

[0061] ① Multi-Q network integration and mitigation of overestimation bias:

[0062] In reinforcement learning, overestimation bias primarily stems from random noise, model errors, and the influence of suboptimal actions, leading to a systematic overestimation of the value function Q(s,a). This bias is particularly pronounced in Q-learning algorithms, especially those relying on a single Q-network for updates, where the policy optimization direction is easily deviated from the true optimal solution due to the estimation bias of the target value.

[0063] To alleviate this problem, this invention optimizes the loss function calculation method of Q-networks in MiQL by introducing multiple Q-networks for integration, thereby reducing excessive estimation bias. The core idea is to utilize the prediction results of multiple Q-networks to reduce the noise impact of a single model, thus limiting the upper bound of the target value and making the Q-value estimation more conservative. In this method, multiple independent Q-networks are first defined, and the parameters of each network are initialized independently, specifically as follows:

[0064]

[0065] These Q-networks capture different value estimation patterns during varying initialization and training processes, thus providing more diverse data representations. This diversity helps improve the stability and generalization ability of the Q-value. The ultimate Q-value target... Various calculation methods can be used, such as the minimum value method and the weighted average method.

[0066] In this embodiment of the invention, the minimum value method follows the principle of "more conservative value estimation," that is, selecting the minimum value as the target value from the prediction results of multiple Q networks to suppress overestimation and thus improve the robustness of the strategy.

[0067]

[0068] Where γ is the discount factor, r is the immediate reward, and s′ and a′ represent the next state and the next action, respectively. This method can effectively suppress the overestimation of Q-values, but it may also introduce a certain degree of conservatism, making policy updates more cautious and thus affecting exploration capabilities.

[0069] Another approach is to take a weighted average of the outputs of multiple Q-networks to balance the conservatism of the estimation with the flexibility of the learning:

[0070]

[0071] This method alleviates the overestimation bias while making policy updates smoother and avoids overly conservative policies due to underestimation of a single Q-network, thus achieving a better balance between robustness and exploration.

[0072] In this embodiment of the invention, during the policy update process, policy π is optimized based on the integrated value of multiple Q networks, and the update method is as follows:

[0073]

[0074] Here, β is a temperature coefficient used to adjust the policy's sensitivity to different action values. A smaller β makes the policy more inclined to choose actions with high Q values, thus promoting more deterministic decision-making, while a larger β increases exploration, allowing the policy to try more different actions to improve generalization and adaptability.

[0075] Multi-Q network ensemble methods can effectively mitigate overestimation bias. By fusing the predictions of multiple Q networks, the overestimation problem that may be caused by a single network can be reduced, thereby improving the stability of the value function. Secondly, this method helps to reduce the impact of model errors. In complex environments, a single Q network may be affected by data noise or model errors, while the ensemble model effectively reduces the impact of individual errors on the estimation by taking the mean or minimum value.

[0076] Furthermore, the multi-Q network method enhances the generalization ability of the policy, capturing data distribution more comprehensively and making the policy more adaptable to different task environments, especially suitable for offline reinforcement learning tasks with noisy or suboptimal data. Finally, this method improves training stability. Due to the more robust target value calculation, the impact of error propagation during policy updates is significantly reduced, making the training process smoother and reducing drastic oscillations.

[0077] This invention effectively alleviates the problem of excessive estimation bias in offline RL training by introducing multi-Q network ensemble. Compared with traditional single-Q network methods, multi-Q networks provide a more robust and conservative value estimation method, making policy optimization more stable and generalizable. Furthermore, in complex environments, this method can significantly improve sample utilization efficiency, especially in tasks containing noisy or suboptimal data.

[0078] ②Importance sampling:

[0079] In the policy and value function update process of the Miql algorithm, this invention uses the log probabilities of the behavioral policy and the target policy to calculate the importance sampling weights to correct for data distribution bias. The formula for calculating the importance sampling weights is as follows:

[0080]

[0081] Where w(a|s) represents the importance sampling weight, π target (a|s) represents the target policy for the current optimization, π behavior (a|s) represents the behavioral strategy used during the data collection process.

[0082] In offline reinforcement learning scenarios, data is typically collected from multiple behavioral policies, so its distribution may deviate significantly from the current target policy being optimized, i.e., a distribution mismatch. This deviation introduces policy bias, reduces data utilization efficiency, and may even cause the policy to get stuck in a suboptimal solution or encounter out-of-distribution problems.

[0083] The core idea of ​​importance sampling is to adjust the weights of the data when updating the value function and policy, thereby more accurately estimating the expected value of the target policy. Traditional reinforcement learning's experience replay generally assumes that the data are independent and identically distributed. However, offline RL is constrained by the data distribution being limited by the behavioral policy, thus hindering the learning of the target policy. By introducing importance sampling, this invention can reweight the experience data to better align with the distribution of the target policy, thereby improving the effectiveness of policy optimization.

[0084] ③ Cosine annealing learning rate decay:

[0085] During MiQL training, this invention employs a cosine annealing strategy to adjust the learning rate, achieving a good balance between exploration and stability. Specifically, a higher learning rate is used initially so that the strategy can quickly adapt to the distribution of behavioral data; as training progresses, the learning rate gradually decreases, effectively reducing the risk of overfitting and improving the model's generalization ability.

[0086] The mathematical expression for the learning rate decay in cosine annealing is as follows:

[0087]

[0088] Where, η t η represents the learning rate at the current step size t. max and η min These are the maximum and minimum values ​​of the learning rate, respectively, and T is the total training step size.

[0089] This smooth decay process allows the model to adapt to offline data more quickly in the early stages of training, improving the sampling efficiency of the policy. Simultaneously, it effectively suppresses gradient explosion / vanishing gradient problems in the later stages of training, thereby improving the stability of the algorithm. Furthermore, this gradual reduction of the learning rate ensures that the model can perform finer-grained optimization in later stages, promoting the gradual transition of the policy from suboptimal solutions to better solutions and reducing the risk of getting trapped in local optima.

[0090] See Figure 2 As shown, to further enhance the practical application capability of the strategy, this invention provides a strategy optimization framework by combining an offline / online reinforcement learning strategy training and optimization method described in the above embodiments. This framework integrates the data-efficient utilization capability of offline reinforcement learning with the real-time optimization capability of online reinforcement learning, enabling the strategy to dynamically adjust during task execution to adapt to changes in different environmental conditions.

[0091] In this framework, policy training is mainly divided into two key stages:

[0092] Offline Pretraining: MiQL is trained using historical data to learn an initial policy that can better adapt to the state-action distribution covered by the training data, providing a good foundation for subsequent online optimization.

[0093] Online Fine-tuning: During actual task execution, we use online reinforcement learning (such as PPO) to fine-tune the strategy and introduce an adaptive adjustment strategy to further improve the generalization ability and real-time response capability of the strategy in different task scenarios.

[0094] This framework not only fully utilizes offline data for efficient policy training but also incorporates online learning methods, enabling the policy to continuously adjust during execution for more stable and efficient optimization. In other words, this invention combines MiQL with this framework to propose a new scheme, named AO2D-MiQL (Adaptive Offline-to-Online Decision Optimization with MiQL), to further promote the application and development of reinforcement learning in complex environments.

[0095] like Figure 2 As shown, the environment section represents a drone docking task. Miql combines IQL and multi-Q network ensembles, importance sampling, and a cosine annealing learning rate decay component. Overall, AO2D-Miql is an adaptive decision optimization framework that incorporates both online and offline reinforcement learning.

[0096] First, the Miql offline reinforcement learning algorithm is used to learn from offline buffer data obtained through expert policies. Iterative testing is performed continuously during training. When the success rate (SR) reaches a preset threshold α, online reinforcement learning is initiated to further optimize the policy. Data is acquired through interaction with the environment and placed into the online buffer. Online learning iterates by fusing online and offline data into a fusion buffer database. During this process, the success rate is continuously tested. If the success rate is below the threshold, offline learning is performed; if it is greater than or equal to the threshold, online training is initiated. This process is repeated iteratively until the task reaches a fixed number of rounds or exceeds a certain threshold, at which point the task ends.

[0097] This invention dynamically adjusts the training method, enabling the strategy to fully utilize offline data for pre-training in the early stages, and then transition to online training once a certain performance level is achieved, further optimizing the strategy's performance. This avoids the generalization problem that pure offline RL may face, while also mitigating the high costs and security risks inherent in pure online RL during the exploration process.

[0098] The policy training optimization framework combining offline and online reinforcement learning provided in this embodiment of the invention has the same implementation principle and technical effect as the aforementioned method embodiment. For the sake of brevity, any parts not mentioned in this embodiment can be referred to the corresponding content in the aforementioned method embodiment, and will not be repeated here.

[0099] II. Experimental Verification:

[0100] Comprehensive experiments were conducted in various aerial docking scenarios, covering different initial docking configurations; the specific test scenario was an aerial docking mission involving unmanned aerial vehicles.

[0101] Based on this task, various noise levels and initial pose scenarios for two different UAVs were set up, and different reinforcement learning algorithms were used for training and testing. The initial environment included various forms, as shown in Table 1 below:

[0102] Table 1 Initial Environment

[0103]

[0104] The “Environment” column in Table 1 represents the naming conventions for different docking environment initialization methods, including six initial attitude and velocity settings. In all environments, the UAV B (UAVB) is positioned 0.15 meters from the target docking point along the docking direction (y-axis). Initial velocities and angular velocities are randomly sampled within the ranges of (-0.1, 0.1) m / s and (-0.2, 0.2) radians / s. If the “Velocity” column is set to “T”, randomness is introduced into the velocity and angular velocity; if set to “F”, no randomness is added, and both velocity and angular velocity are initialized to 0. The “xz-axis” column specifies the displacement range in the x-axis and z-axis directions relative to the target docking position. Column Angle A rp and A y These represent the initial roll, pitch, and yaw angles of UAV A, respectively. If the values ​​in these columns are set to "F", it indicates that no randomness is introduced, and the angles are initialized to 0. Similarly, angle B... rp and B y Following the same logic, the initial angle of drone B is also indicated.

[0105] First, two datasets were generated using an expert strategy, collected under different docking environments and state noise settings. Dataset Data1 was generated in the docking-v2 environment by combining four different noise settings; while Dataset Data2 was generated in the docking-v1 environment by combining the default noise setting and the noise-free setting. See Table 2 below:

[0106] Table 2 Different expert data tests

[0107]

[0108] The first column of Table 2 represents the index, the second column lists the offline reinforcement learning algorithms, and the remaining columns report the success rate of each algorithm in 100 tests across different docking environments. The results show that Align-iql, IQL, and Miql outperform Data1 on Data2.

[0109] Furthermore, after introducing the AO2D-Miql framework on top of Miql, the overall model was trained, and the results are shown in Table 3 below:

[0110] Table 3 Comparison results of Miql and AO2D-Miql

[0111]

[0112] As shown in Table 3, A2D-Miql improves performance in different environments.

[0113] As described in the above embodiments, those skilled in the art will understand that this invention addresses the challenges of UAV aerial docking missions by proposing a policy training optimization method combining online and offline reinforcement learning. It improves the performance of the benchmark algorithm—Implicit Q-Learning (IQL)—by introducing several improvement techniques, including integrating multiple Q-networks to alleviate overestimation problems, employing importance sampling to improve data utilization efficiency, and utilizing cosine learning rate decay to enhance the stability of policy training. Furthermore, an adaptive offline-online decision optimization framework is constructed to improve the model's success rate during online testing. This framework can dynamically and seamlessly switch between offline and online training phases, fully utilizing the efficiency of offline learning while ensuring the model's adaptability to the online environment.

[0114] This invention underwent comprehensive experiments in various aerial docking scenarios, covering different initial docking configurations. Experimental results show that the proposed algorithm outperforms existing offline reinforcement learning baseline methods under all test conditions, significantly improving the success rate. In particular, the success rate is significantly improved compared to the purely offline MiQL algorithm. This result demonstrates that the proposed method can effectively bridge the gap between offline reinforcement learning and online environment adaptability. Furthermore, the method has been successfully applied to real-world scenarios, further validating its feasibility and superiority in actual UAV aerial docking tasks.

[0115] Furthermore, refer to Figure 3 As shown, this embodiment of the invention also provides an electronic device that can execute the above-described methods and systems. The electronic device may include a processor 10, a memory 11, a communication bus 12, and a communication interface 13, and may also include a computer program stored in the memory 11 and executable on the processor 10.

[0116] In some embodiments, the processor 10 may be composed of integrated circuits, such as a single packaged integrated circuit or multiple integrated circuits packaged with the same or different functions, including combinations of one or more central processing units (CPUs), microprocessors, digital processing chips, graphics processors, and various control chips. The processor 10 is the control unit of the electronic device, connecting various components of the entire electronic device through various interfaces and lines. It executes programs or modules stored in the memory 11 and calls data stored in the memory 11 to perform various functions of the electronic device and process data.

[0117] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, devices, or computer program products, etc. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0118] It should be noted that the word "comprising" does not exclude the presence of components or steps not listed in the claims. The words "a" or "an" preceding a component do not exclude the presence of a plurality of such components. This invention can be implemented by means of hardware comprising several different components and by means of a suitably programmed computer.

[0119] In the description of this invention, it should be noted that some processes described in this application specification and drawings include multiple operations that appear in a specific order. However, it should be clearly understood that these operations may be performed in any order or in parallel. Furthermore, various numbers are used for descriptive purposes only and should not be construed as indicating or implying relative importance.

[0120] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.

[0121] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present invention. The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for training and optimizing UAV docking strategies by combining online and offline reinforcement learning, characterized in that, The optimized UAV docking strategy trained based on this method is used to achieve UAV aerial docking tasks, including the following steps: Step 1: Acquire offline data of the UAV, which includes data on the speed and angle of the UAV relative to the target docking position. Use offline reinforcement learning algorithm to train the policy on the offline data, learn an initial policy, and evaluate it in an online environment. When the success rate of the policy is greater than or equal to the conversion threshold, switch to online reinforcement learning training. Step 2: Use an online reinforcement learning algorithm to optimize the initial policy in real time, so that the policy is continuously adjusted during execution; In step 1, the offline reinforcement learning algorithm is an improved implicit Q-learning algorithm; its improvement methods include: Estimation bias is reduced through multi-Q network integration; specifically: Multiple independent Q-networks are defined to reduce the uncertainty of a single Q-value estimate. The parameters of each network are initialized independently, as follows: In the formula, Represents an independent Q-parameter network; Then, by calculating the minimum or average of multiple Q values, the overestimation bias is reduced; where: The formula for calculating the minimum value is: The formula for calculating the mean is: in, This represents the Q-value objective, where γ is the discount factor. r For instant rewards, s This indicates the state, specifically the state comprised of offline data from the drone. a This refers to the action output based on the drone docking strategy. s′ and a′ These represent the next state and the next action, respectively. N Indicates the number of Q-networks; Importance sampling is used to adjust data weights and correct data distribution bias. The learning rate is adjusted by using a cosine learning rate decay.

2. The method for training and optimizing UAV docking strategies by combining online and offline reinforcement learning according to claim 1, characterized in that, The formula for calculating the importance sampling weight is: in, Indicates the importance sampling weight, This represents the target strategy for current optimization. This indicates the behavioral strategies used during the data collection process.

3. The method for training and optimizing UAV docking strategies by combining online and offline reinforcement learning according to claim 1, characterized in that, The formula for calculating the cosine learning rate decay is: in, η t This represents the learning rate at the current step size t. η max and η min These are the maximum and minimum values ​​of the learning rate, respectively. T This is the total training step length.

4. The method for training and optimizing UAV docking strategies by combining online and offline reinforcement learning according to claim 1, characterized in that, In step 2, the online reinforcement learning algorithm is the PPO algorithm.

5. An electronic device, characterized in that, It includes a processor and a memory, the memory storing machine-executable instructions that can be executed by the processor, the processor executing the machine-executable instructions to implement a training optimization method for UAV docking strategy combining offline and online reinforcement learning as described in any one of claims 1 to 4.

Images