Strategy training optimization method and framework combining off-line and on-line reinforcement learning

Abstract

The invention discloses a strategy training optimization method and framework combining off-line and on-line reinforcement learning, and belongs to the technical field of intelligent control and reinforcement learning. According to the method, an improved offline reinforcement learning algorithm is utilized to carry out strategy training through offline data, an initial strategy is learned, evaluation is carried out in an online environment, and when the success rate of the strategy is larger than or equal to a conversion threshold value, online reinforcement learning training is switched to; and then, an online reinforcement learning algorithm is adopted to optimize the initial strategy in real time, so that the strategy is continuously adjusted in the execution process. According to the method, the efficient data utilization capability of offline reinforcement learning and the real-time optimization capability of online reinforcement learning are fused, so that the strategy can be dynamically adjusted in the task execution process, the method can adapt to changes of different environmental conditions, the applicability and robustness of the strategy in complex tasks are improved, and the method is suitable for being applied to the task execution process. And a better decision strategy can be learned in a more challenging environment, so that the reliability of task execution can be ensured.

Description

Technical Field

[0001] The present invention relates to the technical fields of intelligent control and reinforcement learning, and more specifically, to a method and framework for optimizing policy training by combining off-line and on-line reinforcement learning. Background Art

[0002] In the field of reinforcement learning (RL), on-line reinforcement learning and off-line reinforcement learning are two main training paradigms. On-line reinforcement learning dynamically collects data and optimizes policies through the real-time interaction between an agent and the environment, enabling it to adapt to environmental changes. It is suitable for scenarios where task goals change continuously or the environmental characteristics are unknown. However, existing on-line reinforcement learning methods have the following problems: First, their sample utilization efficiency is low. In a high-dimensional state space, a large amount of interaction data is often required to learn an effective policy, resulting in high training costs. Second, due to the randomness of on-line exploration, the agent may experience sub-optimal or even incorrect policies, leading to a decline in performance during training and even irreversible damage in practical applications. In addition, in high-risk and high-cost tasks (such as drone docking, robotics, autonomous driving, medical decision-making, etc.), the unpredictability of on-line exploration may pose safety hazards, making it difficult to directly apply this method to the actual environment.

[0003] In contrast, off-line reinforcement learning relies on a pre-collected historical data set for training, avoiding the instability brought by on-line exploration and enabling safe and efficient policy optimization in high-risk scenarios. However, existing off-line reinforcement learning methods still have the following technical bottlenecks: On the one hand, since off-line training cannot perform active exploration, the policy optimization of the agent is strictly limited by the quality of the data set. If the data set lacks sufficient exploration samples, the learned policy may not cover all potential situations, resulting in limited generalization ability. On the other hand, off-line reinforcement learning generally faces the problem of distributional shift, that is, the behavior distribution of the policy does not match the distribution of the training data, which may lead to a decrease in the applicability of the model in the real environment, thus affecting the reliability of task execution.

[0004] In summary, existing reinforcement learning methods have certain limitations in different application scenarios. In high-risk and high-cost tasks, the safety issues of on-line reinforcement learning limit its application, while the insufficient generalization ability and distributional shift problems of off-line reinforcement learning affect the applicability of the policy.

[0005] Therefore, how to enable the policy to dynamically adjust during task execution to adapt to changes in different environmental conditions, achieve more stable and efficient optimization effects, improve its applicability and robustness in complex tasks, and ensure the reliability of task execution is a technical problem that needs to be solved in this field. Summary of the Invention

[0006] In view of this, the present invention provides a strategy training optimization method and framework combining offline and online reinforcement learning, which integrates the data-efficient utilization ability of offline reinforcement learning and the real-time optimization ability of online reinforcement learning, enabling the strategy to be dynamically adjusted during task execution, adapting to changes in different environmental conditions, enhancing the applicability and robustness of the strategy in complex tasks, being able to learn better decision-making strategies in more challenging environments, and thus contributing to ensuring the reliability of task execution.

[0007] To achieve the above object, the technical solution adopted by the present invention is as follows:

[0008] In a first aspect, the present invention provides a strategy training optimization method combining offline and online reinforcement learning, and the method includes the following steps:

[0009] Step 1: Use an offline reinforcement learning algorithm to train a strategy through offline data, learn an initial strategy, and evaluate it in an online environment. When the success rate of the strategy is greater than or equal to the conversion threshold, switch to online reinforcement learning training;

[0010] Step 2: Use an online reinforcement learning algorithm to perform real-time optimization on the initial strategy, so that the strategy is continuously adjusted during execution.

[0011] Further, in Step 1, the offline reinforcement learning algorithm is: an improved IQL algorithm; its improvement methods include:

[0012] Reduce the estimation bias through multi-Q network integration;

[0013] Adjust the data weights using importance sampling to correct the data distribution bias;

[0014] Adopt cosine learning rate decay to adjust the learning rate.

[0015] Further, reducing the estimation bias through multi-Q network integration is specifically:

[0016] Define multiple independent Q networks, use multiple Q networks to reduce the uncertainty of single Q value estimation, and the parameters of each network are independently initialized, specifically expressed as:

[0017]

[0018] In the formula, represents an independent Q parameter network;

[0019] Then, by calculating the minimum value or mean value of multiple Q values, reduce the overestimation bias; among them:

[0020] The calculation formula for the minimum value is:

[0021]

[0022] The mean calculation formula is:

[0023]

[0024] Wherein, represents the Q-value target, γ is the discount factor, r is the immediate reward, s and a represent the state and action respectively, s′ and a′ represent the next state and the next action respectively; N represents the number of Q networks.

[0025] Furthermore, the calculation formula of the importance sampling weight is:

[0026]

[0027] Wherein, w(a|s) represents the importance sampling weight, and π target (a|s) represents the currently optimized target policy, and π behavior (a|s) represents the behavior policy used during the data collection process.

[0028] Furthermore, the calculation formula of the cosine learning rate decay is:

[0029]

[0030] Wherein, η t represents the learning rate at the current step t, and η max and η min are the maximum and minimum values of the learning rate respectively, and T is the total number of training steps.

[0031] Furthermore, in the step 2, the online reinforcement learning algorithm is the PPO algorithm.

[0032] In a second aspect, the present invention further provides a policy training optimization framework combining off-line and on-line reinforcement learning. By applying the above-mentioned policy training optimization method combining off-line and on-line reinforcement learning, policy training optimization is performed. The framework includes:

[0033] An off-line training module, which is used to perform policy training through off-line data by using an off-line reinforcement learning algorithm, learn an initial policy, and evaluate it in an on-line environment. When the success rate of the policy is greater than or equal to the conversion threshold, it switches to on-line reinforcement learning training; wherein, the off-line reinforcement learning algorithm is: an improved IQL algorithm;

[0034] An on-line training module, which is used to perform real-time optimization of the initial policy by using an on-line reinforcement learning algorithm, so that the policy is continuously adjusted during the execution process.

[0035] In a third aspect, an embodiment of the present invention further provides an electronic device, including a processor and a memory. The memory stores machine-executable instructions that can be executed by the processor, and the processor executes the machine-executable instructions to implement a policy training optimization method that can execute the above-mentioned combination of offline and online reinforcement learning.

[0036] As can be seen from the above technical solutions, the present invention provides a policy training optimization method and framework that combines offline and online reinforcement learning. Compared with the prior art, the present invention has at least the following beneficial effects:

[0037] 1. The present invention integrates the data-efficient utilization ability of offline reinforcement learning and the real-time optimization ability of online reinforcement learning, enabling the policy to be dynamically adjusted during task execution, adapting to changes in different environmental conditions, enhancing the applicability and robustness of the policy in complex tasks, being able to learn better decision-making strategies in more challenging environments, and helping to ensure the reliability of task execution.

[0038] 2. The present invention has made multiple optimizations and improvements to IQL, including integrating multiple Q networks to alleviate the overestimation problem, using importance sampling to improve data utilization efficiency, and using cosine learning rate decay to enhance the stability of policy training; enhancing the training efficiency and quality of the policy in complex tasks, not only accelerating the convergence speed of offline RL training, but also improving the performance of the policy in high-dimensional complex tasks (such as drone docking tasks).

[0039] 3. The present invention helps to promote the application and development of reinforcement learning in complex environments.

[0040] Other features and advantages of the present invention will be described in the following specification, and, in part, will be obvious from the specification, or will be understood by implementing the present invention. The objectives and other advantages of the present invention can be achieved and obtained by the structures specifically pointed out in the written specification and the drawings.

[0041] The technical solutions of the present invention will be further described in detail below through the drawings and embodiments. Description of the Drawings

[0042] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings required for the description of the embodiments or the prior art. Obviously, the following drawings are some embodiments of the present application. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0043] The accompanying drawings are used to provide further understanding of the present invention and constitute a part of the specification. They are used to explain the present invention together with the embodiments of the present invention and do not constitute a limitation of the present invention.

[0044] Figure 1 A schematic diagram of a strategy training optimization method combining offline and online reinforcement learning provided by an embodiment of the present invention.

[0045] Figure 2 A schematic diagram of a strategy training optimization framework combining offline and online reinforcement learning provided by an embodiment of the present invention.

[0046] Figure 3 A schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0047] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments.

[0048] Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the invention as claimed, but rather merely represents selected embodiments of the present invention. All other embodiments derived by persons of ordinary skill in the art based on the embodiments of the present invention without creative effort shall fall within the scope of protection of the present invention.

[0049] See also Figure 1 and Figure 2 As shown, an embodiment of the present invention provides a strategy training optimization method combining offline and online reinforcement learning, which mainly includes the following steps:

[0050] Step 1: Use an offline reinforcement learning algorithm to train the policy using offline data, learn an initial policy, and evaluate it in an online environment. When the success rate of the policy is greater than or equal to the switching threshold, switch to online reinforcement learning training. The offline reinforcement learning algorithm is the improved IQL algorithm.

[0051] Step 2: Use online reinforcement learning algorithm to optimize the initial strategy in real time so that the strategy can be continuously adjusted during execution.

[0052] This invention utilizes an offline-online fusion method to further leverage the advantages of reinforcement learning. This is particularly true during drone docking and robot intelligent control training, where environmental interaction is relatively expensive. This method not only improves training efficiency but also enhances the overall performance of decision-making strategies.

[0053] I. The principles and specific implementation manners involved in the present invention are introduced in detail as follows:

[0054] IQL (Implicit Q-Learning) is an efficient offline reinforcement learning algorithm. Since its proposal, it has quickly become an important research progress in this field. Its core innovation lies in introducing the method of implicit policy optimization. By decoupling the behavior policy and the target policy, it effectively alleviates the instability problem caused by out-of-distribution (OOD) actions during the offline RL training process.

[0055] Traditional offline RL methods usually rely on explicit behavior constraint mechanisms (Behavior Constraints), such as behavior cloning (Behavior Cloning, BC) loss or policy constraint terms, to ensure that the learned policy does not deviate from the training data distribution. However, these methods often introduce additional optimization difficulties and may lead to the policy converging to a sub-optimal solution. Without explicit behavior constraints, IQL can still effectively learn the optimal policy through quantile regression Q-learning and advantage-weighted policy extraction, thus greatly simplifying the algorithm structure and at the same time improving the stability and generalization ability of the policy.

[0056] On this basis, the present invention makes multiple optimizations and improvements to IQL to further improve its training efficiency and policy quality in complex tasks. Specifically, the present invention introduces the following key optimization methods:

[0057] Multiple Q-Networks integration: Multiple Q networks (similar to Ensemble Q-Learning) are adopted to reduce the uncertainty of single Q-value estimation and improve the stability of the value function. By calculating the mean or minimum value of multiple Q-values, the overestimation bias is effectively reduced, thereby improving the policy performance.

[0058] Importance Sampling (IS): During the training process, importance sampling is used to adjust the weights of different data samples to enhance the learning ability of the policy on out-of-distribution samples (OOD Samples). This method can effectively alleviate the distribution shift problem of the data set, thereby improving the generalization ability of the policy.

[0059] Cosine Learning Rate Decay: The present invention adopts cosine annealing learning rate decay, enabling a relatively large learning rate to be used for rapid exploration in the initial stage of training, while gradually reducing the learning rate in the later stage of training to improve convergence stability and avoid policy oscillation.

[0060] In the embodiments of the present invention, this optimized IQL variant is referred to as MiQL (Modified IQL). This improvement method can not only accelerate the convergence speed of offline RL training, but also enhance the performance of the policy in high-dimensional complex tasks, enabling it to learn better decision-making strategies in more challenging environments. Specifically as follows:

[0061] ① Multi-Q network integration and mitigation of overestimation bias:

[0062] In reinforcement learning, overestimation bias mainly stems from the effects of random noise, model error, and sub-optimal actions, resulting in the systematic overestimation of the value function Q(s,a). This bias is particularly significant in Q-learning-based algorithms, especially those that rely on a single Q network for update, where the policy optimization direction is prone to deviate from the true optimal solution due to the estimation bias of the target value.

[0063] To mitigate this problem, the present invention optimizes the calculation method of the Q network's loss function in MiQL and introduces multiple Q-networks for integration to reduce overestimation bias. The core idea is to use the prediction results of multiple Q networks to reduce the noise impact of a single model, thereby limiting the upper bound of the target value and making the Q-value estimation more conservative. In this method, multiple independent Q networks are first defined, and the parameters of each network are independently initialized, specifically expressed as:

[0064]

[0065] These Q networks capture different value estimation patterns during different initializations and training processes, thus providing a more diverse data representation. This diversity helps to improve the stability and generalization ability of the Q value. The final Q-value target Can adopt various calculation methods, such as the minimum value method and the weighted average method, etc.

[0066] In the embodiments of the present invention, the minimum value method follows the principle of "more conservative value estimation", that is, the minimum value among the prediction results of multiple Q networks is selected as the target value to suppress the overestimation problem and thus enhance the robustness of the policy:

[0067]

[0068] Among them, γ is the discount factor, r is the immediate reward, and s′ and a′ represent the next state and the next action respectively. This method can effectively suppress the overestimation problem of Q-values, but at the same time, it may introduce a certain degree of conservatism, making the policy update tend to be cautious, thus affecting the exploration ability.

[0069] Another method is to perform weighted averaging on the outputs of multiple Q-networks to balance the conservatism of estimation and the flexibility of learning:

[0070]

[0071] While alleviating the overestimation bias, this method makes the policy update smoother, avoiding the policy being too conservative due to the underestimation of a single Q-network, thus achieving a better balance between robustness and exploration.

[0072] In the embodiments of the present invention, during the policy update process, the policy π depends on the values integrated by multiple Q-networks for optimization, and its update method is as follows:

[0073]

[0074] Among them, β is the temperature coefficient, which is used to adjust the sensitivity of the policy to different action values. A smaller β makes the policy more inclined to select actions with high Q-values, thus promoting more deterministic decisions, while a larger β increases the exploration degree, enabling the policy to try more different actions to improve the generalization ability and adaptability.

[0075] The multi-Q-network integration method can effectively alleviate the overestimation bias. By fusing the prediction results of multiple Q-networks, the overestimation problem caused by a single network can be reduced, thereby improving the stability of the value function. Secondly, this method helps to reduce the impact of model errors. In a complex environment, a single Q-network may be interfered by data noise or model errors, while the integrated model effectively reduces the impact of individual errors on estimation by taking the mean or minimum value.

[0076] In addition, the multi-Q-network method can also enhance the generalization ability of the policy. It can capture the data distribution more comprehensively, making the policy more adaptable in different task environments, especially suitable for offline reinforcement learning tasks with noise or sub-optimal data. Finally, this method can improve the training stability. Since the calculation of the target value is more robust, the impact of error propagation during the policy update process is significantly reduced, making the training process smoother and reducing violent oscillations.

[0077] The present invention effectively alleviates the overestimation bias problem in offline RL training by introducing multi-Q network integration. Compared with the traditional single-Q network method, the multi-Q network provides a more robust and conservative value estimation method, making the policy optimization more stable and generalizable. In addition, in complex environments, this method can significantly improve the sample utilization efficiency, especially in tasks containing noise or sub-optimal data.

[0078] ② Importance sampling:

[0079] In the process of updating the policy and value function of the Miql algorithm, the present invention uses the logarithmic probabilities of the behavior policy and the target policy to calculate the importance sampling weights to correct the data distribution bias. The calculation formula of the importance sampling weights is as follows:

[0080]

[0081] where w(a|s) represents the importance sampling weight, π target (a|s) represents the currently optimized target policy, and π behavior (a|s) represents the behavior policy used in the data collection process.

[0082] In the offline reinforcement learning scenario, data is usually collected by multiple behavior policies, so its distribution may deviate significantly from the currently optimized target policy, that is, the distribution does not match. This deviation will introduce policy bias, reduce the utilization efficiency of data, and may even cause the policy to fall into a sub-optimal solution or encounter out-of-distribution problems.

[0083] The core idea of importance sampling is to adjust the weights of data when updating the value function and policy, so as to more accurately estimate the expected value of the target policy. The experience replay in traditional reinforcement learning generally assumes that the data is independent and identically distributed, while in offline RL, due to the data distribution being restricted by the behavior policy, the learning of the target policy is restricted. By introducing importance sampling, the present invention can re-weight the empirical data to make it more consistent with the distribution of the target policy, thereby improving the effectiveness of policy optimization.

[0084] ③ Cosine annealing learning rate decay:

[0085] In the MiQL training process, the present invention adopts a cosine annealing strategy to adjust the learning rate to achieve a good balance between exploration and stability. Specifically, a relatively high learning rate is used at the beginning so that the policy can quickly adapt to the distribution of the behavior data; as the training progresses, the learning rate gradually decays, thereby effectively reducing the risk of overfitting and improving the generalization ability of the model.

[0086] The mathematical expression of cosine annealing learning rate decay is as follows:

[0087]

[0088] Among them, η t represents the learning rate at the current step size t, η max and η min are the maximum and minimum values of the learning rate respectively, and T is the total number of training steps.

[0089] This smooth decay process enables the model to adapt to offline data faster in the initial stage of training, improving the sampling efficiency of the policy; at the same time, it can effectively suppress problems such as gradient explosion / gradient vanishing in the later stage of training, thereby improving the stability of the algorithm. In addition, this way of gradually reducing the learning rate can also ensure that the model performs more fine-grained optimization in the later stage, promoting the transition of the policy from sub-optimal solutions to more optimal solutions and reducing the risk of falling into local optimal solutions.

[0090] See Figure 2 As shown, in order to further improve the practical application ability of the policy, combining with the policy training optimization method combining off-policy and on-policy reinforcement learning described in the above embodiments, the present invention provides a policy optimization framework. This framework integrates the data-efficient utilization ability of off-policy reinforcement learning and the real-time optimization ability of on-policy reinforcement learning, enabling the policy to be dynamically adjusted during the task execution process to adapt to changes in different environmental conditions.

[0091] In this framework, policy training is mainly divided into two key stages:

[0092] Offline Pretraining stage: Train MiQL through historical data to learn an initial policy, enabling it to better adapt to the state-action distribution covered by the training data and providing a good foundation for subsequent online optimization.

[0093] Online Fine-tuning stage: During the actual task execution process, we use on-policy reinforcement learning (such as PPO) to fine-tune the policy and introduce an Adaptive Adjustment Strategy to further improve the generalization ability and real-time response ability of the policy in different task scenarios.

[0094] This framework not only makes full use of offline data for efficient policy training, but also combines online learning methods so that the policy can be continuously adjusted during execution to achieve a more stable and efficient optimization effect. That is, the present invention combines MiQL with this framework and proposes a new solution named AO2D-MiQL (Adaptive Offline-to-Online Decision Optimization with MiQL) to further promote the application and development of reinforcement learning in complex environments.

[0095] As Figure 2 shown, the environmental part is the UAV docking task. Miql combines IQL and multi-Q network integration, importance sampling, and cosine annealing learning rate decay parts. The overall AO2D-Miql is an adaptive decision optimization framework that combines offline and online reinforcement learning.

[0096] First, use the offline reinforcement learning algorithm Miql to learn using the offline buffer data obtained through the expert policy, and continuously perform iterative testing during the training process. When the success rate (SR) reaches the preset threshold α, switch to online reinforcement learning to further optimize the policy. Interact with the environment to obtain data and put the data into the online buffer. Complete the iteration of online learning by fusing online and offline data into the fusion buffer database. Continuously test the success rate during this process. If the success rate is less than the threshold, perform offline learning; if it is greater than or equal to the threshold, perform online training. And so on, continuously iterate until the task reaches a fixed number of rounds or exceeds a certain threshold, and the task ends.

[0097] The present invention dynamically adjusts the training method, enabling the policy to make full use of offline data for pre-training in the initial stage and switch to the online training stage after the performance reaches a certain level to further optimize the policy performance. This not only avoids the generalization problems that pure offline RL may face but also alleviates the high cost and safety hazards existing in the exploration process of pure online RL.

[0098] A policy training optimization framework that combines offline and online reinforcement learning provided by an embodiment of the present invention has the same implementation principle and technical effects as those of the foregoing method embodiment. For a brief description, for the parts not mentioned in this embodiment, reference can be made to the corresponding content in the foregoing method embodiment, and details will not be repeated here.

[0099] II. Experimental verification:

[0100] Comprehensive experiments were conducted in various aerial docking scenarios, covering different initial docking configurations; the specific test scenario is the UAV aerial docking task.

[0101] Based on this task, multiple different noise and two UAV initial pose scenarios were set up, and different reinforcement learning algorithms were used for training and testing. The initial environment includes multiple forms, as shown in Table 1 below:

[0102] Table 1 Initial Environment

[0103]

[0104] The "Environment" column in Table 1 represents the naming convention for different docking environment initialization methods, including six initial attitude and speed settings. In all environments, UAV B (UAVB) is positioned 0.15 meters away from the target docking point along the docking direction (y-axis). The initial speed and angular velocity are randomly sampled within the ranges of (-0.1, 0.1) m / s and (-0.2, 0.2) rad / s. If the "Speed" column is set to "T", it means randomness is introduced in the speed and angular velocity; if set to "F", no randomness is added and the speed and angular velocity are both initialized to 0. The "xz-axis" column specifies the displacement range in the x-axis and z-axis directions relative to the target docking position. Columns Angle A rp and A y respectively represent the initial roll angle, pitch angle, and yaw angle of UAV A. If the values in these columns are set to "F", it means no randomness is introduced and the angles are initialized to 0. Similarly, Angle B rp and B y also follow the same logic, representing the initial angles of UAV B.

[0105] First, two datasets were generated using an expert policy and collected under different docking environments and state noise settings. Dataset Data1 was generated by combining four different noise settings in the environment docking-v2; while Dataset Data2 was generated by combining the default noise (default) and no noise (None) settings in the environment docking-v1. As shown in Table 2 below:

[0106] Table 2 Different Expert Data Tests

[0107]

[0108] The first column of Table 2 represents the index, the second column lists the offline reinforcement learning algorithms, and the remaining columns report the success rates of each algorithm in 100 tests under different docking environments. The results show that Align-iql, IQL, and Miql perform better on Data2 than on Data1.

[0109] Furthermore, after introducing the AO2D-Miql framework based on Miql, the overall model was trained, and the results are shown in Table 3 below:

[0110] Table 3 Comparison Results between Miql and AO2D-Miql

[0111]

[0112] As shown in the results of Table 3, it can be seen that A2D-Miql has an improvement effect in different environments.

[0113] From the description of the above embodiments, those skilled in the art can know that: In response to the challenges in the UAV aerial docking task, the present invention proposes a strategy training optimization method combining off-line and on-line reinforcement learning. The performance of the baseline algorithm - Implicit Q Learning (IQL) is improved, and a variety of improvement techniques are introduced, including integrating multiple Q networks to alleviate the overestimation problem, using importance sampling to improve the data utilization efficiency, and using cosine learning rate decay to enhance the stability of policy training. On this basis, an adaptive off-line - on-line decision optimization framework is further constructed to improve the success rate of the model in the on-line test stage. This framework can dynamically and seamlessly switch between the off-line training and on-line training stages, while making full use of the high efficiency of off-line learning, ensuring the adaptability of the model to the on-line environment.

[0114] The present invention has conducted comprehensive experiments in a variety of aerial docking scenarios, covering different initial docking configurations. The experimental results show that the proposed algorithm is superior to the existing off-line reinforcement learning baseline methods under all test conditions, and the success rate is significantly improved. In particular, compared with the pure off-line MiQL algorithm, the success rate is significantly improved. This result indicates that the method can effectively bridge the gap between off-line reinforcement learning and on-line environment adaptability. In addition, the method has been successfully applied to real-world scenarios, further verifying its feasibility and superiority in actual UAV aerial docking tasks.

[0115] Furthermore, referring to Figure 3 as shown, an embodiment of the present invention further provides an electronic device that can execute the above method and system. The electronic device may include a processor 10, a memory 11, a communication bus 12, and a communication interface 13, and may further include a computer program stored in the memory 11 and executable on the processor 10.

[0116] Among them, in some embodiments, the processor 10 may be composed of an integrated circuit. For example, it may be composed of a single packaged integrated circuit, or may be composed of multiple packaged integrated circuits with the same or different functions, including a combination of one or more central processing units (CPUs), microprocessors, digital processing chips, graphics processors, and various control chips. The processor 10 is the control core (Control Unit) of the electronic device, connecting various components of the entire electronic device through various interfaces and circuits, and executing various functions of the electronic device and processing data by running or executing programs or modules stored in the memory 11, and calling data stored in the memory 11.

[0117] Those skilled in the art should understand that the embodiments of the present invention can be provided as methods, systems, devices, or computer program products, etc. Therefore, the present invention can adopt the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present invention can adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0118] It should be noted that the word "comprising" does not exclude the existence of components or steps not listed in the claims. The word "a" or "an" preceding a component does not exclude the existence of multiple such components. The present invention can be implemented by means of hardware including several different components and by means of a properly programmed computer.

[0119] In the description of the present invention, it should be noted that: in some processes described in the specification and drawings of this application, multiple operations appear in a specific order, but it should be clearly understood that these operations can be executed not in the order in which they appear in this article or in parallel. In addition, various serial numbers, etc. are only for descriptive purposes and cannot be understood as indicating or implying relative importance.

[0120] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.

[0121] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present invention. The above is only the specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art within the technical scope disclosed by the present invention can easily think of changes or substitutions, which should all be covered within the protection scope of the present invention.

Claims

1. A method for optimizing policy training by combining offline and online reinforcement learning, characterized in that The method includes the following steps: Step 1: Use an offline reinforcement learning algorithm to train a policy through offline data, learn an initial policy, and evaluate it in an online environment. When the success rate of the policy is greater than or equal to the conversion threshold, switch to online reinforcement learning training; Step 2: Use an online reinforcement learning algorithm to optimize the initial policy in real time, so that the policy is continuously adjusted during execution.

2. The method for optimizing policy training by combining offline and online reinforcement learning according to claim 1, wherein, In the above Step 1, the offline reinforcement learning algorithm is: an improved IQL algorithm; The improvement method includes: Reduce the estimation bias through multi-Q network integration; Use importance sampling to adjust the data weights and correct the data distribution bias; Use cosine learning rate decay to adjust the learning rate.

3. The method for optimizing strategy training by combining offline and online reinforcement learning according to claim 2, wherein Reducing the estimation bias through multi-Q network integration specifically means: Define multiple independent Q networks, and use multiple Q networks to reduce the uncertainty of a single Q value estimation. The parameters of each network are independently initialized, specifically expressed as: In the formula, represents an independent Q-parameter network; Then calculate the minimum value or mean value of multiple Q values to reduce the overestimation bias; where: The calculation formula for the minimum value is: The calculation formula for the mean value is: wherein, represents the Q-value target, γ is the discount factor, r is the immediate reward, s and a respectively represent the state and the action, s' and a' respectively represent the next state and the next action; N represents the number of Q networks.

4. A strategy training optimization method combining offline and online reinforcement learning according to claim 2, characterized in that The calculation formula for the importance sampling weight is: Among them, w(a|s) represents the importance sampling weight, and π target (a|s) represents the target policy being optimized currently, and π behavior (a|s) represents the behavior policy used during the data collection process.

5. A strategy training optimization method combining offline and online reinforcement learning according to claim 2, characterized in that The calculation formula for cosine learning rate decay is: Among them, η t represents the learning rate at the current step size t, η max and η min are the maximum and minimum values of the learning rate respectively, and T is the total number of training steps.

6. The method for optimizing policy training by combining offline and online reinforcement learning according to claim 1, characterized in that, In the above Step 2, the online reinforcement learning algorithm is the PPO algorithm.

7. A policy training optimization framework combining offline and online reinforcement learning, characterized in that Apply a policy training optimization method combining offline and online reinforcement learning as described in any one of claims 1 to 6 to perform policy training optimization. The framework includes: An offline training module, which is used to use an offline reinforcement learning algorithm to train a policy through offline data, learn an initial policy, and evaluate it in an online environment. When the success rate of the policy is greater than or equal to the conversion threshold, switch to online reinforcement learning training; An online training module, which is used to use an online reinforcement learning algorithm to optimize the initial policy in real time, so that the policy is continuously adjusted during execution.

8. An electronic device, characterized in that, It includes a processor and a memory. The memory stores machine-executable instructions that can be executed by the processor. The processor executes the machine-executable instructions to implement a policy training optimization method combining offline and online reinforcement learning as described in any one of claims 1 to 6.

Images