A design method, device and equipment of a same-strategy multi-objective reinforcement learning framework

Abstract

The present application relates to a kind of same strategy multi-objective reinforcement learning framework design method, device and equipment.The method includes: the construction same strategy multi-objective optimization model.Same strategy multi-objective optimization model includes: pre-training model and online reinforcement model.Pre-training model is based on expert demonstration data to optimize training current action strategy, obtains the strategy to be reinforced.According to the target completion ability of the target completion ability of the strategy optimization model to the strategy to be reinforced, according to target sampling probability distribution acquisition target sampling probability distribution, according to the behavior target of sampling, according to the behavior target of sampling and collected training data, using KL regularization reinforcement learning algorithm on the training data collected, the strategy to be reinforced is optimized online, and the optimal target strategy is output.The reasoning efficiency and precision of the strategy of fixed-wing aircraft speed vector control can be improved by using the method.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of multi-objective policy optimization, and in particular to a same-policy multi-objective reinforcement learning framework design method, device and equipment. BACKGROUND

[0002] Compared with traditional reinforcement learning algorithms, multi-objective reinforcement learning algorithms need to additionally explore the target space, and a larger exploration space means that it is more difficult to explore the reward signal in training for the reinforcement learning algorithm, resulting in low exploration efficiency and difficulty in training an effective policy. Therefore, multi-objective reinforcement learning algorithms face more severe exploration challenges.

[0003] In order to improve the exploration efficiency, existing multi-objective reinforcement learning algorithms generally use the hindsight experience replay technology. The hindsight experience replay assumes a certain state in the sampled failure trajectory in training as the original target, so that the failure trajectory with the replaced target becomes a successful trajectory. The hindsight experience replay technology improves the reward signal density in the training data by modifying the target, and in turn improves the training efficiency. However, the hindsight experience replay technology leads to:

[0004] (1) Since the training data is modified, the strategy of the sampled data is no longer the strategy being trained, so it can only be applied to off-policy reinforcement learning algorithms;

[0005] (2) Since only a certain state can be assumed as the target, it is not applicable to non-Markov reward problems. The calculation of the non-Markov reward depends on multiple steps of states and actions, and even multiple steps of states and actions on a failure trajectory may not meet the requirements of the non-Markov reward function. Therefore, in the non-Markov reward problem, the hindsight experience replay technology is difficult to improve the reward signal density in the training data by modifying the target. SUMMARY

[0006] Therefore, it is necessary to provide a same-policy multi-objective reinforcement learning framework design method, device and equipment capable of improving the efficiency and accuracy of solving non-Markov reward multi-objective problems.

[0007] A same-policy multi-objective reinforcement learning framework design method, the method comprising:

[0008] The same strategy multi-objective optimization model comprises a pre-training model and a strategy optimization model.

[0009] The pre-training model optimally trains the current action strategy based on expert demonstration data to obtain a to-be-reinforced strategy.

[0010] On the strategy optimization model, the target completion capability of the to-be-reinforced strategy is evaluated, a target sampling probability distribution is obtained according to the target completion capability, a behavior target is sampled according to the target sampling probability distribution, training data is collected according to the sampled behavior target, and a KL regularization reinforcement learning algorithm is used on the collected training data to perform online optimization on the to-be-reinforced strategy, thereby outputting an optimal target strategy.

[0011] A design device of a same strategy multi-objective reinforcement learning framework, the device comprises:

[0012] The model construction module is configured to construct a same strategy multi-objective optimization model. The same strategy multi-objective optimization model comprises a pre-training model and a strategy optimization model.

[0013] The pre-training module is configured to perform optimal training on the current action strategy based on expert demonstration data by using the pre-training model, thereby obtaining a to-be-reinforced strategy.

[0014] The online reinforcement module is configured to perform online optimization on the to-be-reinforced strategy by using a KL regularization reinforcement learning algorithm on the collected training data, thereby outputting an optimal target strategy.

[0015] A computer device comprises a memory and a processor, the memory stores a computer program, and the processor implements the following steps when executing the computer program:

[0016] The same strategy multi-objective optimization model comprises a pre-training model and a strategy optimization model.

[0017] The pre-training model optimally trains the current action strategy based on expert demonstration data to obtain a to-be-reinforced strategy.

[0018] On the strategy optimization model, the target completion capability of the to-be-reinforced strategy is evaluated, a target sampling probability distribution is obtained according to the target completion capability, a behavior target is sampled according to the target sampling probability distribution, training data is collected according to the sampled behavior target, and a KL regularization reinforcement learning algorithm is used on the collected training data to perform online optimization on the to-be-reinforced strategy, thereby outputting an optimal target strategy.

[0019] The above-mentioned design method, device and equipment of the same-strategy multi-objective reinforcement learning framework first optimize and train the current action strategy using expert demonstration data to form a to-be-reinforced strategy. In this way, the model can capture the strategy selection and decision-making process of the expert, thereby laying a solid foundation for subsequent reinforcement learning. This step not only improves the initial quality of the strategy, but also provides a good starting point for subsequent multi-objective optimization. Next, the introduction of the online reinforcement model enables the model to systematically evaluate the target completion ability of the to-be-reinforced strategy. By evaluating the target completion ability of the to-be-reinforced strategy, the model can effectively identify the completion difficulty of any target for the to-be-reinforced strategy, and then sample a moderately difficult target for sampling training data according to the evaluation of the task completion. This process can dynamically adjust the complexity of the learning task and gradually improve the learning effect of the model, thereby ensuring the adaptability and effectiveness of the strategy in a multi-objective context. Based on the training data sampled by the above method, the model uses a reinforcement learning algorithm with KL regularization to optimize the to-be-reinforced strategy online. This method not only ensures the stability of strategy updating, but also effectively avoids the decline in strategy performance due to excessive exploration. The use of KL regularization enables the updated strategy to maintain similarity with the original strategy, thereby achieving a smooth learning process and improving the convergence speed of the model and the quality of the final strategy. Through the above steps, the same-strategy multi-objective optimization model effectively solves the multi-objective problem. Since it does not contain post-experience replay, which is not suitable for solving non-Markov reward problems, it can be applied to solving both Markov reward problems and non-Markov reward problems. Finally, by integrating these technical solutions, the strategy reasoning efficiency and accuracy of the model on multi-objective problems are significantly improved, thereby achieving accurate and efficient reasoning of the speed vector control of the fixed-wing aircraft. BRIEF DESCRIPTION OF DRAWINGS

[0020] Figure 1 A flowchart of a design method of a same-strategy multi-objective reinforcement learning framework in one embodiment;

[0021] Figure 2 A flowchart of a training step of a same-strategy multi-objective reinforcement learning in one embodiment;

[0022] Figure 3 A structural block diagram of a design device of a same-strategy multi-objective reinforcement learning framework in one embodiment;

[0023] Figure 4 An internal structure diagram of a computer device in one embodiment. DETAILED DESCRIPTION

[0024] In order to make the purposes, technical solutions and advantages of the present application clearer, the present application is further described in detail below in combination with the drawings and examples. It should be understood that the specific examples described herein are only used to explain the present application and do not limit the present application.

[0025] In one embodiment, as shown in Figure 1 A design method of a same-policy multi-objective reinforcement learning framework is provided, comprising the following steps:

[0026] Step 102, a same-policy multi-objective optimization model is constructed.

[0027] The same-policy multi-objective optimization model comprises a pre-training model and a policy optimization model.

[0028] Step 104, the pre-training model optimizes and trains the current action policy based on expert demonstration data to obtain a to-be-reinforced policy.

[0029] Step 106, on the policy optimization model, the target completion capability of the to-be-reinforced policy is evaluated, the target sampling probability distribution is obtained according to the target completion capability, the behavior target is sampled according to the target sampling probability distribution, the training data is collected according to the sampled behavior target, the to-be-reinforced policy is online optimized on the collected training data by using the reinforcement learning algorithm with KL regularization, and the optimal target policy is output.

[0030] In the above method for designing a same-strategy multi-objective reinforcement learning framework, first, the pre-training model optimizes the current action strategy using expert demonstration data to form a to-be-reinforced strategy. In this way, the model can capture the strategy selection and decision-making process of the expert, thereby laying a solid foundation for subsequent reinforcement learning. This step not only improves the initial quality of the strategy, but also provides a good starting point for subsequent multi-objective optimization. Next, the introduction of the online reinforcement model enables the model to systematically evaluate the target completion ability of the to-be-reinforced strategy. By evaluating the target completion ability of the to-be-reinforced strategy, the model can effectively identify the difficulty of any one target for the to-be-reinforced strategy, and then sample a moderately difficult target based on the evaluation of the task completion to obtain training data. This process can dynamically adjust the complexity of the learning task and gradually improve the learning effect of the model, thereby ensuring the adaptability and effectiveness of the strategy in a multi-objective context. Based on the training data sampled using the above method, the model uses a reinforcement learning algorithm with KL regularization to optimize the to-be-reinforced strategy online. This method not only ensures the stability of strategy updates, but also effectively avoids the decline in strategy performance due to excessive exploration. The use of KL regularization enables the updated strategy to maintain similarity with the original strategy, thereby achieving a smooth learning process and improving the convergence speed of the model and the quality of the final strategy. Through the above steps, the same-strategy multi-objective optimization model effectively solves the multi-objective problem. Since it does not include post-experience replay, which is not suitable for solving non-Markovian reward problems, it can be applied to both Markovian reward problems and non-Markovian reward problems. Finally, by integrating these technical solutions, the model's strategy reasoning efficiency and accuracy in multi-objective problems have been significantly improved, thereby achieving accurate and efficient reasoning of the speed vector control of the fixed-wing aircraft.

[0031] In one embodiment, the pre-training model optimizes the parameters of the model based on expert demonstration data using a behavior cloning method to obtain an optimization target for the pre-training model parameters:

[0032]

[0033] wherein, is the optimization target for the pre-training model parameters, a is the current action of the fixed-wing aircraft, s is the current state of the fixed-wing aircraft; is a set of expert demonstration data composed of state-action pairs (s, a), is a state-action pair sampled from the set of expert demonstration data, is the expectation of the state-action pair, π θ is a to-be-reinforced strategy composed of pre-training model parameters. The current action strategy is optimized and trained based on the optimization target to obtain the to-be-reinforced strategy.

[0034] In one embodiment, the to-be-reinforced policy is periodically evaluated on the policy optimization model to obtain evaluation data, the evaluation data is recorded in a preset policy evaluation set, and a Gaussian mixture model is used to estimate the target completion capability of the to-be-reinforced policy according to the evaluation data in the policy evaluation set. The policy evaluation set includes target samples that can be completed by the to-be-reinforced policy in evaluation and weights corresponding to the target samples.

[0035] In one embodiment, the target sampling probability distribution in the training process is generated according to the target completion capability:

[0036]

[0037] Wherein, f(g) is the policy probability of sampling target g, p ag (g) is the probability of completing target g by the to-be-reinforced policy, g is the current evaluated probability target, and g' is the set of evaluated probability targets traversed.

[0038] In one embodiment, the target is the target speed vector reached by the fixed-wing unmanned aerial vehicle. The behavior target is the target used when collecting data.

[0039] In one embodiment, the expected discounted cumulative reward of the to-be-reinforced policy is optimized by using a reinforcement learning algorithm with KL regularization:

[0040]

[0041] Wherein, J(π θ ) is the expected discounted cumulative reward of the to-be-reinforced policy, π θ is the to-be-reinforced policy, is the optimal target expectation, γ is the discount factor, r is the reward function, λ∈[0, 1] is the regularization strength, θ0 is the pre-training model parameter, a t is the action corresponding to the current t time of the fixed-wing aircraft, s t is the state corresponding to the current t time of the fixed-wing aircraft. The to-be-reinforced policy corresponding to the maximum value of the expected discounted cumulative reward is taken as the optimal target policy output.

[0042] In one embodiment, as Figure 2 shown, a same-policy multi-target reinforcement learning training step is provided, which specifically includes the following contents:

[0043] (1) Pre-training

[0044] The purpose of the pre-training stage is to enable the policy to have initial task completion capability. The pre-training stage trains the policy based on expert data using a behavior cloning method, and the optimization target is as follows:

[0045]

[0046] wherein, is a set of expert demonstration data composed of multiple expert trajectories, each trajectory is a sequence of state-action pairs, π θ denotes a policy parameterized by θ.

[0047] (2) Online self-course learning

[0048] The purpose of the online self-course learning phase is to continuously improve the task completion ability of the policy through online learning. In the training process of the multi-objective reinforcement learning algorithm, sampling data according to too simple objectives is not conducive to continuously improving the task completion ability of the policy; sampling data according to too difficult objectives results in almost no positive reward signals being obtained, which is also not suitable for continuously improving the task completion ability of the policy. Therefore, this phase selects objectives with moderate difficulty for the current policy for training. Specifically, the following three parts are included:

[0049] ① Evaluate the ability P of the policy currently being trained ag

[0050] During training, a policy evaluation set B g = {(g, w)} is maintained, where g is an evaluation successful objective, and w is the weight of the sample. During training, the current policy is periodically evaluated. Before evaluation, the weights w of all samples in B g are decayed w←κw, and during evaluation, the objectives used for evaluation are sampled according to p dg , and the successful objectives and weights (g, 1.0) are recorded to B g . Based on the Gaussian Mixture Model (GMM) and B g , the p a，g is estimated.

[0051] ② Sample objectives with moderate difficulty for training

[0052] The core idea of this phase is to sample objectives that the policy can complete but not well enough for training. For an objective g, the probability of sampling g is:

[0053]

[0054] ③ Online update of the policy using the same policy reinforcement learning algorithm with KL regularization

[0055] Based on the objective sampling data selected in step ②, the policy is optimized using the following formula:

[0056]

[0057] Where λ∈[0,1] is the regularization strength, and θ0 is the parameter of the pre-trained policy. The optimization objective is equivalent to searching for a policy in the vicinity of the pre-trained policy.

[0058] It should be understood that, although Figures 1-2 The steps in the flowchart are shown sequentially as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order in which these steps are executed, and they can be performed in other orders. Figures 1-2 At least some of the steps in the process may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least some of the sub-steps or stages of other steps.

[0059] In one embodiment, such as Figure 3 As shown, a design apparatus for a multi-objective reinforcement learning framework with the same policy is provided, including: a model building module 302, a pre-training module 304, and an online reinforcement module 306, wherein:

[0060] Model building module 302 is used to build a multi-objective optimization model with the same policy. The multi-objective optimization model with the same policy includes: a pre-trained model and a policy optimization model.

[0061] The pre-training module 304 is used to pre-train the model to optimize the current action strategy based on expert demonstration data, so as to obtain the strategy to be strengthened.

[0062] The online reinforcement module 306 is used to evaluate the target completion ability of the policy to be reinforced in the policy optimization model, obtain the target sampling probability distribution based on the target completion ability, sample behavioral targets based on the target sampling probability distribution, collect training data based on the sampled behavioral targets, and use the KL regularized reinforcement learning algorithm to optimize the policy to be reinforced online on the collected training data, and output the optimal target policy.

[0063] The specific limitation of the apparatus for designing a same-policy multi-objective reinforcement learning framework can refer to the limitation of the method for designing a same-policy multi-objective reinforcement learning framework, which will not be repeated here. Each module in the apparatus for designing a same-policy multi-objective reinforcement learning framework can be realized by software, hardware, and combinations thereof, in whole or in part. Each module can be embedded in or independent of the processor in the computer device in hardware form, or stored in the memory in the computer device in software form, so that the processor can call and execute the operations corresponding to each module.

[0064] In one embodiment, a computer device, which can be a terminal, is provided, and an internal structure diagram of the computer device can be as shown in Figure 4 The computer device includes a processor, a memory, a network interface, a display screen, and an input device connected through a system bus. The processor of the computer device is configured to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for running the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is configured to communicate with external terminals through a network connection. The computer program is executed by the processor to implement a method for designing a same-policy multi-objective reinforcement learning framework. The display screen of the computer device can be a liquid crystal display screen or an electronic ink display screen. The input device of the computer device can be a touch layer overlaid on the display screen, or a key, trackball, or touchpad arranged on the shell of the computer device, or an external keyboard, touchpad, or mouse, etc.

[0065] Those skilled in the art can understand that Figures 3-4 The structure shown in the above

[0066] In one embodiment, a computer device is provided, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the following steps:

[0067] A same-policy multi-objective optimization model is constructed. The same-policy multi-objective optimization model includes a pre-training model and a policy optimization model.

[0068] The pre-training model optimizes and trains the current action policy based on expert demonstration data to obtain a to-be-reinforced policy.

[0069] On the policy optimization model, the target completion capability of the to-be-reinforced policy is evaluated, a target sampling probability distribution is obtained according to the target completion capability, a behavior target is sampled according to the target sampling probability distribution, training data is collected according to the sampled behavior target, a reinforcement learning algorithm with KL regularization is used on the collected training data to perform online optimization on the to-be-reinforced policy, and an optimal target policy is output.

[0070] In one embodiment, the processor, when executing the computer program, also implements the following steps: the pre-training model optimizes the parameters of the model based on the expert demonstration data using a behavior cloning method to obtain an optimization target of the pre-training model parameters:

[0071]

[0072] wherein, is the optimization target of the pre-training model parameters, a is the current action of the fixed-wing aircraft, and s is the current state of the fixed-wing aircraft; is an expert demonstration data set composed of state-action (s, a), is a state-action pair sampled from the expert demonstration data set, is the expectation of the state-action pair, and θ is a to-be-reinforced policy constituted by the pre-training model parameters. The current action policy is optimized and trained according to the optimization target to obtain the to-be-reinforced policy.

[0073] In one embodiment, the processor, when executing the computer program, also implements the following steps: on the policy optimization model, the to-be-reinforced policy being trained is periodically evaluated to obtain evaluation data, the evaluation data is recorded in a preset policy evaluation set, and the target completion capability of the to-be-reinforced policy is estimated using a Gaussian mixture model according to the evaluation data in the policy evaluation set. The policy evaluation set includes: target samples that can be completed by the to-be-reinforced policy in evaluation and weights corresponding to the target samples.

[0074] In one embodiment, the processor, when executing the computer program, also implements the following steps: a target sampling probability distribution in a training process is generated according to the target completion capability:

[0075]

[0076] wherein, f(g) is the strategy probability of sampling target g, p ag (g) is the probability of the to-be-reinforced policy completing target g, g is the current evaluated probability target, and g' is the set of evaluated probability targets traversed.

[0077] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer readable storage medium, and when the computer program is executed, the processes of the above-mentioned embodiments of the methods can be included. Any reference to memory, storage, database or other medium used in the embodiments provided by the present application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration but not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0078] Any combination of the technical features of the above embodiments can be made. In order to make the description simple, all possible combinations of the technical features in the above embodiments are not described, however, as long as the combination of the technical features does not exist, it should be considered as the scope of the present application.

[0079] The above embodiments only express several embodiments of the present application, and the description is more specific and detailed, but it should not be understood as a limitation on the scope of the patent. It should be pointed out that for ordinary skilled in the art, without departing from the concept of the present application, a number of modifications and improvements can be made, which are within the scope of the present application. Therefore, the scope of protection of the present application should be subject to the appended claims.

Claims

1. A design method for a multi-objective reinforcement learning framework with the same policy, characterized in that, The method includes: Construct a multi-objective optimization model with the same policy; the multi-objective optimization model with the same policy includes: a pre-trained model and a policy optimization model; The pre-trained model optimizes the current action strategy based on expert demonstration data to obtain the strategy to be strengthened. Specifically, the pre-trained model optimizes its parameters using a behavior cloning method based on expert demonstration data, yielding the optimization objective for the pre-trained model parameters. ； in, The optimization objective for the pre-trained model parameters is... For the current actions of a fixed-wing aircraft, This represents the current state of the fixed-wing aircraft. For an expert demonstration dataset consisting of state-action (s, a), For state-action pairs sampled from the expert demonstration dataset, For the expectation of state-action pairs, The reinforcement strategy is composed of the parameters of the pre-trained model; The current action strategy is optimized and trained according to the optimization objective to obtain the strategy to be strengthened. On the policy optimization model, the target completion capability of the policy to be strengthened is evaluated. The specific steps are as follows: On the policy optimization model, the policy to be strengthened being trained is periodically evaluated to obtain evaluation data. The evaluation data is recorded in a preset policy evaluation set. Based on the evaluation data in the policy evaluation set, a Gaussian mixture model is used to estimate the target completion capability of the policy to be strengthened. The policy evaluation set includes: target samples that the policy to be strengthened can complete in the evaluation and the weights corresponding to the target samples. Based on the target completion capability, obtain the target sampling probability distribution, sample behavioral targets based on the target sampling probability distribution, collect training data based on the sampled behavioral targets, and use a KL regularized reinforcement learning algorithm on the collected training data to optimize the policy to be reinforced online and output the optimal target policy. The target is the velocity vector reached by the fixed-wing UAV. The behavioral objective is the objective used when collecting data.

2. The method according to claim 1, characterized in that, Obtaining the target sampling probability distribution based on the target completion capability includes: Based on the target completion capability, generate the target sampling probability distribution during the training process: ； in, For sampling target The strategy probability, Let g be the probability that the strategy to be strengthened will achieve the objective g. For the target whose probability is currently being evaluated, The target set whose probabilities are being evaluated is the set to be traversed.

3. The method according to claim 2, characterized in that, The reinforcement learning algorithm with KL regularization is used to optimize the policy to be reinforced online, and the optimal target policy is output, including: The expected discount cumulative reward of the policy to be reinforced is optimized using a reinforcement learning algorithm with KL regularization. ； in, Accumulated rewards for the expected discount of the strategy to be strengthened. For strategies to be strengthened, For the optimal target expectation, As a discount factor, For the reward function, For regularization strength, For pre-trained model parameters, For the actions of a fixed-wing aircraft at time t, This represents the current state of the fixed-wing aircraft at time t. The target strategy to be strengthened corresponding to the maximum value of the expected cumulative reward is taken as the optimal target strategy output.

4. A design device for a multi-objective reinforcement learning framework with the same policy, characterized in that, The apparatus for implementing the method according to any one of claims 1 to 3 comprises: The model building module is used to build a multi-objective optimization model with the same policy; the multi-objective optimization model with the same policy includes: a pre-trained model and a policy optimization model; The pre-training module is used to optimize and train the pre-trained model based on expert demonstration data to obtain the strategy to be strengthened. The online reinforcement module evaluates the target completion capability of the policy to be reinforced on the policy optimization model, obtains the target sampling probability distribution based on the target completion capability, samples behavioral targets based on the target sampling probability distribution, collects training data based on the sampled behavioral targets, and uses a KL regularized reinforcement learning algorithm to optimize the policy to be reinforced online on the collected training data, and outputs the optimal target policy.

5. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 3.

Images