Design method, device and equipment for same-strategy multi-target reinforcement learning framework

Abstract

The invention relates to a design method, device and equipment of a same-strategy multi-objective reinforcement learning framework. The method comprises the following steps: constructing a same-strategy multi-objective optimization model; the same-strategy multi-objective optimization model comprises a pre-training model and an online strengthening model. And the pre-training model performs optimization training on the current action strategy based on the expert demonstration data to obtain a to-be-enhanced strategy. On the strategy optimization model, the target completion capability of a to-be-enhanced strategy is evaluated, target sampling probability distribution is obtained according to the target completion capability, a behavior target is sampled according to the target sampling probability distribution, and training data is collected according to the behavior target obtained through sampling; and on the basis of the collected training data, a KL regularized reinforcement learning algorithm is adopted to carry out online optimization on the to-be-reinforced strategy, and an optimal target strategy is output. By adopting the method, the reasoning efficiency and precision of the strategy of velocity vector control of the fixed-wing aircraft can be improved.

Description

Technical Field

[0001] The present invention relates to the technical field of multi-objective strategy optimization, and in particular to a design method, device and equipment for a same-strategy multi-objective reinforcement learning framework. Background Art

[0002] Compared with traditional reinforcement learning algorithms, multi-goal reinforcement learning algorithms (Goal-Conditioned Reinforcement Learning or Multi-Goal Reinforcement Learning) require additional exploration of the goal space. A larger exploration space means that it is more difficult for reinforcement learning algorithms to explore reward signals during training, resulting in low exploration efficiency and difficulty in training effective strategies. Therefore, multi-goal reinforcement learning algorithms face more severe exploration challenges.

[0003] In order to improve the exploration efficiency, existing multi-objective reinforcement learning algorithms generally use Hindsight Experience Replay technology. Hindsight Experience Replay assumes a state in the failed trajectory sampled during training as the initial goal, so that the failed trajectory that replaces the goal becomes a successful trajectory. Hindsight Experience Replay technology improves the reward signal density in the training data by modifying the goal, thereby improving the training efficiency. However, Hindsight Experience Replay technology leads to:

[0004] (1) Since the training data is modified, the policy of the sampled data is no longer the policy currently being trained, so it can only be applied to the off-policy reinforcement learning algorithm;

[0005] (2) Since only one state can be used as a hypothetical target, it is not applicable to non-Markov reward problems. The calculation of non-Markov rewards depends on multiple steps of states and actions. Even multiple consecutive steps of states and actions on a failure trajectory may not necessarily meet the requirements of a non-Markov reward function. Therefore, in non-Markov reward problems, it is difficult for post-experience replay technology to improve the reward signal density in training data by modifying the target. Summary of the invention

[0006] Based on this, it is necessary to provide a design method and device for a same-strategy multi-objective reinforcement learning framework that can improve the efficiency and accuracy of solving multi-objective problems with non-Markov rewards in response to the above technical problems.

[0007] A design method for a same-strategy multi-objective reinforcement learning framework, the method comprising:

[0008] Construct a same-strategy multi-objective optimization model. The same-strategy multi-objective optimization model includes: pre-training model and strategy optimization model.

[0009] The pre-training model optimizes and trains the current action strategy based on expert demonstration data to obtain the strategy to be strengthened.

[0010] In the strategy optimization model, the goal completion ability of the strategy to be reinforced is evaluated, the target sampling probability distribution is obtained according to the goal completion ability, the behavior goals are sampled according to the target sampling probability distribution, and training data is collected according to the sampled behavior goals. The KL regularized reinforcement learning algorithm is used on the collected training data to perform online optimization on the strategy to be reinforced and output the optimal target strategy.

[0011] A design device for a same-strategy multi-objective reinforcement learning framework, the device comprising:

[0012] The model building module is used to build a same-strategy multi-objective optimization model. The same-strategy multi-objective optimization model includes: a pre-trained model and a strategy optimization model.

[0013] The pre-training module is used to optimize the current action strategy based on the expert demonstration data of the pre-training model to obtain the strategy to be strengthened.

[0014] The online reinforcement module is used to evaluate the target completion ability of the strategy to be reinforced on the policy optimization model, obtain the target sampling probability distribution according to the target completion ability, sample the behavior target according to the target sampling probability distribution, collect training data according to the sampled behavior target, and use the KL regularized reinforcement learning algorithm on the collected training data to perform online optimization on the strategy to be reinforced and output the optimal target strategy.

[0015] A computer device comprises a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the following steps are implemented:

[0016] Construct a same-strategy multi-objective optimization model. The same-strategy multi-objective optimization model includes: pre-training model and strategy optimization model.

[0017] The pre-training model optimizes and trains the current action strategy based on expert demonstration data to obtain the strategy to be strengthened.

[0018] In the strategy optimization model, the goal completion ability of the strategy to be reinforced is evaluated, the target sampling probability distribution is obtained according to the goal completion ability, the behavior goals are sampled according to the target sampling probability distribution, and training data is collected according to the sampled behavior goals. The KL regularized reinforcement learning algorithm is used on the collected training data to perform online optimization on the strategy to be reinforced and output the optimal target strategy.

[0019] The design method, device and equipment of the same-strategy multi-objective reinforcement learning framework are as follows: first, the pre-training model uses expert demonstration data to optimize the current action strategy and form a strategy to be reinforced. In this way, the model can capture the expert's strategy selection and decision-making process, thereby laying a solid foundation for subsequent reinforcement learning. This step not only improves the initial quality of the strategy, but also provides a good starting point for subsequent multi-objective optimization. Next, the introduction of the online reinforcement model enables the model to systematically evaluate the goal completion ability of the strategy to be reinforced. By evaluating the goal completion ability of the strategy to be reinforced, the model can effectively identify the difficulty of completing any goal for the strategy to be reinforced, and then, according to the task completion of the evaluation, a goal with moderate difficulty is sampled for sampling training data. This process can achieve the purpose of dynamically adjusting the complexity of the learning task, and can gradually improve the learning effect of the model, thereby ensuring the adaptability and effectiveness of the strategy in a multi-objective scenario. Based on the training data sampled by the above method, the model uses the KL regularized reinforcement learning algorithm to optimize the strategy to be reinforced online. This method can not only ensure the stability of the strategy update, but also effectively avoid the degradation of the strategy performance due to excessive exploration. The use of KL regularization allows the similarity with the original policy to be maintained when updating the policy, thereby achieving a smooth learning process, thereby improving the convergence speed of the model and the quality of the final policy. Through the above steps, the same-policy multi-objective optimization model effectively solves the multi-objective problem. Since it does not include post-experience replay, a technology that is not suitable for solving non-Markov reward problems, it can be applied to solving both Markov reward problems and non-Markov reward problems. Finally, by integrating these technical solutions, the efficiency and accuracy of the model's policy reasoning on multi-objective problems have been significantly improved, thereby achieving accurate and efficient reasoning for the velocity vector control of fixed-wing aircraft. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 A schematic diagram of a flow chart of a design method of a same-strategy multi-objective reinforcement learning framework in one embodiment;

[0021] Figure 2 A flowchart of a same-strategy multi-objective reinforcement learning training step in one embodiment;

[0022] Figure 3 A structural block diagram of a design device for a same-strategy multi-objective reinforcement learning framework in one embodiment;

[0023] Figure 4 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0024] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0025] In one embodiment, Figure 1 As shown, a design method for a same-strategy multi-objective reinforcement learning framework is provided, including the following steps:

[0026] Step 102, construct a same-strategy multi-objective optimization model.

[0027] The same-strategy multi-objective optimization model includes: pre-training model and strategy optimization model.

[0028] Step 104: The pre-trained model optimizes and trains the current action strategy based on the expert demonstration data to obtain the strategy to be strengthened.

[0029] Step 106, in the strategy optimization model, by evaluating the target completion capability of the strategy to be reinforced, obtaining the target sampling probability distribution according to the target completion capability, sampling the behavior target according to the target sampling probability distribution, collecting training data according to the sampled behavior target, and using the KL regularized reinforcement learning algorithm on the collected training data to perform online optimization on the strategy to be reinforced, and output the optimal target strategy.

[0030] In the above-mentioned design method of a same-strategy multi-objective reinforcement learning framework, first, the pre-training model uses expert demonstration data to optimize the current action strategy and form a strategy to be reinforced. In this way, the model can capture the expert's strategy selection and decision-making process, thereby laying a solid foundation for subsequent reinforcement learning. This step not only improves the initial quality of the strategy, but also provides a good starting point for subsequent multi-objective optimization. Next, the introduction of the online reinforcement model enables the model to systematically evaluate the goal completion ability of the strategy to be reinforced. By evaluating the goal completion ability of the strategy to be reinforced, the model can effectively identify the difficulty of completing any goal for the strategy to be reinforced, and then, according to the evaluated task completion, a moderately difficult goal is sampled for sampling training data. This process can achieve the purpose of dynamically adjusting the complexity of the learning task, and can gradually improve the learning effect of the model, thereby ensuring the adaptability and effectiveness of the strategy in a multi-objective scenario. Based on the training data sampled by the above method, the model uses the KL regularized reinforcement learning algorithm to optimize the strategy to be reinforced online. This method can not only ensure the stability of the strategy update, but also effectively avoid the degradation of strategy performance due to excessive exploration. The use of KL regularization allows the similarity with the original policy to be maintained when updating the policy, thereby achieving a smooth learning process, thereby improving the convergence speed of the model and the quality of the final policy. Through the above steps, the same-policy multi-objective optimization model effectively solves the multi-objective problem. Since it does not include post-experience replay, a technology that is not suitable for solving non-Markov reward problems, it can be applied to solving both Markov reward problems and non-Markov reward problems. Finally, by integrating these technical solutions, the efficiency and accuracy of the model's policy reasoning on multi-objective problems have been significantly improved, thereby achieving accurate and efficient reasoning for the velocity vector control of fixed-wing aircraft.

[0031] In one embodiment, the pre-trained model optimizes the parameters of the model using the behavior cloning method based on expert demonstration data to obtain the optimization target of the pre-trained model parameters:

[0032]

[0033] in, is the optimization target of the pre-trained model parameters, a is the current action of the fixed-wing aircraft, and s is the current state of the fixed-wing aircraft; is an expert demonstration dataset consisting of state-action (s, a), is a state-action pair sampled from the expert demonstration dataset, is the expectation of the state-action pair, π θ The strategy to be strengthened is composed of the pre-trained model parameters. According to the optimization goal, the current action strategy is optimized and trained to obtain the strategy to be strengthened.

[0034] In one embodiment, on the strategy optimization model, the strategy to be strengthened being trained is periodically evaluated to obtain evaluation data, and the evaluation data is recorded in a preset strategy evaluation set, and the target completion capability of the strategy to be strengthened is estimated using a Gaussian mixture model based on the evaluation data in the strategy evaluation set. The strategy evaluation set includes: target samples that can be completed by the strategy to be strengthened in the evaluation and weights corresponding to the target samples.

[0035] In one embodiment, the target sampling probability distribution during training is generated according to the target completion capability:

[0036]

[0037] Among them, f(g) is the strategy probability of sampling target g, p ag (g) is the probability of the strategy to be strengthened to complete the goal g, g is the target of the current evaluated probability, and g' is the set of targets of the traversed evaluated probability.

[0038] In one embodiment, the target is a target velocity vector to be achieved by the fixed-wing drone. The behavioral target is a target used when collecting data.

[0039] In one embodiment, a KL regularized reinforcement learning algorithm is used to optimize the expected discounted cumulative reward of the reinforcement policy:

[0040]

[0041] Among them, J(π θ ) is the expected discounted cumulative reward of the strategy to be strengthened, π θ To strengthen the strategy, is the optimal target expectation, γ is the discount factor, r is the reward function, λ∈[0,1] is the regularization strength, θ0 is the pre-trained model parameter, and a t is the action of the fixed-wing aircraft at the current time t, s t is the state of the fixed-wing aircraft at the current time t. Take the target reinforcement strategy corresponding to the maximum value of the expected discounted cumulative reward as the optimal target strategy output.

[0042] In one embodiment, if Figure 2 As shown, a same-strategy multi-objective reinforcement learning training step is provided, which specifically includes the following contents:

[0043] (1) Pre-training

[0044] The purpose of the pre-training phase is to enable the strategy to have preliminary task completion capabilities. The pre-training phase uses the behavior cloning method to train the strategy based on expert data, and the optimization objectives are as follows:

[0045]

[0046] in, is an expert demonstration dataset consisting of multiple expert trajectories, each of which is a sequence of state-action pairs, π θ represents the policy consisting of parameters θ.

[0047] (2) Online self-learning

[0048] The purpose of the online self-learning stage is to continuously improve the strategy's task completion ability through online learning. In the training process of the multi-objective reinforcement learning algorithm, sampling data based on too simple objectives is not conducive to continuously improving the strategy's task completion ability; sampling data based on too difficult objectives results in almost no positive reward signals, which is also not suitable for continuously improving the strategy's task completion ability. Therefore, this stage is to select objectives that are moderately difficult for the current strategy for training. Specifically, it includes the following three parts:

[0049] ① Evaluate the ability P of the strategy currently being trained ag

[0050] During the training process, maintain a strategy evaluation set B g ={(g, w)}, where g is the target of successful evaluation and w is the weight of the sample. During training, the current strategy is evaluated periodically, and B is decayed before the evaluation begins. g The weights of all samples in the evaluation are w←κw, according to p dg Sample the targets used for evaluation and record the successful targets with weight (g, 1.0) to B g , based on Gaussian Mixture Model (GMM) and B g P a，g Make an estimate.

[0051] ② Targets with moderate sampling difficulty are used for training

[0052] The core idea of ​​this stage is to sample targets that the strategy can accomplish but not accomplish well enough for training. For a target g, the probability of sampling g is:

[0053]

[0054] ③Use the same-policy reinforcement learning algorithm with KL regularization (Kullback-Leibler divergence) to update the strategy online

[0055] Based on the target sampling data selected in step ②, use the following optimization strategy:

[0056]

[0057] Where λ∈[0,1] is the regularization strength and θ0 is the parameter of the pre-trained policy. This optimization objective is equivalent to searching for a policy near the pre-trained policy.

[0058] It should be understood that although Figure 1-Figure 2 The steps in the flowchart are shown in sequence as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. Moreover, Figure 1-Figure 2 At least part of the steps may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least part of the sub-steps or stages of other steps.

[0059] In one embodiment, Figure 3 As shown, a design device for a same-strategy multi-objective reinforcement learning framework is provided, including: a model building module 302, a pre-training module 304 and an online reinforcement module 306, wherein:

[0060] The model building module 302 is used to build a same-strategy multi-objective optimization model. The same-strategy multi-objective optimization model includes: a pre-training model and a strategy optimization model.

[0061] The pre-training module 304 is used for the pre-training model to optimize the current action strategy based on the expert demonstration data to obtain the strategy to be strengthened.

[0062] The online reinforcement module 306 is used to evaluate the target completion ability of the strategy to be reinforced on the strategy optimization model, obtain the target sampling probability distribution according to the target completion ability, sample the behavior target according to the target sampling probability distribution, collect training data according to the sampled behavior target, use the KL regularized reinforcement learning algorithm on the collected training data to perform online optimization on the strategy to be reinforced, and output the optimal target strategy.

[0063] For the specific definition of a design device for a same-strategy multi-objective reinforcement learning framework, please refer to the definition of a design method for a same-strategy multi-objective reinforcement learning framework mentioned above, which will not be repeated here. Each module in the above-mentioned design device for a same-strategy multi-objective reinforcement learning framework can be implemented in whole or in part by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0064] In one embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as follows: Figure 4 As shown. The computer device includes a processor, a memory, a network interface, a display screen and an input device connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a design method for a same-strategy multi-objective reinforcement learning framework is implemented. The display screen of the computer device can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covered on the display screen, or a button, trackball or touchpad set on the computer device housing, or an external keyboard, touchpad or mouse, etc.

[0065] Those skilled in the art will understand that Figure 3-Figure 4 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present invention, and does not constitute a limitation on the computer device to which the solution of the present invention is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0066] In one embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the following steps are implemented:

[0067] Construct a same-strategy multi-objective optimization model. The same-strategy multi-objective optimization model includes: pre-training model and strategy optimization model.

[0068] The pre-training model optimizes and trains the current action strategy based on expert demonstration data to obtain the strategy to be strengthened.

[0069] In the strategy optimization model, the goal completion ability of the strategy to be reinforced is evaluated, the target sampling probability distribution is obtained according to the goal completion ability, the behavior goals are sampled according to the target sampling probability distribution, and training data is collected according to the sampled behavior goals. The KL regularized reinforcement learning algorithm is used on the collected training data to perform online optimization on the strategy to be reinforced and output the optimal target strategy.

[0070] In one embodiment, when the processor executes the computer program, the following steps are further implemented: the pre-trained model optimizes the parameters of the model using the behavior cloning method based on the expert demonstration data to obtain the optimization target of the pre-trained model parameters:

[0071]

[0072] in, is the optimization target of the pre-trained model parameters, a is the current action of the fixed-wing aircraft, and s is the current state of the fixed-wing aircraft; is an expert demonstration dataset consisting of state-action (s, a), is a state-action pair sampled from the expert demonstration dataset, is the expectation of the state-action pair, π θ The strategy to be strengthened is composed of the pre-trained model parameters. According to the optimization goal, the current action strategy is optimized and trained to obtain the strategy to be strengthened.

[0073] In one embodiment, when the processor executes the computer program, the following steps are further implemented: on the strategy optimization model, the strategy to be strengthened being trained is periodically evaluated to obtain evaluation data, the evaluation data is recorded in a preset strategy evaluation set, and the target completion capability of the strategy to be strengthened is estimated using a Gaussian mixture model based on the evaluation data in the strategy evaluation set. The strategy evaluation set includes: target samples that can be completed by the strategy to be strengthened in the evaluation and the weights corresponding to the target samples.

[0074] In one embodiment, when the processor executes the computer program, the following steps are further implemented: generating a target sampling probability distribution in the training process according to the target completion capability:

[0075]

[0076] Among them, f(g) is the strategy probability of sampling target g, p ag (g) is the probability of the strategy to be strengthened to complete the goal g, g is the target of the current evaluated probability, and g' is the set of targets of the traversed evaluated probability.

[0077] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided by the present invention can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0078] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0079] The above-mentioned embodiments only express several implementation methods of the present invention, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for ordinary technicians in this field, several variations and improvements can be made without departing from the concept of the present invention, and these all belong to the protection scope of the present invention. Therefore, the protection scope of the patent of the present invention shall be subject to the attached claims.

Claims

1. A design method for a same-strategy multi-objective reinforcement learning framework, characterized in that: The method comprises: Constructing a same-strategy multi-objective optimization model; the same-strategy multi-objective optimization model includes: a pre-training model and a strategy optimization model; The pre-training model optimizes and trains the current action strategy based on expert demonstration data to obtain the strategy to be strengthened; In the strategy optimization model, the target completion ability of the strategy to be strengthened is evaluated, the target sampling probability distribution is obtained according to the target completion ability, the behavior target is sampled according to the target sampling probability distribution, and training data is collected according to the sampled behavior target. The KL regularized reinforcement learning algorithm is used on the collected training data to perform online optimization on the strategy to be strengthened, and the optimal target strategy is output.

2. The method according to claim 1, characterized in that The pre-training model optimizes and trains the current action strategy based on expert demonstration data to obtain the strategy to be strengthened, including: The pre-trained model uses the behavior cloning method to optimize the parameters of the model based on the expert demonstration data to obtain the optimization target of the pre-trained model parameters: in, is the optimization target of the pre-trained model parameters, a is the current action of the fixed-wing aircraft, and s is the current state of the fixed-wing aircraft; is an expert demonstration dataset consisting of state-action (s, a), is a state-action pair sampled from the expert demonstration dataset, is the expectation of the state-action pair, π θ The strategy to be strengthened for the pre-trained model parameters; The current action strategy is optimized and trained according to the optimization target to obtain the strategy to be strengthened.

3. The method according to claim 1, characterized in that: In the strategy optimization model, the target completion capability of the strategy to be strengthened is evaluated, including: On the strategy optimization model, the strategy to be strengthened being trained is periodically evaluated to obtain evaluation data, the evaluation data is recorded in a preset strategy evaluation set, and a Gaussian mixture model is used to estimate the target completion capability of the strategy to be strengthened according to the evaluation data in the strategy evaluation set; The strategy evaluation set includes: target samples that can be completed by the strategy to be strengthened in the evaluation and weights corresponding to the target samples.

4. The method according to claim 3, characterized in that Obtaining a target sampling probability distribution according to the target completion capability includes: Generate the target sampling probability distribution during training according to the target completion capability: Among them, f(g) is the strategy probability of sampling target g, p ag (g) is the probability of the strategy to be strengthened to complete the goal g, g is the target of the current evaluated probability, and g' is the set of targets of the traversed evaluated probability.

5. The method according to any one of claims 1 to 4, characterized in that: The target is a target velocity vector to be achieved by the fixed-wing UAV; The behavioral goals described are the goals used when collecting data.

6. The method according to claim 5, characterized in that The KL regularized reinforcement learning algorithm is used to perform online optimization on the strategy to be reinforced, and the optimal target strategy is output, including: The KL regularized reinforcement learning algorithm is used to optimize the expected discounted cumulative reward of the strategy to be reinforced: Among them, J(π θ ) is the expected discounted cumulative reward of the strategy to be strengthened, π θ To strengthen the strategy, is the optimal target expectation, γ is the discount factor, r is the reward function, λ∈[0,1] is the regularization strength, θ0 is the pre-trained model parameter, and a t is the action of the fixed-wing aircraft at the current time t, s t is the state of the fixed-wing aircraft corresponding to the current time t; The target strategy to be reinforced corresponding to the maximum value of the expected discounted cumulative reward is taken as the optimal target strategy output.

7. A design device for a same-strategy multi-objective reinforcement learning framework, characterized in that: The device comprises: A model building module is used to build a same-strategy multi-objective optimization model; the same-strategy multi-objective optimization model includes: a pre-training model and a strategy optimization model; A pre-training module, used for the pre-training model to optimize the current action strategy based on expert demonstration data to obtain a strategy to be strengthened; The online reinforcement module evaluates the target completion capability of the strategy to be reinforced on the strategy optimization model, obtains the target sampling probability distribution according to the target completion capability, samples the behavior target according to the target sampling probability distribution, collects training data according to the sampled behavior target, and uses the KL regularized reinforcement learning algorithm on the collected training data to perform online optimization on the strategy to be reinforced, and outputs the optimal target strategy.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.

Images