Strategy generation method and device based on multi-objective reinforcement learning

Abstract

The invention relates to the technical field of reinforcement learning, and provides a strategy generation method and device based on multi-objective reinforcement learning. The method comprises the following steps: receiving a multi-target strategy optimization request; constructing multi-target feedback index data according to a plurality of state action sequence fragments corresponding to the multi-target strategy optimization request, the multi-target weight features and a binary evaluator used for evaluating the state action sequence fragments; establishing an explicit multi-target reward model; training a multi-target reward model by using the multi-target feedback index data; establishing a Markov decision process model according to the states and actions corresponding to the plurality of state action sequence fragments and the trained multi-target reward model; and according to the Markov decision process model, in a learning environment corresponding to the multi-target strategy optimization request, training by using a multi-target reinforcement learning algorithm to obtain a multi-target strategy. According to the embodiment of the invention, the strategy generation precision based on multi-target reinforcement learning can be improved.

Description

Technical Field

[0001] The present invention relates to the field of reinforcement learning technology, and in particular to a strategy generation method and device based on multi-objective reinforcement learning. Background Art

[0002] In the field of multi-objective reinforcement learning, traditional research focuses on guiding policy optimization through predefined reward functions. However, due to the inherent complexity of the multi-dimensional optimization objectives themselves, and the uncertainty or difficulty in measuring the states of some optimization dimensions, traditional methods face challenges in dealing with complex real-world objectives. For example, when a robot dog is learning a backflip, it is difficult to design an accurate reward function to guide the robot dog's learning process; the cooling of a data center computer room needs to consider both the risk of IT equipment overheating and energy efficiency, and the internal chip temperature of IT equipment is often difficult to measure directly, resulting in the inability to accurately measure the optimization objective, i.e., the value of the internal chip temperature. In order to overcome the limitations caused by the inherent complexity of multi-dimensional optimization objectives, existing technologies use human feedback to guide the behavior of reinforcement learning agents, improve the performance and flexibility of single-objective optimization, and thus avoid the problems of uncertain target values ​​and complex reward engineering.

[0003] However, in terms of multi-objective reinforcement learning modeling, due to the inherent complexity of the multi-dimensional optimization objectives themselves, the existing technology is not sufficient to solve the problem of policy generation in multi-objective reinforcement learning scenarios. Therefore, there is an urgent need for a policy generation method based on multi-objective reinforcement learning to achieve a more flexible reward model in multi-objective reinforcement learning scenarios based on the inherent complexity of the multi-dimensional optimization objectives themselves, avoid dependence on predefined reward functions, and improve the accuracy of policy generation based on multi-objective reinforcement learning. Summary of the invention

[0004] In view of the inherent complexity of the multi-dimensional optimization objectives themselves in multi-objective reinforcement learning modeling, the existing technology is not sufficient to solve the problem of strategy generation in multi-objective reinforcement learning scenarios. This solution is proposed to overcome the above problems or at least partially solve the above problems.

[0005] On the one hand, an object of some embodiments of this specification is to provide a strategy generation method for multi-objective reinforcement learning, the method comprising:

[0006] Receiving a multi-objective strategy optimization request;

[0007] Constructing multi-objective feedback indicator data according to a plurality of state-action sequence fragments corresponding to the multi-objective strategy optimization request, multi-objective weight features, and a binary evaluator for evaluating the state-action sequence fragments;

[0008] Building explicit multi-objective reward models;

[0009] Training the multi-objective reward model using the multi-objective feedback indicator data;

[0010] Establishing a Markov decision process model according to the states, actions and the trained multi-objective reward model corresponding to the plurality of state-action sequence fragments;

[0011] According to the Markov decision process model, in the learning environment corresponding to the multi-objective strategy optimization request, a multi-objective reinforcement learning algorithm is used to train and obtain a multi-objective strategy.

[0012] Furthermore, according to a plurality of state-action sequence fragments corresponding to the multi-objective strategy optimization request, multi-objective weight features, and a binary evaluator for evaluating the state-action sequence fragments, multi-objective feedback index data is constructed, including:

[0013] Each state-action sequence fragment, the multi-objective weight feature, and the binary evaluator used to evaluate the state-action sequence fragment are respectively used as tuple elements, so as to construct multi-objective feedback indicator data according to the tuple elements.

[0014] Further, using the multi-objective feedback indicator data to train the multi-objective reward model includes:

[0015] Selecting the state and action at a specified time from the state-action sequence fragments of the multi-objective feedback indicator data;

[0016] Inputting the state and action at a specified time into the multi-objective reward model to obtain the corresponding predicted reward;

[0017] Establishing a feedback indicator predictor based on the correlation between the predicted reward and the multi-objective weight feature in the multi-objective feedback indicator data;

[0018] Inputting the predicted reward and the multi-objective weight features in the multi-objective feedback indicator data into the feedback indicator predictor to obtain a feedback indicator prediction result;

[0019] The loss corresponding to the feedback indicator prediction result is calculated according to a preset loss function, and the feedback indicator predictor is trained using a parameter optimization algorithm to update the model parameters in the multi-objective reward model.

[0020] Furthermore, the feedback indicator predictor is established in the following manner:

[0021]

[0022] Among them, P ψ [σ l >σ k |w] represents the feedback indicator predictor, > represents after, σ land σ k All represent fragments of state-action sequences, w represents multi-objective weight features, exp represents an exponential function with the natural constant e as the base, {l,k} represents a set of binary scalar categories, i represents an element in the set of binary scalar categories, It means that at time t and when the binary scalar category is i, the state s it and action a it The corresponding prediction reward, T Indicates transpose.

[0023] Furthermore, a Markov decision process model is established according to the states, actions and the trained multi-objective reward model corresponding to the plurality of state-action sequence fragments, including:

[0024] Determine the current state, current action and next state according to the states and actions corresponding to the plurality of state-action sequence fragments;

[0025] Input the current state and current action into the trained multi-objective reward model to obtain the current multi-objective reward;

[0026] According to the current state, current action, current multi-objective reward and next state, state transition data is constructed to establish a Markov decision process model.

[0027] Further, according to the Markov decision process model, in the learning environment corresponding to the multi-objective strategy optimization request, a multi-objective reinforcement learning algorithm is used to train a multi-objective strategy, including:

[0028] Step 0: Initialize network parameters;

[0029] Step 1: Get the current state and determine whether the current number of steps exceeds the preset threshold;

[0030] Step 2: If the preset threshold is not exceeded, the current action is obtained by random sampling; if the preset threshold is not exceeded, the current action is obtained by sampling based on the control strategy;

[0031] Step 3: Obtain a state transition tuple consisting of the current state, current action, next state, and multi-objective weight features in the multi-objective feedback indicator data;

[0032] Step 4: Call the corresponding trained multi-objective reward model according to the state transition tuple, and input the current state and the current action into the trained multi-objective reward model to obtain the current multi-objective reward;

[0033] Step 5: Add the state transition data consisting of the current state, current action, current multi-target reward and next state to the replay buffer of this iteration; the replay buffer is used to store the state transition data required for each iteration;

[0034] Step 6: Partial batch sampling is performed on the state transition data in the replay buffer of this iteration to obtain sampled batch data, and based on the sampled batch data, multi-objective reinforcement learning is performed using the reinforcement learning loss function to update the network parameters;

[0035] Repeat steps 1 to 6 above until the preset number of iterations is reached.

[0036] Furthermore, in some embodiments, it further comprises:

[0037] Get the preset update frequency of multi-objective feedback indicator data;

[0038] As the number of iterations increases, when the relationship between the preset update frequency and the current number of iterations satisfies a preset condition, a preset number of state-action sequence segments are sampled from the replay buffer, and a preset number of multi-objective weight features are sampled from a preset data set;

[0039] Generate new multi-objective feedback indicator data based on the sampled multi-objective weight features, state-action sequence fragments and the binary evaluator;

[0040] Adding the new multi-objective feedback indicator data to the feedback indicator data set;

[0041] Using the data in the feedback indicator data set, updating the trained multi-objective reward model according to a preset loss function;

[0042] The replay buffer is updated using the updated multi-objective reward model.

[0043] On the other hand, some embodiments of this specification also provide a strategy generation device based on multi-objective reinforcement learning, the device comprising:

[0044] A receiving module, used for receiving a multi-objective strategy optimization request;

[0045] A feedback indicator data construction module, used to construct multi-objective feedback indicator data according to a plurality of state-action sequence fragments corresponding to the multi-objective strategy optimization request, multi-objective weight features, and a binary evaluator for evaluating the state-action sequence fragments;

[0046] Multi-objective reward model building module, used to build an explicit multi-objective reward model;

[0047] A multi-objective reward model training module, used to train the multi-objective reward model using the multi-objective feedback indicator data;

[0048] A Markov decision process model building module is used to build a Markov decision process model according to the states, actions and the trained multi-objective reward model corresponding to the plurality of state-action sequence fragments;

[0049] The multi-objective reinforcement learning module is used to train a multi-objective strategy using a multi-objective reinforcement learning algorithm according to the Markov decision process model in a learning environment corresponding to the multi-objective strategy optimization request.

[0050] On the other hand, some embodiments of the present specification further provide a computer device, including a memory, a processor, and a computer program stored in the memory, wherein when the computer program is executed by the processor, the instructions of the above method are executed.

[0051] On the other hand, some embodiments of the present specification further provide a computer storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor of a computer device, the instructions of the above method are executed.

[0052] On the other hand, some embodiments of the present specification further provide a computer program product, wherein the computer program product comprises a computer program, and when the computer program is executed by a processor of a computer device, the instructions of the above method are executed.

[0053] One or more technical solutions provided by some embodiments of this specification have at least the following technical effects:

[0054] The embodiments of the present specification first automatically receive a multi-objective strategy optimization request, and obtain several state-action sequence fragments, multi-objective weight features and a binary evaluator for evaluating the state-action sequence fragments corresponding to the multi-objective strategy optimization request to construct multi-objective feedback indicator data, and then establish an explicit multi-objective reward model, and use the multi-objective feedback indicator data to train the multi-objective reward model so that the trained multi-objective reward model can be used to characterize the strategy generation tendency corresponding to the multi-objective feedback indicator data, and then establish a Markov decision process model based on the states and actions corresponding to the several state-action sequence fragments and the trained multi-objective reward model, and apply a multi-objective reinforcement learning algorithm for training to achieve efficient and accurate output of multi-objective strategies in complex multi-objective scenarios.

[0055] The above description is only an overview of the technical solutions of some embodiments of this specification. In order to more clearly understand the technical means of some embodiments of this specification, they can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of some embodiments of this specification more obvious and easy to understand, the specific implementation methods of some embodiments of this specification are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] In order to more clearly illustrate some embodiments of this specification or technical solutions in the prior art, the following briefly introduces the drawings required for the embodiments or prior art descriptions. Obviously, the drawings described below are only some embodiments recorded in this specification. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative labor. In the drawings:

[0057] Figure 1 A schematic diagram of an implementation system of a strategy generation method based on multi-objective reinforcement learning in some embodiments of this specification is shown;

[0058] Figure 2 A flowchart of a strategy generation method based on multi-objective reinforcement learning in some embodiments of this specification is shown;

[0059] Figure 3 A schematic diagram of the steps of training a multi-objective reward model using multi-objective feedback indicator data in some embodiments of this specification;

[0060] Figure 4 This is a schematic diagram of the steps of using a multi-objective reinforcement learning algorithm to train a multi-objective strategy in some embodiments of this specification;

[0061] Figure 5 A schematic diagram of the steps of updating the replay buffer using the updated multi-objective reward model in some embodiments of this specification;

[0062] Figure 6 This is a schematic diagram of the structure of a strategy generation device based on multi-objective reinforcement learning in some embodiments of this specification;

[0063] Figure 7 This is a schematic diagram of the computer device structure provided in some embodiments of this specification.

[0064] [Description of Reference Numerals]

[0065] 101. Terminal;

[0066] 102. Server;

[0067] 601, receiving module;

[0068] 602. Feedback indicator data construction module;

[0069] 603. Multi-objective reward model building module;

[0070] 604. Multi-objective reward model training module;

[0071] 605. Markov decision process model building module;

[0072] 606. Multi-objective reinforcement learning module;

[0073] 702. Computer equipment;

[0074] 704, processor;

[0075] 706. Memory;

[0076] 708, driving mechanism;

[0077] 710, input / output interface;

[0078] 712. Input devices;

[0079] 714. Output devices;

[0080] 716. Presentation equipment;

[0081] 718. Graphical user interface;

[0082] 720, network interface;

[0083] 722. Communication link;

[0084] 724. Communication bus. DETAILED DESCRIPTION

[0085] In order to enable those skilled in the art to better understand the technical solutions in this specification, the technical solutions in the embodiments of this specification will be clearly and completely described below in conjunction with the drawings in some embodiments of this specification. Obviously, the described embodiments are only part of the embodiments of this specification, not all of the embodiments. Based on some embodiments in this specification, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of this specification.

[0086] It should be noted that the terms "first", "second", etc. in the specification and claims of this document and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of this document described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, device, product or equipment that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or equipment. It should be noted that the acquisition, storage, use, processing, etc. of data in the technical solution of this application comply with the relevant provisions of relevant laws and regulations.

[0087] like Figure 1 The diagram is a schematic diagram of an implementation system of a strategy generation method based on multi-objective reinforcement learning according to an embodiment of the present invention, which may include: a terminal 101 and a server 102, wherein the terminal 101 and the server 102 communicate with each other through a network, and the network may include a local area network (LAN), a wide area network (WAN), the Internet, or a combination thereof, and is connected to a website, a user device (such as a computing device), and a back-end system. A staff member may send a strategy generation request to the server 102 through the terminal 101, and after receiving the strategy generation request, the server 102 calls the data in the database for calculation and processing to obtain a strategy generation result, and sends the strategy generation result to the terminal 101, so that the staff member processes the business according to the strategy generation result.

[0088] In the embodiments of this specification, the server 102 can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDN, Content Delivery Network), and big data and artificial intelligence platforms.

[0089] In an optional embodiment, the terminal 101 may include but is not limited to electronic devices such as self-service terminal devices, desktop computers, tablet computers, and laptop computers. Optionally, the operating system running on the electronic device may include but is not limited to Android system, IOS system, Linux, Windows, etc. Of course, the terminal 101 is not limited to the above-mentioned electronic devices with a certain entity, and it can also be software running in the above-mentioned electronic devices.

[0090] In addition, it should be noted that Figure 1 What is shown is only an application environment provided by the present disclosure. In actual application, multiple terminals 101 may be included, and this specification does not limit this.

[0091] Figure 2 It is a flowchart of a strategy generation method based on multi-objective reinforcement learning provided by an embodiment of the present invention. This specification provides method operation steps as described in the embodiment or flowchart, but more or fewer operation steps may be included based on conventional or non-creative labor. The order of steps listed in the embodiment is only one way of executing the steps among many orders, and does not represent the only order of execution. When the system or device product is executed in practice, it can be executed in the order of the method shown in the embodiment or the accompanying drawings or in parallel. Specifically, Figure 2 As shown, applied to the above-mentioned server side, the method may include:

[0092] S201: receiving a multi-objective strategy optimization request;

[0093] Different from single-objective optimization scenarios, the optimization processes of various objectives in multi-objective optimization decision scenarios influence or even conflict with each other. Therefore, multi-objective optimization decision scenarios aim to solve the problem of trade-offs between multiple objectives, such as the optimization of safety, speed, and comfort of autonomous driving vehicles. The traditional multi-objective optimization method is to make multi-objective optimization decisions through predefined multi-objective reward functions. However, in actual complex control scenarios, predefined multi-objective reward functions often lack accuracy and flexibility, which makes it difficult to accurately and effectively make effective decisions in complex control scenarios.

[0094] S202: constructing multi-objective feedback indicator data according to a plurality of state-action sequence fragments corresponding to the multi-objective strategy optimization request, multi-objective weight features, and a binary evaluator for evaluating the state-action sequence fragments;

[0095] When constructing multi-objective feedback index data, each state-action sequence fragment, multi-objective weight feature, and binary evaluator for evaluating the state-action sequence fragment can be used as tuple elements to construct the multi-objective feedback index data according to the tuple elements. Specifically, the multi-objective feedback index data can be represented as a four-tuple (σ0, σ1, w, p), where σ0 and σ1 both represent state-action sequence fragments, and the forms of σ0 and σ1 can be expressed as (s0, a0, …, s H-1 ,a H-1), w represents the multi-objective weight feature, and p represents the binary evaluator, which can be understood as a binary scalar p∈{0,1}, which is used to indicate which state-action sequence fragment is more correlated with the multi-objective weight feature.

[0096] S203: Establishing an explicit multi-objective reward model;

[0097] The explicit multi-objective reward model can be defined as in, represents the multi-objective reward model, S represents the state space, A represents the action space, Represents a vector space of a multi-objective reward model, and each dimension of the multi-objective reward model corresponds to a different objective.

[0098] S204: Training the multi-objective reward model using the multi-objective feedback indicator data;

[0099] After the multi-objective reward model is established, it is necessary to use the multi-objective feedback indicator data to train the multi-objective reward model so that the multi-objective reward model can output a reward signal that meets the requirements of the multi-objective feedback indicator data. Figure 3 , using the multi-objective feedback indicator data to train the multi-objective reward model may include:

[0100] S301: Selecting a state and an action at a specified time from the state-action sequence fragments of the multi-objective feedback indicator data;

[0101] S302: Inputting the state and action at a specified time into the multi-objective reward model to obtain a corresponding predicted reward;

[0102] S303: Establishing a feedback indicator predictor based on the correlation between the predicted reward and the multi-objective weight feature in the multi-objective feedback indicator data;

[0103] S304: Inputting the predicted reward and the multi-objective weight feature in the multi-objective feedback indicator data into the feedback indicator predictor to obtain a feedback indicator prediction result;

[0104] S305: Calculate the loss corresponding to the feedback indicator prediction result according to a preset loss function, and train the feedback indicator predictor using a parameter optimization algorithm to update the model parameters in the multi-objective reward model.

[0105] The feedback indicator predictor is established in the following manner:

[0106]

[0107] Among them, P ψ [σ l >σk |w] represents the feedback indicator predictor, > represents after, σ l and σ k All represent fragments of state-action sequences, w represents multi-objective weight features, exp represents an exponential function with the natural constant e as the base, {l,k} represents a set of binary scalar categories, i represents an element in the set of binary scalar categories, It means that at time t and when the binary scalar category is i, the state s it and action a it The corresponding prediction reward, T Indicates transpose.

[0108] The default loss function is established as follows:

[0109]

[0110] Among them, L p represents the loss function, represents the expectation, (σ0,σ1,w,p) represents the multi-objective feedback indicator data, D represents the multi-objective feedback indicator dataset, P ψ [σ0>σ1|w] and P ψ [σ1>σ0|w] represents the feedback indicator prediction results output by the feedback indicator predictor, and p(0) and p(1) represent two categories of binary scalars.

[0111] It can be understood that when training a multi-objective reward model, first select the state and action data at a certain moment from the state-action sequence fragment, and the moment can be pre-specified, that is, the above-mentioned specified moment, and then input the state and action at the specified moment into the multi-objective reward model, and the corresponding predicted reward can be obtained, and the feedback index predictor is used to evaluate the correlation between the corresponding predicted reward and the multi-objective weight feature in the multi-objective feedback index data to determine whether the current predicted reward meets the requirements of the multi-objective weight feature in the multi-objective feedback index data, and establish a loss function according to the output result of the feedback index predictor (i.e., the feedback index prediction result) and calculate the corresponding loss, which can be used to measure the degree of correlation between the current predicted reward and the multi-objective weight feature in the multi-objective feedback index data, and train the feedback index predictor using a parameter optimization algorithm. Since the feedback index predictor is established based on the multi-objective reward model, while training the feedback index predictor, the multi-objective reward model is also updated and iterated, thereby updating the model parameters in the multi-objective reward model. In some embodiments, the parameter optimization algorithm can be a gradient descent / ascent method, etc., which is not limited in this article.

[0112] S205: Establishing a Markov decision process model according to the states, actions and the trained multi-objective reward model corresponding to the plurality of state-action sequence fragments;

[0113] The traditional Markov decision process is constructed based on (s, a, s′) and can be expressed as M =<S,A,P,R,γ> , where S represents the state space, A represents the action space, P, namely P(s′|s,a), represents the probability of transferring from the current state s to the next state s′ through the current action a, R represents the reward function, γ is the discount factor, and in the embodiment of this specification, the Markov decision process model is constructed based on (s,a,s′,w), and R is expanded to R(s,a), which is a vector function that maps the state and action to The above contains real-time rewards for m different goals. Specifically, the current state, current action and next state can be selected from the states and actions corresponding to several state-action sequence fragments, and the current state and current action are input into the trained multi-objective reward model to obtain the current multi-objective reward R(s,a). Then, according to the current state, current action, current multi-objective reward and next state, a state transition tuple (s,a,s′,w) is constructed to establish a Markov decision process model. Furthermore, the goal of the agent is to find an optimized strategy a=π(s|w), where the multi-objective weight feature w is used to reflect the relative importance of different goals. The strategy π(s|w) selects actions according to the multi-objective weight feature w to maximize the expected value of the weighted return.

[0114] S206: According to the Markov decision process model, in the learning environment corresponding to the multi-objective strategy optimization request, a multi-objective reinforcement learning algorithm is used to train to obtain a multi-objective strategy.

[0115] Specifically, based on the determined Markov decision process model, a multi-objective reinforcement learning algorithm can be used to train a multi-objective strategy in a learning environment corresponding to a multi-objective strategy optimization request. During the training process, the trained multi-objective reward model is used to predict the multi-objective reward in the state transfer data, and the multi-objective reward model supports updating with the change of the multi-objective feedback indicator data, so that the final strategy always meets the requirements of the multi-objective feedback indicator data. Figure 4 , the specific training process can be:

[0116] Step 0: Initialize network parameters;

[0117] Step 1: Get the current state and determine whether the current number of steps exceeds the preset threshold;

[0118] Step 2: If the preset threshold is not exceeded, the current action is obtained by random sampling; if the preset threshold is not exceeded, the current action is obtained by sampling based on the control strategy;

[0119] Step 3: Obtain a state transition tuple consisting of the current state, current action, next state, and multi-objective weight features in the multi-objective feedback indicator data;

[0120] Step 4: Call the corresponding trained multi-objective reward model according to the state transition tuple, and input the current state and the current action into the trained multi-objective reward model to obtain the current multi-objective reward;

[0121] Step 5: Add the state transition data consisting of the current state, current action, current multi-target reward and next state to the replay buffer of this iteration; the replay buffer is used to store the state transition data required for each iteration;

[0122] Step 6: Partial batch sampling is performed on the state transition data in the replay buffer of this iteration to obtain sampled batch data, and based on the sampled batch data, multi-objective reinforcement learning is performed using the reinforcement learning loss function to update the network parameters;

[0123] Repeat steps 1 to 6 above until the preset number of iterations is reached.

[0124] Furthermore, in some embodiments, when the multi-objective feedback indicator data changes, the trained multi-objective reward model needs to be updated synchronously to ensure the accuracy of the output results of the multi-objective reward model. At this time, the change of the multi-objective feedback indicator data can be monitored regularly by using a preset update frequency. Figure 5 , updating the trained multi-objective reward model to update the multi-objective reward data in the replay buffer may further include:

[0125] S501: Obtaining a preset update frequency of multi-objective feedback indicator data;

[0126] S502: As the number of iterations increases, when the relationship between the preset update frequency and the current number of iterations satisfies a preset condition, a preset number of state-action sequence segments are sampled from the replay buffer, and a preset number of multi-objective weight features are sampled from a preset data set;

[0127] S503: Generate new multi-objective feedback index data according to the sampled multi-objective weight features, state-action sequence fragments and the binary evaluator;

[0128] S504: adding the new multi-objective feedback indicator data to the feedback indicator data set;

[0129] S505: Using the data in the feedback indicator data set, updating the trained multi-objective reward model according to a preset loss function;

[0130] S506: Using the updated multi-objective reward model, update the replay buffer.

[0131] Specifically, according to the Markov decision process model, in the learning environment corresponding to the multi-objective strategy optimization request, the detailed process of using the multi-objective reinforcement learning algorithm to train the multi-objective strategy can be referred to as shown in Algorithm 1 below.

[0132]

[0133] It can be understood that the embodiment of the present specification first automatically receives a multi-objective strategy optimization request, and obtains several state-action sequence fragments, multi-objective weight features and a binary evaluator for evaluating the state-action sequence fragments corresponding to the multi-objective strategy optimization request to construct multi-objective feedback index data, and then establishes an explicit multi-objective reward model, and uses the multi-objective feedback index data to train the multi-objective reward model so that the trained multi-objective reward model can be used to characterize the strategy generation tendency corresponding to the multi-objective feedback index data. Then, according to the states and actions corresponding to the several state-action sequence fragments and the trained multi-objective reward model, a Markov decision process model is established, and a multi-objective reinforcement learning algorithm is applied for training to achieve efficient and accurate output of multi-objective strategies in complex multi-objective scenarios.

[0134] To facilitate understanding by those skilled in the art, this specification provides a typical multi-objective strategy optimization scenario 1. The task scenario is to train a robot dog to imitate the behavior of a real dog. The corresponding multiple training goals may include the robot dog running forward T1, the robot dog jumping T2, the coordination of the robot dog's actions T3, and the robot dog ensuring its own equipment safety while performing actions T4. The state-action sequence fragments (i.e., trajectories) are: σ1: the robot dog stays still, σ2: the robot dog runs forward in an unstable state (for example, staggering forward), σ3: the robot dog runs forward quickly and the movement amplitude is large, which may cause equipment safety problems, σ4: the robot dog runs forward quickly, jumping while running. Afterwards, a comprehensive evaluation analysis method (such as hierarchical analysis method, grey correlation analysis method, etc.) is used to obtain the multi-objective weight feature w corresponding to the current task scenario. For example, when w = [1, 0, 0, 0], it means that the robot dog running forward T1 is important at this time, and other goals are not concerned. Compared with the trajectory (σ1, σ2), since σ1 does not move in place and σ2 runs forward, the evaluation result given by the binary evaluator is p = (0, 1); when w = [0.5, 0.5, 0, 0], it means that the robot dog running forward T1 and the robot dog jumping T2 are equally important, and other goals are not concerned. Compared with the trajectory (σ2, σ4), the binary evaluator gives an evaluation result p = (0,1), because trajectory σ4 has both running and jumping states, while σ2 only has a running state; when w = [0.3, 0, 0.7, 0], it means that compared with the training goal T1 of the robot dog running forward, more attention is paid to the coordination of the robot dog's movements T3, and other goals are not paid attention to. The comparison trajectory is (σ1, σ2), and the feedback index p = (1, 0). The comparison trajectory (σ1, σ2) shows that σ2 has a forward running action but the action is not coordinated, while σ1 does not move (there is no forward running action), but there is no coordination problem, so p = (1, 0). In this way, in the scenario of training a robot dog to imitate the behavior of a real dog, the multi-objective feedback index data can be quickly constructed according to the multi-objective strategy optimization request corresponding to several state-action sequence fragments, multi-objective weight features, and the binary evaluator used to evaluate the state-action sequence fragments. It should be noted that, when the binary evaluator is actually used, it can provide evaluation results of two or more state-action sequence fragments by comparing them pairwise, and this article does not limit this.

[0135] To facilitate understanding by those skilled in the art, this specification also provides a typical multi-objective strategy optimization scenario 2, where the task scenario is to train a multi-objective autonomous driving system, and multiple training objectives are: the speed of vehicle movement G1; the smoothness of vehicle driving G2 (for example, no expectation of stopping and waiting); the safety of vehicle driving G3 (for example, speeding is not allowed, a certain distance from pedestrians needs to be maintained, etc.); the comfort level of passengers G4 (for example, braking and throttle with excessive frequency / amplitude are not expected). Accordingly, the state-action sequence fragments (i.e., trajectories) are: β1: the vehicle is stationary, β2: the vehicle is driving normally on an ordinary road, β3: the vehicle is traveling in a downtown area with frequent lane changes, and β4: the vehicle is driving slowly in a downtown area, at a speed slower than β3, with frequent braking. When w=[1,0,0,0], and the comparison trajectory is (σ1,σ2), since the vehicle speed G1 is of interest at this time, σ1 is stationary, and σ2 is moving forward, p=(0,1); when w=[0.3,0.7,0,0], and the comparison trajectory is (σ3,σ4), since the vehicle driving smoothness G2 is more of an issue than the vehicle speed G1, and other targets are ignored, σ4 is slow and brakes frequently, and σ3 is faster than σ4 and drives smoothly, so p=(1,0). In the scenario of training a multi-objective autonomous driving system, the multi-objective feedback index data can be quickly constructed according to the multi-objective strategy optimization request corresponding to several state-action sequence fragments, multi-objective weight features, and the binary evaluator used to evaluate the state-action sequence fragments, thereby ensuring the effectiveness of the subsequent training of the multi-objective reward model and the multi-objective reinforcement learning training process.

[0136] It should be noted that, although the operations of the method of the present invention are described in a specific order in the above embodiments and the accompanying drawings, this does not require or imply that the operations must be performed in the specific order, or that all the operations shown must be performed to achieve the desired results. Additionally or alternatively, some steps may be omitted, multiple steps may be combined into one step, and / or one step may be decomposed into multiple steps.

[0137] Corresponding to the above-mentioned strategy generation method based on multi-objective reinforcement learning, some embodiments of this specification also provide a strategy generation device based on multi-objective reinforcement learning, referring to Figure 6 As shown, in some embodiments, the apparatus may include:

[0138] Receiving module 601, used for receiving a multi-objective strategy optimization request;

[0139] A feedback indicator data construction module 602 is used to construct multi-objective feedback indicator data according to a plurality of state-action sequence fragments corresponding to the multi-objective strategy optimization request, multi-objective weight features, and a binary evaluator for evaluating the state-action sequence fragments;

[0140] A multi-objective reward model building module 603, used to build an explicit multi-objective reward model;

[0141] A multi-objective reward model training module 604 is used to train the multi-objective reward model using the multi-objective feedback indicator data;

[0142] A Markov decision process model building module 605 is used to build a Markov decision process model according to the states, actions and the trained multi-objective reward model corresponding to the plurality of state-action sequence fragments;

[0143] The multi-objective reinforcement learning module 606 is used to train a multi-objective strategy using a multi-objective reinforcement learning algorithm according to the Markov decision process model in a learning environment corresponding to the multi-objective strategy optimization request.

[0144] For the convenience of description, the above device is described in various units according to their functions. Of course, when implementing this specification, the functions of each unit can be implemented in the same or multiple software and / or hardware.

[0145] It should be noted that in the embodiments of this specification, the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved are all information and data authorized by the user and fully authorized by all parties.

[0146] It should be noted that the computer program product described in this specification is a software product that mainly implements the method described in this specification through a computer program.

[0147] The embodiments of this specification also provide a computer device. Figure 7As shown, in some embodiments of the present specification, the computer device 702 may include one or more processors 704, such as one or more central processing units (CPUs) or graphics processing units (GPUs), and each processing unit may implement one or more hardware threads. The computer device 702 may also include any memory 706, which is used to store any kind of information such as code, settings, data, etc. In a specific embodiment, a computer program on the memory 706 and that can be run on the processor 704, when the computer program is run by the processor 704, the instructions of the method described in any of the above embodiments may be executed. Non-limitingly, for example, the memory 706 may include any one or more combinations of the following: any type of RAM, any type of ROM, flash memory device, hard disk, optical disk, etc. More generally, any memory may use any technology to store information. Further, any memory may provide volatile or non-volatile retention of information. Further, any memory may represent a fixed or removable component of the computer device 702. In one case, when the processor 704 executes an associated instruction stored in any memory or a combination of memories, the computer device 702 may perform any operation of the associated instruction. The computer device 702 also includes one or more drive mechanisms 708 for interacting with any storage, such as a hard disk drive mechanism, an optical disk drive mechanism, etc.

[0148] The computer device 702 may also include an input / output interface 710 (I / O) for receiving various inputs (via input device 712) and for providing various outputs (via output device 714). A specific output mechanism may include a presentation device 716 and an associated graphical user interface 718 (GUI). In other embodiments, the input / output interface 710 (I / O), input device 712, and output device 714 may not be included, and the computer device 702 may be used as a computer device in a network. The computer device 702 may also include one or more network interfaces 720 for exchanging data with other devices via one or more communication links 722. One or more communication buses 724 couple the components described above together.

[0149] The communication link 722 may be implemented in any manner, for example, through a local area network, a wide area network (e.g., the Internet), a point-to-point connection, etc., or any combination thereof. The communication link 722 may include any combination of hardwired links, wireless links, routers, gateway functions, name servers, etc. governed by any protocol or combination of protocols.

[0150] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), computer-readable storage media, and computer program products of some embodiments of the present specification. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processor to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processor generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0151] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processor to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0152] These computer program instructions can also be loaded onto a computer or other programmable data processor so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for executing on the computer or other programmable device to implement the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0153] In a typical configuration, a computer device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0154] The memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0155] Computer-readable media include permanent and non-permanent, removable and non-removable media that can be used to store information by any method or technology. Information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computer device. As defined in this specification, computer-readable media does not include temporary computer-readable media (transitory media), such as modulated data signals and carrier waves.

[0156] Those skilled in the art will appreciate that the embodiments of this specification may be provided as methods, systems or computer program products. Therefore, the embodiments of this specification may take the form of complete hardware embodiments, complete software embodiments or embodiments combining software and hardware. Moreover, the embodiments of this specification may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0157] The present specification embodiments may be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present specification embodiments may also be practiced in distributed computing environments where tasks are performed by remote processors connected through a communication network. In a distributed computing environment, program modules may be located in local and remote computer storage media, including storage devices.

[0158] It should also be understood that in the embodiments of this specification, the term "and / or" is only a description of the association relationship of the associated objects, indicating that three relationships can exist. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship.

[0159] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0160] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the embodiments of this specification. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, without contradiction.

[0161] The above is only an embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the scope of the claims of the present application.

Claims

1. A strategy generation method based on multi-objective reinforcement learning, characterized in that: The method comprises: Receiving a multi-objective strategy optimization request; Constructing multi-objective feedback indicator data according to a plurality of state-action sequence fragments corresponding to the multi-objective strategy optimization request, multi-objective weight features, and a binary evaluator for evaluating the state-action sequence fragments; Building explicit multi-objective reward models; Training the multi-objective reward model using the multi-objective feedback indicator data; Establishing a Markov decision process model according to the states, actions and the trained multi-objective reward model corresponding to the plurality of state-action sequence fragments; According to the Markov decision process model, in the learning environment corresponding to the multi-objective strategy optimization request, a multi-objective reinforcement learning algorithm is used to train and obtain a multi-objective strategy.

2. The method according to claim 1, characterized in that According to a plurality of state-action sequence fragments corresponding to the multi-objective strategy optimization request, a multi-objective weight feature, and a binary evaluator for evaluating the state-action sequence fragments, multi-objective feedback indicator data is constructed, including: Each state-action sequence fragment, the multi-objective weight feature, and the binary evaluator used to evaluate the state-action sequence fragment are respectively used as tuple elements, so as to construct multi-objective feedback indicator data according to the tuple elements.

3. The method according to claim 2, characterized in that Training the multi-objective reward model using the multi-objective feedback indicator data includes: Selecting the state and action at a specified time from the state-action sequence fragments of the multi-objective feedback indicator data; Inputting the state and action at a specified time into the multi-objective reward model to obtain the corresponding predicted reward; Establishing a feedback indicator predictor based on the correlation between the predicted reward and the multi-objective weight feature in the multi-objective feedback indicator data; Inputting the predicted reward and the multi-objective weight features in the multi-objective feedback indicator data into the feedback indicator predictor to obtain a feedback indicator prediction result; The loss corresponding to the feedback indicator prediction result is calculated according to a preset loss function, and the feedback indicator predictor is trained using a parameter optimization algorithm to update the model parameters in the multi-objective reward model.

4. The method according to claim 3, characterized in that The feedback indicator predictor is established in the following way: Among them, P ψ [σ l >σ k |w] represents the feedback indicator predictor, > represents after, σ l and σ k All represent fragments of state-action sequences, w represents multi-objective weight features, exp represents an exponential function with the natural constant e as the base, {l,k} represents a set of binary scalar categories, i represents an element in the set of binary scalar categories, It means that at time t and when the binary scalar category is i, the state s it and action a it The corresponding prediction reward, T Indicates transpose.

5. The method according to claim 1, characterized in that According to the states, actions and the trained multi-objective reward model corresponding to the plurality of state-action sequence fragments, a Markov decision process model is established, including: Determine the current state, current action and next state according to the states and actions corresponding to the plurality of state-action sequence fragments; Input the current state and current action into the trained multi-objective reward model to obtain the current multi-objective reward; According to the current state, current action, current multi-objective reward and next state, state transition data is constructed to establish a Markov decision process model.

6. The method according to claim 1, characterized in that According to the Markov decision process model, in the learning environment corresponding to the multi-objective strategy optimization request, a multi-objective reinforcement learning algorithm is used to train a multi-objective strategy, including: Step 0: Initialize network parameters; Step 1: Get the current state and determine whether the current number of steps exceeds the preset threshold; Step 2: If the preset threshold is not exceeded, the current action is obtained by random sampling; if the preset threshold is not exceeded, the current action is obtained by sampling based on the control strategy; Step 3: Obtain a state transition tuple consisting of the current state, current action, next state, and multi-objective weight features in the multi-objective feedback indicator data; Step 4: Call the corresponding trained multi-objective reward model according to the state transition tuple, and input the current state and the current action into the trained multi-objective reward model to obtain the current multi-objective reward; Step 5: Add the state transition data consisting of the current state, current action, current multi-target reward and next state to the replay buffer of this iteration; the replay buffer is used to store the state transition data required for each iteration; Step 6: Partial batch sampling is performed on the state transition data in the replay buffer of this iteration to obtain sampled batch data, and based on the sampled batch data, multi-objective reinforcement learning is performed using the reinforcement learning loss function to update the network parameters; Repeat steps 1 to 6 above until the preset number of iterations is reached.

7. The method according to claim 6, characterized in that Further including: Get the preset update frequency of multi-objective feedback indicator data; As the number of iterations increases, when the relationship between the preset update frequency and the current number of iterations satisfies a preset condition, a preset number of state-action sequence segments are sampled from the replay buffer, and a preset number of multi-objective weight features are sampled from a preset data set; Generate new multi-objective feedback indicator data based on the sampled multi-objective weight features, state-action sequence fragments and the binary evaluator; Adding the new multi-objective feedback indicator data to the feedback indicator data set; Using the data in the feedback indicator data set, updating the trained multi-objective reward model according to a preset loss function; The replay buffer is updated using the updated multi-objective reward model.

8. A strategy generation device based on multi-objective reinforcement learning, characterized in that: The device comprises: A receiving module, used for receiving a multi-objective strategy optimization request; A feedback indicator data construction module, used to construct multi-objective feedback indicator data according to a plurality of state-action sequence fragments corresponding to the multi-objective strategy optimization request, multi-objective weight features, and a binary evaluator for evaluating the state-action sequence fragments; Multi-objective reward model building module, used to build an explicit multi-objective reward model; A multi-objective reward model training module, used for training the multi-objective reward model using the multi-objective feedback indicator data; A Markov decision process model building module is used to build a Markov decision process model according to the states, actions and the trained multi-objective reward model corresponding to the plurality of state-action sequence fragments; The multi-objective reinforcement learning module is used to train a multi-objective strategy using a multi-objective reinforcement learning algorithm according to the Markov decision process model in a learning environment corresponding to the multi-objective strategy optimization request.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory, characterized in that: When the computer program is executed by the processor, the computer program executes the instructions of the method according to any one of claims 1 to 7.

10. A computer storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor of a computer device, the computer program executes instructions of the method according to any one of claims 1 to 7.

Images