A strategy generation method and device based on multi-objective reinforcement learning

Abstract

The present specification relates to the technical field of reinforcement learning, and provides a strategy generation method and device based on multi-objective reinforcement learning. The method comprises: receiving a multi-objective strategy optimization request; constructing multi-objective feedback index data according to a plurality of state-action sequence fragments corresponding to the multi-objective strategy optimization request, multi-objective weight features, and a binary evaluator used to evaluate the state-action sequence fragments; establishing an explicit multi-objective reward model; training the multi-objective reward model using the multi-objective feedback index data; establishing a Markov decision process model according to states and actions corresponding to the plurality of state-action sequence fragments and the trained multi-objective reward model; and training a multi-objective strategy using a multi-objective reinforcement learning algorithm in a learning environment corresponding to the multi-objective strategy optimization request according to the Markov decision process model. The present specification can improve the strategy generation precision based on multi-objective reinforcement learning.

Description

Technical Field

[0001] This specification relates to the field of reinforcement learning technology, and in particular to a policy generation method and apparatus based on multi-objective reinforcement learning. Background Technology

[0002] In the field of multi-objective reinforcement learning, traditional research focuses on guiding policy optimization through predefined reward functions. However, due to the inherent complexity of multi-dimensional optimization objectives and the uncertainty or difficulty in measuring the states of some optimization dimensions, traditional methods face challenges in handling complex real-world objectives. For example, when a robot dog learns a backflip, it is difficult to design an accurate reward function to guide its learning process; data center cooling needs to consider both the risk of IT equipment overheating and energy efficiency, but the internal chip temperature of IT equipment is often difficult to measure directly, making it impossible to accurately measure the optimization objective, i.e., the internal chip temperature. To overcome the limitations caused by the inherent complexity of multi-dimensional optimization objectives, existing technologies utilize human feedback to guide the behavior of reinforcement learning agents, improving the performance and flexibility of single-objective optimization, thereby avoiding the problems of uncertain target values ​​and complex reward engineering.

[0003] However, in terms of multi-objective reinforcement learning modeling, existing technologies are insufficient to solve the policy generation problem in multi-objective reinforcement learning scenarios due to the inherent complexity of multi-dimensional optimization objectives. Therefore, there is an urgent need for a policy generation method based on multi-objective reinforcement learning to build a more flexible reward model in multi-objective reinforcement learning scenarios that addresses the inherent complexity of multi-dimensional optimization objectives, avoids dependence on predefined reward functions, and improves the accuracy of policy generation based on multi-objective reinforcement learning. Summary of the Invention

[0004] Given that current technologies are insufficient to solve the policy generation problem in multi-objective reinforcement learning scenarios due to the inherent complexity of multi-dimensional optimization objectives, this solution is proposed to overcome or at least partially solve the aforementioned problems.

[0005] On the one hand, some embodiments of this specification aim to provide a policy generation method for multi-objective reinforcement learning, the method comprising:

[0006] Receive multi-objective strategy optimization requests;

[0007] Based on the several state-action sequence fragments corresponding to the multi-objective strategy optimization request, the multi-objective weight features, and the binary evaluator used to evaluate the state-action sequence fragments, construct multi-objective feedback index data;

[0008] Establish an explicit multi-objective reward model;

[0009] The multi-objective reward model is trained using the multi-objective feedback index data;

[0010] Based on the states, actions, and trained multi-objective reward models corresponding to the aforementioned state-action sequence segments, a Markov decision process model is established.

[0011] Based on the Markov decision process model, a multi-objective policy is trained using a multi-objective reinforcement learning algorithm in the learning environment corresponding to the multi-objective policy optimization request.

[0012] Further, based on the several state-action sequence fragments corresponding to the multi-objective policy optimization request, the multi-objective weight features, and the binary evaluator used to evaluate the state-action sequence fragments, multi-objective feedback index data is constructed, including:

[0013] Each state-action sequence segment, multi-objective weight features, and a binary evaluator used to evaluate the state-action sequence segment are respectively used as tuple elements to construct multi-objective feedback index data based on the tuple elements.

[0014] Further, training the multi-objective reward model using the multi-objective feedback index data includes:

[0015] Select the state and action at a specified moment from the state-action sequence fragment of the multi-objective feedback index data;

[0016] The state and action at a specified time are input into the multi-objective reward model to obtain the corresponding predicted reward;

[0017] Based on the correlation between the predicted reward and the multi-objective weight features in the multi-objective feedback index data, a feedback index predictor is established.

[0018] The predicted reward and the multi-objective weight features in the multi-objective feedback index data are input into the feedback index predictor to obtain the feedback index prediction result;

[0019] The loss corresponding to the prediction result of the feedback index is calculated according to the preset loss function, and the feedback index predictor is trained using a parameter optimization algorithm to update the model parameters in the multi-objective reward model.

[0020] Furthermore, the feedback indicator predictor is established using the following method:

[0021]

[0022] Among them, P ψ [σ l >σ k |w] represents the feedback indicator predictor, > indicates after, σ land σ k Each represents a segment of a state-action sequence, w represents a multi-objective weighted feature, exp represents an exponential function with the natural constant e as its base, {l,k} represents the set of categories of a binary scalar, and i represents an element in the set of categories of a binary scalar. This represents the state s at time t when the category of the binary scalar is i. it and action a it The corresponding predicted reward, T This indicates transpose.

[0023] Furthermore, based on the states, actions, and trained multi-objective reward model corresponding to the aforementioned state-action sequence segments, a Markov decision process model is established, including:

[0024] The current state, current action, and next state are determined based on the states and actions corresponding to the aforementioned state-action sequence segments.

[0025] The current state and current action are input into the trained multi-objective reward model to obtain the current multi-objective reward;

[0026] Based on the current state, current action, current multi-objective reward, and next state, state transition data is constructed to establish a Markov decision process model.

[0027] Furthermore, based on the Markov decision process model, in the learning environment corresponding to the multi-objective policy optimization request, a multi-objective policy is trained using a multi-objective reinforcement learning algorithm, including:

[0028] Step 0: Initialize network parameters;

[0029] Step 1: Obtain the current state and determine whether the current number of steps exceeds the preset threshold;

[0030] Step 2: If the preset threshold is not exceeded, the current action is obtained by random sampling; if the preset threshold is not exceeded, the current action is obtained by sampling based on the control strategy.

[0031] Step 3: Obtain the state transition tuple consisting of the current state, current action, next state, and multi-objective weight features from the multi-objective feedback index data;

[0032] Step 4: Call the corresponding trained multi-objective reward model according to the state transition tuple, and input the current state and current action into the trained multi-objective reward model to obtain the current multi-objective reward;

[0033] Step 5: Add the state transition data, consisting of the current state, current action, current multi-objective reward, and next state, to the replay buffer for this iteration; the replay buffer is used to store the state transition data required for each iteration;

[0034] Step 6: Perform partial batch sampling of the state transition data in the replay buffer of this iteration to obtain sampled batch data, and based on the sampled batch data, perform multi-objective reinforcement learning using the reinforcement learning loss function to update the network parameters;

[0035] Repeat steps 1-6 above until the preset number of iterations is reached.

[0036] Furthermore, in some embodiments, the method further includes:

[0037] Preset update frequency for acquiring multi-objective feedback indicator data;

[0038] As the number of iterations increases, when the relationship between the preset update frequency and the current number of iterations meets the preset condition, a preset number of state action sequence fragments are sampled from the replay buffer, and a preset number of multi-objective weight features are sampled from the preset dataset.

[0039] Based on the sampled multi-objective weight features, state-action sequence fragments, and the binary evaluator, new multi-objective feedback index data is generated.

[0040] Add the new multi-objective feedback index data to the feedback index dataset;

[0041] Using the data in the feedback index dataset, the trained multi-objective reward model is updated according to a preset loss function;

[0042] The replay buffer is updated using the updated multi-objective reward model.

[0043] On the other hand, some embodiments of this specification also provide a policy generation apparatus based on multi-objective reinforcement learning, the apparatus comprising:

[0044] The receiving module is used to receive multi-objective strategy optimization requests;

[0045] The feedback index data construction module is used to construct multi-objective feedback index data based on several state-action sequence fragments corresponding to the multi-objective strategy optimization request, multi-objective weight features, and a binary evaluator for evaluating the state-action sequence fragments.

[0046] The multi-objective reward model building module is used to build explicit multi-objective reward models;

[0047] A multi-objective reward model training module is used to train the multi-objective reward model using the multi-objective feedback index data;

[0048] The Markov decision process model building module is used to build a Markov decision process model based on the states, actions, and trained multi-objective reward model corresponding to the several state-action sequence segments.

[0049] The multi-objective reinforcement learning module is used to train a multi-objective policy using a multi-objective reinforcement learning algorithm in the learning environment corresponding to the multi-objective policy optimization request, based on the Markov decision process model.

[0050] On the other hand, some embodiments of this specification also provide a computer device, including a memory, a processor, and a computer program stored in the memory, which, when run by the processor, executes instructions for the methods described above.

[0051] On the other hand, some embodiments of this specification also provide a computer storage medium having a computer program stored thereon, which, when run by the processor of a computer device, executes instructions for the methods described above.

[0052] On the other hand, some embodiments of this specification also provide a computer program product, which includes a computer program that, when run by the processor of a computer device, executes instructions for the methods described above.

[0053] Some embodiments of this specification provide one or more technical solutions, which have at least the following technical effects:

[0054] The embodiments of this specification first automatically receive multi-objective policy optimization requests and obtain several state-action sequence fragments, multi-objective weight features, and a binary evaluator for evaluating the state-action sequence fragments corresponding to the multi-objective policy optimization requests, so as to construct multi-objective feedback index data. Then, an explicit multi-objective reward model is established, and the multi-objective reward model is trained using the multi-objective feedback index data so that the trained multi-objective reward model can be used to characterize the policy generation tendency corresponding to the multi-objective feedback index data. Then, based on the states and actions corresponding to the several state-action sequence fragments and the trained multi-objective reward model, a Markov decision process model is established, and a multi-objective reinforcement learning algorithm is applied for training to achieve efficient and accurate output of multi-objective policies in complex multi-objective scenarios.

[0055] The above description is merely an overview of some embodiments of the technical solutions in this specification. In order to better understand the technical means of some embodiments of this specification and to implement them in accordance with the content of the specification, and to make the above and other objects, features and advantages of some embodiments of this specification more apparent and understandable, specific implementation methods of some embodiments of this specification are given below. Attached Figure Description

[0056] To more clearly illustrate some embodiments or technical solutions in the prior art of this specification, the accompanying drawings used in the description of the embodiments or prior art will be briefly introduced below. Obviously, the drawings described below are merely some embodiments recorded in this specification. For those skilled in the art, other drawings can be obtained based on these drawings without any creative effort. In the drawings:

[0057] Figure 1 A schematic diagram of an implementation system for a policy generation method based on multi-objective reinforcement learning, as illustrated in some embodiments of this specification, is shown.

[0058] Figure 2 A flowchart of a policy generation method based on multi-objective reinforcement learning is shown in some embodiments of this specification;

[0059] Figure 3 This is a schematic diagram illustrating the steps of training a multi-objective reward model using multi-objective feedback index data in some embodiments of this specification;

[0060] Figure 4 This is a schematic diagram illustrating the steps of training a multi-objective policy using a multi-objective reinforcement learning algorithm in some embodiments of this specification;

[0061] Figure 5 This is a schematic diagram illustrating the steps of updating the replay buffer using the updated multi-objective reward model in some embodiments of this specification;

[0062] Figure 6 This is a schematic diagram of the structure of a policy generation device based on multi-objective reinforcement learning in some embodiments of this specification;

[0063] Figure 7 This is a schematic diagram of the computer device structure provided in some embodiments of this specification.

[0064] [Explanation of Labels in the Attached Image]

[0065] 101. Terminal;

[0066] 102. Server;

[0067] 601. Receiver module;

[0068] 602. Feedback indicator data construction module;

[0069] 603. Multi-objective reward model establishment module;

[0070] 604. Multi-objective reward model training module;

[0071] 605. Markov Decision Process Model Building Module;

[0072] 606. Multi-objective reinforcement learning module;

[0073] 702. Computer equipment;

[0074] 704, Processor;

[0075] 706. Memory;

[0076] 708. Drive mechanism;

[0077] 710. Input / output interfaces;

[0078] 712. Input devices;

[0079] 714. Output devices;

[0080] 716. Presentation equipment;

[0081] 718. Graphical User Interface;

[0082] 720. Network interface;

[0083] 722. Communication link;

[0084] 724. Communication bus. Detailed Implementation

[0085] To enable those skilled in the art to better understand the technical solutions in this specification, the technical solutions in some embodiments of this specification will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this specification, and not all embodiments. Based on some embodiments of this specification, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of this specification.

[0086] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings herein are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such use of data can be interchanged where appropriate so that the embodiments described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, apparatus, product, or device that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or devices. It should be noted that the acquisition, storage, use, and processing of data in the technical solutions of this application comply with the relevant provisions of applicable laws and regulations.

[0087] like Figure 1 The diagram illustrates an implementation system for a policy generation method based on multi-objective reinforcement learning according to an embodiment of the present invention. The system may include a terminal 101 and a server 102. The terminal 101 and server 102 communicate via a network, which may include a Local Area Network (LAN), a Wide Area Network (WAN), the Internet, or a combination thereof, and is connected to a website, user equipment (e.g., computing devices), and a backend system. Staff can send policy generation requests to the server 102 via the terminal 101. Upon receiving the request, the server 102 retrieves data from its database for calculation and processing to obtain the policy generation result, which is then sent back to the terminal 101 so that staff can process business operations based on the generated policy.

[0088] In the embodiments of this specification, the server 102 may be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDN), and big data and artificial intelligence platforms.

[0089] In an optional embodiment, terminal 101 may be an electronic device, including but not limited to self-service terminal equipment, desktop computers, tablet computers, laptop computers, etc. Optionally, the operating system running on the electronic device may include, but is not limited to, Android, iOS, Linux, Windows, etc. Of course, terminal 101 is not limited to the aforementioned physical electronic devices; it may also be software running on the aforementioned electronic devices.

[0090] In addition, it should be noted that, Figure 1 The example shown is merely one application environment provided by this disclosure. In practical applications, it may include multiple terminals 101, and this specification does not impose any restrictions.

[0091] Figure 2 This is a flowchart illustrating a policy generation method based on multi-objective reinforcement learning, as provided in an embodiment of the present invention. This specification provides the operational steps of the method described in the embodiments or flowchart, but based on conventional or non-inventive methods, more or fewer operational steps may be included. The order of steps listed in the embodiments is merely one possible execution order among many and does not represent the only possible execution order. In actual system or device products, the methods shown in the embodiments or accompanying drawings can be executed sequentially or in parallel. Specifically, as shown... Figure 2 As shown, when applied to the server side described above, the method may include:

[0092] S201: Receive multi-objective strategy optimization request;

[0093] Unlike single-objective optimization scenarios, the optimization processes of each objective in multi-objective optimization decision-making scenarios are mutually influential and even conflicting. Therefore, multi-objective optimization decision-making scenarios aim to solve the problem of trade-offs among multiple objectives, such as the optimization of safety, speed, and comfort in autonomous vehicles. Traditional multi-objective optimization methods use predefined multi-objective reward functions to make multi-objective optimization decisions. However, in actual complex control scenarios, predefined multi-objective reward functions often lack accuracy and flexibility, making it difficult to provide effective decisions accurately and efficiently in complex control scenarios.

[0094] S202: Construct multi-objective feedback index data based on several state-action sequence fragments corresponding to the multi-objective strategy optimization request, multi-objective weight features, and a binary evaluator for evaluating the state-action sequence fragments;

[0095] When constructing multi-objective feedback index data, each state-action sequence segment, multi-objective weight features, and a binary evaluator used to evaluate the state-action sequence segment can be used as tuple elements to construct the multi-objective feedback index data. Specifically, the multi-objective feedback index data can be represented as a quadruple (σ0, σ1, w, p), where σ0 and σ1 both represent state-action sequence segments, and both σ0 and σ1 can be represented as (s0, a0, ..., s H-1 ,a H-1), where w represents the multi-objective weight feature, and p represents the binary evaluator, which can be understood as a binary scalar p∈{0,1}, used to indicate which state-action sequence segment is more correlated with the multi-objective weight feature.

[0096] S203: Establish an explicit multi-objective reward model;

[0097] An explicit multi-objective reward model can be defined as follows: in, Let S represent a multi-objective reward model, where S represents the state space and A represents the action space. Let represent the vector space of the multi-objective reward model, where each dimension of the multi-objective reward model corresponds to a different objective.

[0098] S204: Train the multi-objective reward model using the multi-objective feedback index data;

[0099] After establishing the multi-objective reward model, it needs to be trained using multi-objective feedback index data to ensure that the model outputs a reward signal that meets the requirements of the multi-objective feedback index data. (See Appendix...) Figure 3 Training the multi-objective reward model using the multi-objective feedback index data may include:

[0100] S301: Select the state and action at a specified time from the state-action sequence fragment of the multi-target feedback index data;

[0101] S302: Input the state and action at a specified time into the multi-objective reward model to obtain the corresponding predicted reward;

[0102] S303: Based on the correlation between the predicted reward and the multi-objective weight features in the multi-objective feedback index data, establish a feedback index predictor;

[0103] S304: Input the predicted reward and the multi-objective weight features in the multi-objective feedback index data into the feedback index predictor to obtain the feedback index prediction result;

[0104] S305: Calculate the loss corresponding to the prediction result of the feedback index according to the preset loss function, and train the feedback index predictor using the parameter optimization algorithm to update the model parameters in the multi-objective reward model.

[0105] The feedback indicator predictor is established using the following method:

[0106]

[0107] Among them, P ψ [σ l >σk |w] represents the feedback indicator predictor, > indicates after, σ l and σ k Each represents a segment of a state-action sequence, w represents a multi-objective weighted feature, exp represents an exponential function with the natural constant e as its base, {l,k} represents the set of categories of a binary scalar, and i represents an element in the set of categories of a binary scalar. This represents the state s at time t when the category of the binary scalar is i. it and action a it The corresponding predicted reward, T This indicates transpose.

[0108] The preset loss function is established as follows:

[0109]

[0110] Among them, L p Represents the loss function. Let (σ0,σ1,w,p) represent the expected value, (σ0,σ1,w,p) represent the multi-objective feedback index data, D represent the multi-objective feedback index dataset, and P represent the expected value. ψ [σ0>σ1|w] and P ψ [σ1>σ0|w] both represent the feedback index prediction results output by the feedback index predictor, and p(0) and p(1) represent the two categories of the binary scalar.

[0111] This can be understood as follows: When training a multi-objective reward model, firstly, state and action data at a specific moment are selected from the state-action sequence fragment. This moment can be pre-specified, as mentioned above. Then, the state and action at the specified moment are input into the multi-objective reward model to obtain the corresponding predicted reward. A feedback indicator predictor is used to evaluate the correlation between the predicted reward and the multi-objective weight features in the multi-objective feedback indicator data to determine whether the current predicted reward meets the requirements of the multi-objective weight features in the multi-objective feedback indicator data. A loss function is established based on the output of the feedback indicator predictor (i.e., the feedback indicator prediction result), and the corresponding loss is calculated. This loss can be used to measure the degree of correlation between the current predicted reward and the multi-objective weight features in the multi-objective feedback indicator data. A parameter optimization algorithm is then used to train the feedback indicator predictor. Since the feedback indicator predictor is based on the multi-objective reward model, the multi-objective reward model is also updated and iterated while the feedback indicator predictor is being trained, thereby updating the model parameters in the multi-objective reward model. In some embodiments, the parameter optimization algorithm can be gradient descent / ascent, etc., but this paper does not limit this.

[0112] S205: Based on the states and actions corresponding to the aforementioned state-action sequence segments and the trained multi-objective reward model, establish a Markov decision process model;

[0113] Traditional Markov decision processes are constructed based on (s,a,s′) and can be represented as M =<S,A,P,R,γ> In this model, S represents the state space, A represents the action space, P (i.e., P(s′|s,a)) represents the probability of transitioning from the current state s to the next state s′ through the current action a, R represents the reward function, and γ is the discount factor. In the embodiments of this specification, the Markov decision process model is constructed based on (s,a,s′,w), and R is extended to R(s,a), which is a vector function that maps the state and action to... The model contains real-time rewards for m different objectives. Specifically, from several state-action sequence segments, the current state, current action, and next state can be selected. The current state and current action are input into the trained multi-objective reward model to obtain the current multi-objective reward R(s,a). Then, based on the current state, current action, current multi-objective reward, and next state, a state transition tuple (s,a,s′,w) is constructed to establish a Markov decision process model. Further, the agent's goal is to find an optimal policy a = π(s|w), where the multi-objective weight feature w reflects the relative importance of different objectives, and the policy π(s|w) selects actions based on the multi-objective weight feature w to maximize the expected value of the weighted reward.

[0114] S206: Based on the Markov decision process model, in the learning environment corresponding to the multi-objective policy optimization request, a multi-objective policy is trained using a multi-objective reinforcement learning algorithm to obtain the multi-objective policy.

[0115] Specifically, based on the established Markov decision process model, a multi-objective policy can be trained using a multi-objective reinforcement learning algorithm within a learning environment corresponding to a multi-objective policy optimization request. During training, the trained multi-objective reward model is used to predict multi-objective rewards in the state transition data. Furthermore, the multi-objective reward model supports updates as the multi-objective feedback index data changes, ensuring that the final policy always meets the requirements of the multi-objective feedback index data. (See Appendix) Figure 4 The specific training process can be as follows:

[0116] Step 0: Initialize network parameters;

[0117] Step 1: Obtain the current state and determine whether the current number of steps exceeds the preset threshold;

[0118] Step 2: If the preset threshold is not exceeded, the current action is obtained by random sampling; if the preset threshold is not exceeded, the current action is obtained by sampling based on the control strategy.

[0119] Step 3: Obtain the state transition tuple consisting of the current state, current action, next state, and multi-objective weight features from the multi-objective feedback index data;

[0120] Step 4: Call the corresponding trained multi-objective reward model according to the state transition tuple, and input the current state and current action into the trained multi-objective reward model to obtain the current multi-objective reward;

[0121] Step 5: Add the state transition data, consisting of the current state, current action, current multi-objective reward, and next state, to the replay buffer for this iteration; the replay buffer is used to store the state transition data required for each iteration;

[0122] Step 6: Perform partial batch sampling of the state transition data in the replay buffer of this iteration to obtain sampled batch data, and based on the sampled batch data, perform multi-objective reinforcement learning using the reinforcement learning loss function to update the network parameters;

[0123] Repeat steps 1-6 above until the preset number of iterations is reached.

[0124] Furthermore, in some embodiments, when the multi-objective feedback index data changes, it is necessary to update the trained multi-objective reward model synchronously to ensure the accuracy of the multi-objective reward model's output. In this case, a preset update frequency can be used to periodically monitor the changes in the multi-objective feedback index data, as shown in the appendix. Figure 5 Updating the trained multi-objective reward model to update the multi-objective reward data in the replay buffer may further include:

[0125] S501: Preset update frequency for acquiring multi-objective feedback indicator data;

[0126] S502: As the number of iterations increases, when the relationship between the preset update frequency and the current number of iterations meets the preset condition, a preset number of state action sequence fragments are sampled from the replay buffer, and a preset number of multi-target weight features are sampled from the preset dataset.

[0127] S503: Generate new multi-objective feedback index data based on the sampled multi-objective weight features, state-action sequence fragments, and the binary evaluator;

[0128] S504: Add the new multi-objective feedback index data to the feedback index dataset;

[0129] S505: Using the data in the feedback index dataset, update the trained multi-objective reward model according to the preset loss function;

[0130] S506: Update the replay buffer using the updated multi-objective reward model.

[0131] Specifically, based on the Markov decision process model, in the learning environment corresponding to the multi-objective policy optimization request, the detailed process of training a multi-objective policy using a multi-objective reinforcement learning algorithm can be referred to as Algorithm 1 below.

[0132]

[0133] This specification can be understood as follows: First, the embodiments automatically receive multi-objective policy optimization requests and obtain several state-action sequence fragments, multi-objective weight features, and a binary evaluator for evaluating the state-action sequence fragments corresponding to the multi-objective policy optimization requests, in order to construct multi-objective feedback index data. Then, an explicit multi-objective reward model is established, and the multi-objective reward model is trained using the multi-objective feedback index data, so that the trained multi-objective reward model can be used to characterize the policy generation tendency corresponding to the multi-objective feedback index data. Then, based on the states and actions corresponding to the several state-action sequence fragments and the trained multi-objective reward model, a Markov decision process model is established, and a multi-objective reinforcement learning algorithm is applied for training to achieve efficient and accurate output of multi-objective policies in complex multi-objective scenarios.

[0134] To facilitate understanding by those skilled in the art, this specification provides a typical multi-objective strategy optimization scenario 1. The task scenario is to train a robot dog to imitate the behavior of a real dog. The corresponding multiple training objectives may include the robot dog running forward T1, the robot dog jumping T2, the coordination of the robot dog's movements T3, and the robot dog ensuring its own equipment safety while performing actions T4. The state action sequence segment (i.e., trajectory) is as follows: σ1: The robot dog stays still; σ2: The robot dog runs forward in an unstable state (e.g., staggering forward); σ3: The robot dog runs forward quickly with a large range of motion, which may lead to equipment safety issues; σ4: The robot dog runs forward quickly, jumping while running. Then, using comprehensive evaluation and analysis methods (such as analytic hierarchy process, grey relational analysis, etc.), the multi-objective weight features w corresponding to the current task scenario are obtained. For example, when w = [1, 0, 0, 0], it means that the robot dog's forward running T1 is important, and it does not pay attention to other objectives. Comparing the trajectory (σ1, σ2), since σ1 remains stationary, while σ2 moves forward, the evaluation result given by the binary evaluator is p = (0, 1). When w = [0.5, 0.5, 0, 0], it means that the robot dog's forward running T1 and jumping T2 are equally important, and it does not pay attention to other objectives. Comparing the trajectory (σ2, The binary evaluator gives an evaluation result p = (0,1) for trajectory σ4, because trajectory σ4 has both running and jumping states, while σ2 only has a running state. When w = [0.3,0,0.7,0], it means that compared to the training objective T1 of the robot dog running forward, more attention is paid to the coordination T3 of the robot dog's actions, and other objectives are not considered. The comparison trajectory is (σ1,σ2), and the feedback index p = (1,0). In the comparison trajectory (σ1,σ2), σ2 has a forward running action but the action is uncoordinated, while σ1, although stationary (without a forward running action), does not have a coordination problem, so p = (1,0). Thus, in the scenario of training the robot dog to imitate the behavior of a real dog, multi-objective feedback index data can be quickly constructed based on several state-action sequence segments corresponding to the multi-objective strategy optimization request, multi-objective weight features, and a binary evaluator used to evaluate the state-action sequence segments. It should be noted that, in practical use, the binary evaluator can provide evaluation results for two or more state-action sequence segments by comparing them pairwise, but this paper does not limit this.

[0135] To facilitate understanding by those skilled in the art, this specification also provides a typical multi-objective strategy optimization scenario 2. The task scenario is to train a multi-objective autonomous driving system, with multiple training objectives: vehicle speed G1; vehicle driving smoothness G2 (e.g., not wanting to stop and wait); vehicle driving safety G3 (e.g., not allowing speeding, maintaining a certain distance from pedestrians, etc.); passenger comfort G4 (e.g., not wanting excessively high frequency / amplitude braking and acceleration). Correspondingly, the state-action sequence segments (i.e., trajectories) are: β1: vehicle stationary; β2: vehicle driving normally on ordinary roads; β3: vehicle driving through busy urban areas, frequently changing lanes; β4: vehicle driving slowly in busy urban areas, slower than β3, frequently braking. When w = [1,0,0,0] and the comparison trajectory is (σ1,σ2), since the focus is on the vehicle's speed G1, σ1 is stationary, and σ2 is moving forward, therefore p = (0,1). When w = [0.3,0.7,0,0] and the comparison trajectory is (σ3,σ4), since the focus is more on the smoothness of driving G2 than on the vehicle's speed G1, and other targets are not considered, and σ4 is slow and brakes frequently, while σ3 is faster than σ4 and drives smoothly, therefore p = (1,0). This allows for the rapid construction of multi-objective feedback index data in the training scenario of a multi-objective autonomous driving system, based on several state-action sequence segments corresponding to the multi-objective policy optimization request, multi-objective weight features, and a binary evaluator used to evaluate the state-action sequence segments. This ensures the effectiveness of subsequent training of the multi-objective reward model and the multi-objective reinforcement learning training process.

[0136] It should be noted that although the operation of the method of the present invention has been described in a specific order in the above embodiments and figures, this does not require or imply that the operations must be performed in that specific order, or that all the operations shown must be performed to achieve the desired result. Additionally or alternatively, certain steps may be omitted, multiple steps may be combined into one step, and / or one step may be broken down into multiple steps.

[0137] Corresponding to the above-described policy generation method based on multi-objective reinforcement learning, some embodiments of this specification also provide a policy generation apparatus based on multi-objective reinforcement learning, see reference. Figure 6 As shown, in some embodiments, the apparatus may include:

[0138] The receiving module 601 is used to receive multi-objective strategy optimization requests;

[0139] The feedback index data construction module 602 is used to construct multi-objective feedback index data based on several state-action sequence segments corresponding to the multi-objective strategy optimization request, multi-objective weight features, and a binary evaluator for evaluating the state-action sequence segments.

[0140] The multi-objective reward model building module 603 is used to build an explicit multi-objective reward model;

[0141] A multi-objective reward model training module 604 is used to train the multi-objective reward model using the multi-objective feedback index data.

[0142] The Markov decision process model building module 605 is used to build a Markov decision process model based on the states, actions, and trained multi-objective reward model corresponding to the several state-action sequence segments.

[0143] The multi-objective reinforcement learning module 606 is used to train a multi-objective policy using a multi-objective reinforcement learning algorithm in the learning environment corresponding to the multi-objective policy optimization request, based on the Markov decision process model.

[0144] For ease of description, the above devices are described in terms of function, divided into various units. Of course, in implementing this specification, the functions of each unit can be implemented in one or more software and / or hardware components.

[0145] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in the embodiments of this specification are all information and data authorized and agreed upon by the user and fully authorized by all parties.

[0146] It should be noted that the computer program product described in this specification is a software product that mainly implements the methods described in this specification through a computer program.

[0147] Embodiments of this specification also provide a computer device. For example... Figure 7As shown, in some embodiments of this specification, the computer device 702 may include one or more processors 704, such as one or more central processing units (CPUs) or graphics processing units (GPUs), each processing unit implementing one or more hardware threads. The computer device 702 may also include any memory 706 for storing information of any kind, such as code, settings, data, etc. In one specific embodiment, a computer program is stored on the memory 706 and can run on the processor 704. When the computer program is run by the processor 704, it can execute instructions described in any of the above embodiments. Non-limitingly, for example, the memory 706 may include any type of RAM, any type of ROM, flash memory, hard disk, optical disk, etc. More generally, any memory can use any technology to store information. Further, any memory can provide volatile or non-volatile retention of information. Further, any memory can represent a fixed or removable component of the computer device 702. In one case, when the processor 704 executes associated instructions stored in any memory or combination of memories, the computer device 702 can perform any operation of the associated instructions. The computer device 702 also includes one or more drive mechanisms 708 for interacting with any memory, such as a hard disk drive mechanism, an optical disk drive mechanism, etc.

[0148] Computer device 702 may also include an input / output interface 710 (I / O) for receiving various inputs (via input device 712) and providing various outputs (via output device 714). A specific output mechanism may include a presentation device 716 and an associated graphical user interface 718 (GUI). In other embodiments, the input / output interface 710 (I / O), input device 712, and output device 714 may be omitted, and the device may function solely as a computer device within a network. Computer device 702 may also include one or more network interfaces 720 for exchanging data with other devices via one or more communication links 722. One or more communication buses 724 couple the components described above together.

[0149] Communication link 722 can be implemented in any way, such as via a local area network, a wide area network (e.g., the Internet), a point-to-point connection, or any combination thereof. Communication link 722 may include any combination of hardwired links, wireless links, routers, gateway functions, name servers, etc., governed by any protocol or combination of protocols.

[0150] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), computer-readable storage media, and computer program products according to some embodiments of this specification. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processor to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processor, create a mechanism for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0151] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processor to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0152] These computer program instructions may also be loaded onto a computer or other programmable data processor, causing a series of operational steps to be performed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable device for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0153] In a typical configuration, a computer device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0154] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0155] Computer-readable media, including both permanent and non-permanent, removable and non-removable media, can store information using any method or technology. Information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by computer equipment. As defined in this specification, computer-readable media does not include transient media, such as modulated data signals and carrier waves.

[0156] Those skilled in the art will understand that the embodiments of this specification can be provided as methods, systems, or computer program products. Therefore, the embodiments of this specification can take the form of entirely hardware embodiments, entirely software embodiments, or embodiments combining software and hardware aspects. Furthermore, the embodiments of this specification can take the form of computer program products implemented on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0157] The embodiments described in this specification can be described in the general context of computer-executable instructions, such as program modules, that are executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform a specific task or implement a specific abstract data type. The embodiments of this specification can also be practiced in distributed computing environments where tasks are performed by remote processors connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0158] It should also be understood that, in the embodiments of this specification, the term "and / or" is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. Additionally, the character " / " in this document generally indicates that the preceding and following related objects have an "or" relationship.

[0159] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the system embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments.

[0160] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., refer to specific features, structures, materials, or characteristics described in connection with that embodiment or example, which are included in at least one embodiment or example of the embodiments of this specification. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0161] The above description is merely an embodiment of this application and is not intended to limit this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principle of this application should be included within the scope of the claims of this application.

Claims

1. A policy generation method based on multi-objective reinforcement learning, characterized in that, The method includes: Receive multi-objective strategy optimization requests; Based on the several state-action sequence fragments corresponding to the multi-objective strategy optimization request, the multi-objective weight features, and the binary evaluator used to evaluate the state-action sequence fragments, construct multi-objective feedback index data; Establish an explicit multi-objective reward model; The multi-objective reward model is trained using the multi-objective feedback index data; Based on the states, actions, and trained multi-objective reward models corresponding to the aforementioned state-action sequence segments, a Markov decision process model is established. Based on the Markov decision process model, in the learning environment corresponding to the multi-objective policy optimization request, a multi-objective policy is trained using a multi-objective reinforcement learning algorithm to obtain the multi-objective policy. Training the multi-objective reward model using the multi-objective feedback index data includes: Select the state and action at a specified moment from the state-action sequence fragment of the multi-objective feedback index data; The state and action at a specified time are input into the multi-objective reward model to obtain the corresponding predicted reward; Based on the correlation between the predicted reward and the multi-objective weight features in the multi-objective feedback index data, a feedback index predictor is established. The predicted reward and the multi-objective weight features in the multi-objective feedback index data are input into the feedback index predictor to obtain the feedback index prediction result; The loss corresponding to the predicted result of the feedback index is calculated according to the preset loss function, and the feedback index predictor is trained using the parameter optimization algorithm to update the model parameters in the multi-objective reward model. The feedback indicator predictor is built using the following method: in, This indicates a feedback indicator predictor. Indicates after, and Each represents a segment of a state-action sequence, w represents a multi-objective weighted feature, exp represents an exponential function with the natural constant e as its base, {l, k} represents the set of categories of a binary scalar, and i represents an element in the set of categories of a binary scalar. This represents the state at time t when the category of the binary scalar is i. and actions The corresponding predicted reward, Indicates transpose; Based on the Markov decision process model, in the learning environment corresponding to the multi-objective policy optimization request, a multi-objective policy is trained using a multi-objective reinforcement learning algorithm, including: Step 0: Initialize network parameters; Step 1: Obtain the current state and determine whether the current number of steps exceeds the preset threshold; Step 2: If the preset threshold is not exceeded, the current action is obtained by random sampling; if the preset threshold is not exceeded, the current action is obtained by sampling based on the control strategy. Step 3: Obtain the state transition tuple consisting of the current state, current action, next state, and multi-objective weight features from the multi-objective feedback index data; Step 4: Call the corresponding trained multi-objective reward model according to the state transition tuple, and input the current state and current action into the trained multi-objective reward model to obtain the current multi-objective reward; Step 5: Add the state transition data, consisting of the current state, current action, current multi-objective reward, and next state, to the replay buffer for this iteration; the replay buffer is used to store the state transition data required for each iteration; Step 6: Perform partial batch sampling of the state transition data in the replay buffer of this iteration to obtain sampled batch data, and based on the sampled batch data, perform multi-objective reinforcement learning using the reinforcement learning loss function to update the network parameters; Repeat steps 1-6 above until the preset number of iterations is reached; Preset update frequency for acquiring multi-objective feedback indicator data; As the number of iterations increases, when the relationship between the preset update frequency and the current number of iterations meets the preset condition, a preset number of state action sequence fragments are sampled from the replay buffer, and a preset number of multi-objective weight features are sampled from the preset dataset. Based on the sampled multi-objective weight features, state-action sequence fragments, and the binary evaluator, new multi-objective feedback index data is generated. Add the new multi-objective feedback index data to the feedback index dataset; Using the data in the feedback index dataset, the trained multi-objective reward model is updated according to a preset loss function; The replay buffer is updated using the updated multi-objective reward model.

2. The method according to claim 1, characterized in that, Based on the several state-action sequence fragments corresponding to the multi-objective policy optimization request, the multi-objective weight features, and the binary evaluator used to evaluate the state-action sequence fragments, multi-objective feedback index data is constructed, including: Each state-action sequence segment, multi-objective weight features, and a binary evaluator used to evaluate the state-action sequence segment are used as tuple elements to construct multi-objective feedback index data based on the tuple elements.

3. The method according to claim 1, characterized in that, Based on the states, actions, and trained multi-objective reward model corresponding to the aforementioned state-action sequence segments, a Markov decision process model is established, including: The current state, current action, and next state are determined based on the states and actions corresponding to the aforementioned state-action sequence segments. The current state and current action are input into the trained multi-objective reward model to obtain the current multi-objective reward; Based on the current state, current action, current multi-objective reward, and next state, state transition data is constructed to establish a Markov decision process model.

4. A policy generation device based on multi-objective reinforcement learning, characterized in that, The device includes: The receiving module is used to receive multi-objective strategy optimization requests; The feedback index data construction module is used to construct multi-objective feedback index data based on several state-action sequence fragments corresponding to the multi-objective strategy optimization request, multi-objective weight features, and a binary evaluator for evaluating the state-action sequence fragments. The multi-objective reward model building module is used to build explicit multi-objective reward models; A multi-objective reward model training module is used to train the multi-objective reward model using the multi-objective feedback index data; The Markov decision process model building module is used to build a Markov decision process model based on the states, actions, and trained multi-objective reward model corresponding to the several state-action sequence segments. The multi-objective reinforcement learning module is used to train a multi-objective policy using a multi-objective reinforcement learning algorithm in the learning environment corresponding to the multi-objective policy optimization request, based on the Markov decision process model. The multi-objective reward model training module selects the state and action at a specified moment from the state-action sequence fragment of the multi-objective feedback index data. The state and action at a specified time are input into the multi-objective reward model to obtain the corresponding predicted reward; Based on the correlation between the predicted reward and the multi-objective weight features in the multi-objective feedback index data, a feedback index predictor is established. The predicted reward and the multi-objective weight features in the multi-objective feedback index data are input into the feedback index predictor to obtain the feedback index prediction result; The loss corresponding to the predicted result of the feedback index is calculated according to the preset loss function, and the feedback index predictor is trained using the parameter optimization algorithm to update the model parameters in the multi-objective reward model. The feedback indicator predictor is built using the following method: in, This indicates a feedback indicator predictor. Indicates after, and Each represents a segment of a state-action sequence, w represents a multi-objective weighted feature, exp represents an exponential function with the natural constant e as its base, {l, k} represents the set of categories of a binary scalar, and i represents an element in the set of categories of a binary scalar. This represents the state at time t when the category of the binary scalar is i. and actions The corresponding predicted reward, Indicates transpose; The multi-objective reinforcement learning module specifically performs the following steps. Step 0: Initialize network parameters; Step 1: Obtain the current state and determine whether the current number of steps exceeds the preset threshold; Step 2: If the preset threshold is not exceeded, the current action is obtained by random sampling; if the preset threshold is not exceeded, the current action is obtained by sampling based on the control strategy. Step 3: Obtain the state transition tuple consisting of the current state, current action, next state, and multi-objective weight features from the multi-objective feedback index data; Step 4: Call the corresponding trained multi-objective reward model according to the state transition tuple, and input the current state and current action into the trained multi-objective reward model to obtain the current multi-objective reward; Step 5: Add the state transition data, consisting of the current state, current action, current multi-objective reward, and next state, to the replay buffer for this iteration; the replay buffer is used to store the state transition data required for each iteration; Step 6: Perform partial batch sampling of the state transition data in the replay buffer of this iteration to obtain sampled batch data, and based on the sampled batch data, perform multi-objective reinforcement learning using the reinforcement learning loss function to update the network parameters; Repeat steps 1-6 above until the preset number of iterations is reached; Preset update frequency for acquiring multi-objective feedback indicator data; As the number of iterations increases, when the relationship between the preset update frequency and the current number of iterations meets the preset condition, a preset number of state action sequence fragments are sampled from the replay buffer, and a preset number of multi-objective weight features are sampled from the preset dataset. Based on the sampled multi-objective weight features, state-action sequence fragments, and the binary evaluator, new multi-objective feedback index data is generated. Add the new multi-objective feedback index data to the feedback index dataset; Using the data in the feedback index dataset, the trained multi-objective reward model is updated according to a preset loss function; The replay buffer is updated using the updated multi-objective reward model.

5. A computer device comprising a memory, a processor, and a computer program stored in the memory, characterized in that, When the computer program is run by the processor, it executes the instructions of the method according to any one of claims 1-3.

6. A computer storage medium having a computer program stored thereon, characterized in that, When the computer program is run by the processor of the computer device, it executes the instructions of the method according to any one of claims 1-3.

Images