A quasi-metric and planning based multi-objective reinforcement learning method

Abstract

Disclosed is a multi-objective reinforcement learning method based on quasi-metric and planning, comprising: constructing a representation function, a quasi-metric value function and a hidden space dynamics model, and jointly training by using transition data and target states sampled from environment interaction, wherein the quasi-metric value function is optimized to fit the cost of state to target, and is used as a distance metric standard for optimizing the hidden space dynamics model; when making a decision, an MPC framework is used, and CEM is combined for action selection optimization, the process uses the hidden space dynamics model to predict future trajectories, uses the quasi-metric value function to evaluate the cost of the predicted trajectories, and determines a single action applied to the current state according to the lowest cost planning result obtained by CEM optimization. Through learning of the quasi-metric cost function and the hidden space model, and combining with model predictive control planning, the disclosure can effectively cope with sparse rewards, and significantly improve the sample efficiency and final performance of multi-objective reinforcement learning.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the field of reinforcement learning of artificial intelligence, and particularly relates to a multi-objective reinforcement learning method based on quasi-metric and planning. BACKGROUND

[0002] Artificial intelligence, as an important driving force for the new round of scientific and technological revolution and industrial transformation, is experiencing rapid development with the explosive growth of data volume, significant improvement of computing power, and continuous breakthroughs in algorithm models. Reinforcement learning (RL), as a key branch of artificial intelligence, focuses on studying how an agent learns the optimal decision-making strategy through autonomous interaction with the environment. In recent years, reinforcement learning has shown the ability to surpass human experts in many fields, such as the breakthrough of Google DeepMind's AlphaGo system in the field of Go, and the top level achieved by OpenAI team's OpenAI Five system in the complex multiplayer online game Dota 2, which proves the great potential of reinforcement learning technology.

[0003] Traditional reinforcement learning algorithms largely rely on the reward function designed by domain experts based on professional knowledge to guide the optimization process of the policy. However, in many real-world application scenarios, especially in complex robotic systems or interactive tasks, it is often extremely difficult, if not impossible, to design a dense reward function that accurately reflects the task goals and is easy to optimize.

[0004] Under this background, multi-objective reinforcement learning (MORL), especially goal-conditioned reinforcement learning, is particularly important. In a typical multi-objective reinforcement learning environment, the reward function is usually simple, and the agent may face different goal states in each interaction episode. The reward signal provided by the environment is often very sparse. Usually, the environment only provides a simple signal to represent whether the current state has reached the specified goal state of this episode. The agent must use this extremely sparse feedback signal to learn how to explore and master the method of reaching different goal states in a complex environment, which constitutes the core challenge of this field.

[0005] Currently, the academic and industrial communities have proposed some algorithms to try to solve the challenges in multi-objective reinforcement learning. For example, a strategy based on hindsight experience replay (HER) or relabeling treats the interaction trajectories that fail to reach the expected goal as successful trajectories that actually reach some state (as a hindsight goal) in the successful trajectories. In this way, successful experiences are "created", and the algorithm can learn the policy for transferring between different states and rely on the generalization ability of the neural network to master the general policy for reaching different goals. Another method draws on the idea of contrastive learning, such as treating state pairs collected within the same interaction round as positive samples and treating state pairs from different rounds as negative samples to learn a value function or representation that reflects the reachability between states, and then guide the learning of the policy.

[0006] Although the above methods alleviate the learning difficulty caused by reward sparsity in multi-objective reinforcement learning to some extent, they usually face the problem of low sample efficiency. In other words, the agent often needs to interact with the environment a large number of times and collect a large amount of experience data to learn an effective policy. In contrast, model-based reinforcement learning (MBRL) methods show the potential to improve sample efficiency. By learning an environment model (Dynamics Model) to approximate the state transition and reward mechanism of the real environment, the agent can use the model for internal simulation and planning. The model can extract more useful information from limited real interaction data and allow the agent to explore and evaluate different policies in "imagination", so that the agent can effectively learn the policy even in the absence of immediate reward signals.

[0007] Therefore, how to effectively overcome the defects of reward sparsity and low sample efficiency in multi-objective reinforcement learning is an important and valuable research direction in the current reinforcement learning field. SUMMARY

[0008] The present disclosure proposes a multi-objective reinforcement learning method based on quasi-metric and planning, which can effectively overcome the defects of reward sparsity and low sample efficiency in multi-objective reinforcement learning by combining the advantages of the model.

[0009] According to one embodiment of the present disclosure, a multi-objective reinforcement learning method based on quasi-metric and planning is proposed, comprising:

[0010] constructing a representation function, a pseudo-metric value function and a latent space dynamics model, and jointly training the representation function, the pseudo-metric value function and the latent space dynamics model using transition data (s, a, s', r) and a target state g sampled from environment interaction; wherein the representation function is used to map the state sampled from the environment interaction to the corresponding latent space representation; the latent space dynamics model is used to predict the latent space representation of the next state based on the latent space representation of the current state s and the action a applied to the state; the pseudo-metric value function is used to calculate the pseudo-metric distance between the two input latent space representations, and during the training of the latent space dynamics model, it is used as the distance measurement standard between the predicted latent space representation of the next state and the latent space representation of the sampled next state s'; and the reward r is used to optimize the pseudo-metric value function;

[0011] using model predictive control (MPC) and combining cross-entropy method (CEM) to optimize action selection, wherein in the MPC process, the current trained latent space dynamics model is used to predict the future latent space trajectory, the current trained pseudo-metric value function is used to calculate the pseudo-metric distance between the final latent space representation of the predicted latent space trajectory and the latent space representation of the target state as the cost evaluation standard, and the single action applied to the current state is determined according to the lowest cost action planning result obtained by CEM optimization.

[0012] In some embodiments, if the state of the environment is in the form of pixel space, the representation function is composed of a convolutional neural network (CNN) backbone network and a feedforward neural network (FNN); if the state of the environment is in the form of vector space, the representation function is composed of a feedforward neural network (FNN).

[0013] In some embodiments, the pseudo-metric value function is implemented by an interval pseudo-metric embedding model.

[0014] In some embodiments, the calculation process of the interval pseudo-metric embedding model includes:

[0015] For the two input latent space representations, the dimensions of each latent space representation are divided into a plurality of preset categories;

[0016] For each category, the interval union of the values of the two input latent space representations in the dimensions of the category is calculated;

[0017] The lengths of the interval unions of all categories are summed up as the pseudo-metric distance between the two input latent space representations.

[0018] In some embodiments, during the joint training, when the reward r is used to optimize the pseudo-metric value function, the optimization target is set as:

[0019]

[0020] where θ is the parameter of the quasi-metric value function, λ is the Lagrange multiplier, denotes the mathematical expectation, s ~ p s denotes that the sampled state s conforms to the probability distribution p s , g ~ p g denotes that the target state g conforms to the probability distribution p g , (s, a, s', r) ~ p transition denotes that the transition data (s, a, s', r) is sampled based on the real environment model p transition d θ is an end-to-end function for calculating quasi-metric distance, used to perform the process of mapping the input two states into corresponding hidden space representations respectively, and calculating the quasi-metric distance between the two hidden space representations, φ is a pre-selected monotonically increasing convex function, ∈ 2 is a constant threshold.

[0021] In some embodiments, in the joint training, the following loss function is used to train the hidden space dynamics model:

[0022]

[0023] wherein, denotes the predicted hidden space representation of the next state, z' denotes the hidden space representation of the real next state s', and d is the quasi-metric value function.

[0024] In some embodiments, the joint training is implemented in a multi-step form, wherein:

[0025] The optimization target of the quasi-metric value function adopts the following multi-step form:

[0026]

[0027] The loss function of the hidden space dynamics model adopts the following multi-step form:

[0028]

[0029] where θ is the parameter of the quasi-metric value function network, λ is the Lagrange multiplier, H is the preset step size, ρ i is a preset attenuation factor, φ is a pre-selected monotonically increasing convex function, d is the quasi-metric value function, z i+i denotes the hidden space representation of the sampled (t+i)th step state, z g denotes the hidden space representation of the target state g, z t+i+1 denotes the hidden space representation of the sampled (t+i+1)th step state, a latent space representation representing a predicted (t+i+1)-th state, t being a start time step of a current trajectory segment, r t+i is an immediate reward, ∈ 2 is a constant threshold.

[0030] In some embodiments, the joint training is implemented in a multi-step form, including:

[0031] sampling a trajectory segment of transition data of a preset step length from an experience replay buffer;

[0032] mapping states in the trajectory segment to a latent space using a representation function to obtain a sequence of mapped latent space representations;

[0033] iteratively predicting a sequence of latent space representations of a preset step length into the future using a latent space dynamics model in the latent space;

[0034] jointly training the representation function, the quasi-metric value function, and the latent space dynamics model based on the sequence of mapped latent space representations and the predicted sequence of latent space representations using an optimization objective and a loss function in a multi-step form.

[0035] In some embodiments, model predictive control (MPC) is used in combination with cross-entropy method (CEM) for action selection optimization, including:

[0036] iteratively performing the following steps for the current state until a preset condition is met:

[0037] sampling N candidate action sequences of a preset step length from a current action sampling distribution, N being a preset number of candidate action sequences;

[0038] predicting a latent space trajectory corresponding to each candidate action sequence based on the current state using the currently trained latent space dynamics model;

[0039] calculating a quasi-metric distance from a final latent space representation of each predicted latent space trajectory to a latent space representation of a target state as a cost using the currently trained quasi-metric value function;

[0040] selecting K elite action sequences with the lowest costs from the calculated costs, K being a preset value, 0 < K < N;

[0041] updating parameters of the action sampling distribution based on the selected elite action sequences.

[0042] In some embodiments, sampling N candidate action sequences from the current action sampling distribution includes:

[0043] randomly sampling an action sequence of a preset step length from a preset probability distribution, or sampling an action sequence of a preset step length using a trained policy network.

[0044] In some embodiments, the method further comprises:

[0045] Based on the current trained latent space dynamics model and the quasi-metric value function, an action value function Q is extended to satisfy the following approximate relationship * (s, a; g):

[0046]

[0047] where d is the quasi-metric value function, represents the latent space representation of the predicted next state, z g represents the latent space representation of the target state g, and r is the reward.

[0048] The obtained action value function is used to evaluate the cost of performing different actions for updating the policy network.

[0049] The present disclosure provides a multi-objective reinforcement learning method based on quasi-metric and planning. By combining model construction and training and model-based planning decision, it has significant beneficial effects. First, by constructing and jointly training the representation function, the quasi-metric value function and the latent space dynamics model, the present disclosure can effectively deal with the reward sparsity problem commonly existing in multi-objective reinforcement learning. The quasi-metric value function directly fits the distance from the state to the target, fully utilizes the immediate reward provided by the environment (usually sparse) and learns in combination with specific optimization goals, so as to construct an internal cost d that can reflect the accessibility of the target. At the same time, the learned latent space dynamics model enables the agent to predict future state transitions in the latent space. Second, through model-based planning decision and interaction, forward simulation prediction and evaluation based on the learned quasi-metric cost d using the trained model can significantly improve sample efficiency. The agent can explore multiple future trajectories through internal simulation in each decision-making step, and evaluate their pros and cons based on the quasi-metric value function, so as to make more predictable decisions, reduce the dependence on a large amount of real environment interaction data, and speed up the learning process of effective strategies. The present disclosure provides an effective way for efficient multi-objective reinforcement learning in sparse reward environments by proposing a learning and planning combined framework.

[0050] Further, in different embodiments of the present disclosure, by adopting the interval quasimetric embedding model as the implementation manner of the quasi-metric value function, a good foundation is provided for learning the cost function satisfying the quasi-metric property; by adopting a specific optimization objective and loss function for joint training, especially by adopting a specially designed multi-step form, the value information and learning dynamics model can be more stably and effectively propagated, further improving the performance and robustness of learning, especially when processing long time sequence dependent tasks; in the planning stage, the cross-entropy method (CEM) and other optimization algorithms are adopted, which can effectively search and find the optimal action sequence in multiple candidate trajectories, improving the quality of planning decision.

[0051] Other details and advantages of the present disclosure will be described in detail below. BRIEF DESCRIPTION OF DRAWINGS

[0052] The accompanying drawings, which are incorporated herein and form a part of the specification, illustrate embodiments consistent with the present description and, together with the description, further serve to explain the principles of the present description.

[0053] Figure 1 A flowchart of a multi-objective reinforcement learning method based on quasi-metric and planning according to one embodiment of the present disclosure is shown.

[0054] Figure 2 A CNN backbone network schematic diagram for implementing a representation function model according to one exemplary embodiment of the present disclosure is shown.

[0055] Figure 3 A FNN schematic diagram for implementing a representation function model according to one exemplary embodiment of the present disclosure is shown.

[0056] Figure 4 An end-to-end model structure schematic diagram based on quasi-metric according to one exemplary embodiment of the present disclosure is shown.

[0057] Figure 5 A planning process schematic diagram based on model predictive control and cross-entropy method (MPC-CEM) according to one exemplary embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0058] The exemplary embodiments will be described in detail herein with reference to the attached drawings. The following description is made with reference to the accompanying drawings in which like reference numerals refer to like elements in the several figures. The embodiments described in these exemplary embodiments are not intended to represent all embodiments consistent with the present disclosure. Rather, they are merely examples of apparatuses and methods consistent with some aspects of the present disclosure as detailed in the appended claims.

[0059] The present disclosure provides a quasi-metric and planning based multi-objective reinforcement learning method for solving the problems of reward sparsity and low sample efficiency prevalent in the field. Figure 1 A flowchart of a quasi-metric and planning based multi-objective reinforcement learning method according to an embodiment of the present disclosure is shown. As shown in Figure 1 The method mainly includes two core steps: model construction and training (step 1) and model-based planning decision and interaction (step 2).

[0060] Step 1, constructing a representation function, a quasi-metric value function and a latent space dynamics model, and jointly training the representation function, the quasi-metric value function and the latent space dynamics model using transition data (s, a, s', r) and target state g sampled from environment interaction; wherein the representation function is used to map the state sampled from environment interaction to the corresponding latent space representation; the latent space dynamics model is used to predict the latent space representation of the next state based on the latent space representation of the current state s and the action a applied to the state; the quasi-metric value function is used to calculate the quasi-metric distance between two input latent space representations, and during the training of the latent space dynamics model, it is used as the distance measurement standard between the predicted latent space representation of the next state and the latent space representation of the sampled next state s'; the reward r is used to optimize the quasi-metric value function.

[0061] In reinforcement learning, the "environment" refers to the system in which the agent is located and interacts with. The agent affects the environment by performing actions, and receives the next state s' and reward r from the environment feedback. This continuous action and feedback process is called environment interaction. "State" is a specific description of the environment at a certain time, which contains the relevant information needed for the agent to make decisions. "State sampled from environment interaction" refers to the specific state of the environment experienced and recorded by the agent during actual interaction with the environment at a certain time. These state data are the basis for subsequent random extraction (sampling) for training and learning. Reward r is a scalar feedback signal provided by the environment itself according to the preset rules after the agent performs an action a from the current state s to the next state s'.

[0062] In step 1, a series of interrelated neural network models are first constructed and jointly trained through interaction data with the environment. The representation function, quasi-metric value function and latent space dynamics model constructed according to the present embodiment are described below.

[0063] The constructed representation function is used to map the state sampled from environment interaction to the corresponding latent space representation.

[0064] Representation function H θThis is used to map the agent's initial state s, sampled from environmental interactions, and the current task's target state g, to a more information-dense latent space representation z. s and z g Latent space representations are used to capture features that are important for subsequent tasks, such as cost estimation and dynamics prediction.

[0065] In some implementations, if the state of the environment is in pixel space form, the representation function is composed of a convolutional neural network (CNN) backbone and a feedforward neural network (FNN).

[0066] For example, if the input is an image or video frame from a camera, corresponding to an image-based environment, then the representation function can be composed of a convolutional neural network (CNN) backbone and a feedforward neural network (FNN). The CNN backbone (e.g., ...) Figure 2 The convolutional layer shown is responsible for extracting spatial features from high-dimensional pixel data. Its output feature map is then flattened and passed through one or more fully connected layers (such as...). Figure 3 The FNN shown is further processed to finally generate a low-dimensional latent space vector representation.

[0067] In some implementations, if the state of the environment is in vector space form, the representation function is composed of a feedforward neural network (FNN).

[0068] For example, if the input is a feature vector consisting of robot joint angles, velocities, and object coordinates, corresponding to a vector-based environment, then the representation function can be directly constructed by a feedforward neural network (FNN) (e.g., ...). Figure 3 The FNN shown here transforms and abstracts the input feature vector through a series of fully connected layers and nonlinear activation functions (such as ReLU), extracting higher-level features and outputting the final latent space vector representation.

[0069] The characterization function H constructed through the above implementation method θ It can effectively handle different types of input states and provide a high-quality, unified latent space representation for subsequent metric function calculations and latent space dynamics modeling.

[0070] The constructed quasimetric function is used to calculate the quasimetric distance between the two latent space representations of the input.

[0071] Based on the theory that (negative) optimal value functions and quasi-metrics are equivalent in multi-objective reinforcement learning, this disclosure proposes to construct a quasi-metric value function d. θ This function accepts two inputs (e.g., z). s and z g ), and output a scalar value (e.g., d(z)). s ,zs This value represents the metric distance between the two inputs. It can be combined with the representation function H. θ And the quasi-metric function d, we obtain the end-to-end function d for calculating the quasi-metric distance from one state to another. θ . d θ (s,g) is used to fit the expected cost of reaching the target g from state s.

[0072] In one possible implementation, the quasimetric function is implemented using an Interval Quasimetric Embeddings (IQE) model.

[0073] In some examples, the computation process of the interval metric embedding model includes:

[0074] For the two latent space representations of the input, the dimensions of each latent space representation are divided into multiple preset categories;

[0075] For each category, compute the union of the intervals of the two latent space representations of the input, which are divided into the dimensions of that category;

[0076] Sum the lengths of the interval unions for all categories, and use the quasi-metric distance between the two latent space representations as input.

[0077] For example, when computing an interval metric embedding model, two inputs (e.g., z) can be pre-computed. s and z g All dimensions in the z-axis are divided into several categories. For example, some dimensions might describe location information, so they are grouped into one category; some might describe velocity information, so they are grouped into another category, and so on. Then, for each category of feature description, z is... s and z g The ranges spanned by the values ​​on the dimension assigned to that category are merged to obtain the union of intervals for that category. For example, for z... s and z g The ranges spanned by the values ​​in the dimension assigned to location information categories are merged, and z is... s and z g The ranges spanned by the values ​​assigned to the velocity information category are merged, and so on. Finally, the sum of the lengths of the interval unions corresponding to all categories is calculated, for example, the lengths of the interval unions of the location information category, the interval unions of the velocity information category, and so on are added together to obtain the distance as a standard metric.

[0078] Figure 4 A schematic diagram of an end-to-end model structure based on a quasi-metric according to an exemplary embodiment of the present disclosure is shown. As shown, the end-to-end model includes:

[0079] Input and encoding: the current state s and the goal state g are mapped to their respective latent space representations z s and z g respectively by an encoder (i.e., a representation function)

[0080] IQE model: includes a quasi-metric head and its subsequent modules. The latent space representations z s and z g are fed into the quasi-metric head module, go through the IQE model, and obtain the quasi-metric distance between the two latent space representations z s and z g .

[0081] Output d θ (s,g): after the processing of the representation function and the IQE model, a scalar value d θ (s,g) is finally output, which is d(z s ,z g ). This value is the quasi-metric distance from the state s to the goal g calculated sequentially through the representation function and the IQE model.

[0082] According to the above embodiments, the IQE model is used to realize the quasi-metric value function, and combined with its specific interval operation calculation process, a structured way is provided to define the distance from the state to the goal, and the complex relationships, uncertainties or range information in the state space can be better captured through interval representation and operation. At the same time, by using the above IQE model to meet the quasi-metric property, it is beneficial to learn a better cost evaluation function, so as to better perform action planning and improve system performance.

[0083] The constructed latent space dynamics model is used to receive the latent space representation of a state and an action applied to the state to predict the latent space representation of the next state.

[0084] The latent space dynamics model receives the latent space representation of a state (e.g., z s ) and an action (e.g., action a) selected by the agent as input, and predicts the latent space representation of the next state (e.g., z ) after the action is performed, thereby learning the state transition law of the environment.

[0085] The three models (the representation function, the quasi-metric value function, and the latent space dynamics model) are jointly trained using the transition data (state s, action a, next state s', reward r) sampled from the interaction with the environment and the corresponding target state g, which is a key step in the model learning of the present embodiment. To address the reward sparsity problem, in some implementations, a relatively simple reward r can be set, for example, at most time steps, the reward r can be a fixed negative constant (e.g., -1), representing the basic cost of performing an action; only when the next state s' exactly reaches or is very close to the specified target state g of the current round, the reward r can be changed to a different value (e.g., 0 or a positive value), which serves as a sparse signal of target achievement for optimizing the quasi-metric value function.

[0086] According to the present embodiment, the goal of training the latent space dynamics model is to minimize the difference between the predicted latent space representation of the next state z' and the latent space representation z' of the true next state, in particular, using the end-to-end quasi-metric value function d θ that is being learned.

[0087] In some implementations, in the joint training, when optimizing the quasi-metric value function using the reward r, the optimization goal is set as:

[0088]

[0089] where θ is the parameter of the quasi-metric value function, λ is the Lagrange multiplier, denotes the mathematical expectation, s ~ p s denotes that the sampled state s conforms to the probability distribution p s , g ~ p g denotes that the target state g conforms to the probability distribution p g , (s, a, s', r) ~ p transition denotes that the transition data (s, a, s', r) is sampled based on the real environment model p transition , d θ is an end-to-end function for calculating the quasi-metric distance, which is used to perform the process of mapping the input two states into corresponding latent space representations, respectively, and calculating the quasi-metric distance between the two latent space representations, φ is a pre-selected monotonically increasing convex function (e.g., a softplus function), ∈ 2 is a constant threshold value.

[0090] The present embodiment proposes a specific form of optimization goal for optimizing the end-to-end quasi-metric value function d θ . The optimization goal adopts a min-max form, and introduces a Lagrange multiplier λ to handle the constrained optimization problem, on the one hand, through the constraint item, encouraging the learned end-to-end function d θ can effectively distinguish different state-target pairs, so that the state-to-target cost estimation has discriminability; on the other hand, by the constraint item makes the model satisfy local consistency. By setting the optimization target as above, it can be ensured that the learned end-to-end function d θ not only reflects the global cost from state to target, but also coordinates with the local change and actual cost r in state transition, thereby providing a cost evaluation basis for subsequent planning that has both discriminability and high reliability.

[0091] In some embodiments, in the joint training, the following loss function is used Train the latent space dynamics model:

[0092]

[0093] wherein, represents the predicted next state latent space representation, z' represents the actual next state s' latent space representation, and d is the quasi-metric value function.

[0094] According to the present embodiment, a loss function for training the latent space dynamics model in joint training is further proposed The specific form of the loss function is as follows. The loss function is used to measure and minimize the difference between the predicted next time latent space representation and the actual next time latent space representation z', which does not use a general metric such as the standard Euclidean distance, but uses the quasi-metric value function d learned in the present embodiment to measure. By using d as the distance measurement standard, the learning process of the latent space dynamics model satisfies the geometric structure and cost relationship of the latent space defined by the quasi-metric value function, ensuring that the learned state transition model is consistent with the task goal (i.e. reaching the target with the lowest cost), thereby improving the effectiveness of subsequent planning decisions based on the model. The formula contains the sum of the square of the distance in two directions to further consider and adapt to the possible asymmetry of the quasi-metric function.

[0095] Joint training ensures that the model parameters can be optimized collaboratively, and the latent space representation and dynamics model that are beneficial to both prediction and cost estimation are learned.

[0096] Joint training can be implemented in a multi-step form to improve stability and efficiency. In some embodiments, in the joint training implemented in a multi-step form:

[0097] The optimization target of the quasi-metric value function adopts the following multi-step form:

[0098]

[0099] Loss function of the latent space dynamics model In the following multi-step form:

[0100]

[0101] where θ is the parameter of the quasi-metric function network, λ is the Lagrange multiplier, H is the preset step size, ρ i is the preset decay factor, φ is a pre-selected monotonic increasing convex function (such as a softplus function), d is a quasi-metric function used to calculate the quasi-metric distance between two latent space representations of the input, z t+i represents the latent space representation of the t+i-th step state of the sample, z g represents the latent space representation of the target state g, z t+i+1 represents the latent space representation of the t+i+1-th step state of the sample, represents the latent space representation of the predicted t+i+1-th step state, t is the starting time step of the current trajectory segment, r t+i is the immediate reward, ∈ 2 is a constant threshold.

[0102] The above embodiment further proposes a specific multi-step implementation form of the quasi-metric function optimization target and the latent space dynamics model loss function. The multi-step implementation has significant advantages compared to relying only on single-step information. For the optimization of the quasi-metric function, the multi-step form considers the cumulative cost on the future H-step trajectory and the consistency of multi-step transitions, so as to more effectively allocate time credits and more quickly back-propagate long-term target information, thereby improving the ability to learn the cost function in sparse reward or long-time sequence tasks; for the training of the latent space dynamics model, the multi-step form not only makes the model more accurate in single-step prediction, but also enables the model to maintain a lower cumulative error in continuous H-step prediction, which helps to learn more robust and long-term consistent environmental transition rules and reduces the problem of error accumulation in model prediction. Therefore, joint training using the multi-step form proposed in this embodiment can promote more stable and efficient learning and provide a more reliable model basis for subsequent planning and decision-making relying on multi-step prediction.

[0103] In some embodiments, the multi-step form implements joint training, which can specifically include:

[0104] Sampling a trajectory segment of transition data with a preset step length from an experience replay buffer;

[0105] Mapping the states in the trajectory segment to the latent space using the representation function to obtain a sequence of mapped latent space representations;

[0106] In the latent space, a latent space dynamics model is used to predict a sequence of latent space representations of a future preset step length;

[0107] Based on the mapped sequence of latent space representations and the predicted sequence of latent space representations, an optimization objective and a loss function in a multi-step form are used to jointly train the representation function, the quasi-metric value function, and the latent space dynamics model.

[0108] According to the embodiment, a specific implementation process of the joint training in a multi-step form is further proposed. In this way, the multi-step dependency information and the long-term prediction capability evaluation of the model are directly introduced into the training, so that the gradient can be propagated based on longer time series information, which helps to alleviate the short-sighted problem, learn more accurate and more predictive latent space dynamics models and quasi-metric value functions, and thus enhance the learning efficiency and the final performance of the overall method.

[0109] Step 2, action selection optimization is performed using model predictive control (MPC) combined with cross-entropy method (CEM), wherein in the MPC process, the current trained latent space dynamics model is used to predict a future latent space trajectory, the quasi-metric distance from the final latent space representation of the predicted latent space trajectory to the latent space representation of the target state is calculated using the current trained quasi-metric value function as a cost evaluation standard, and a single action applied to the current state is determined according to the lowest cost action planning result obtained by CEM optimization.

[0110] When the model is trained or during the training process, the agent needs to make a decision, and a model-based planning process can be performed. It should be noted that reinforcement learning is usually a continuous learning and interaction process with the environment. The agent needs to make decisions and take actions while learning to collect new experience data to further improve its model or strategy, and usually cannot wait until the model is "fully trained" before taking action. Therefore, step 2 does not strictly occur after step 1 is completely finished, but can be interleaved with step 1.

[0111] In some embodiments, action selection optimization using model predictive control (MPC) combined with cross-entropy method (CEM) can include:

[0112] The following steps are iteratively performed for the current state until a preset condition is met:

[0113] N candidate action sequences of a preset step length are sampled from the current action sampling distribution, and N is a preset number of candidate action sequences;

[0114] Based on the current state, the latent space trajectory corresponding to each candidate action sequence is predicted using the current trained latent space dynamics model;

[0115] Calculate the quasi-metric distance between the final latent space representation of each predicted latent space trajectory and the latent space representation of the target state using the currently trained quasi-metric value function as the cost;

[0116] Select K elite action sequences with the lowest cost according to the calculated cost, where K is a preset value;

[0117] Update the parameters of the action sampling distribution based on the selected elite action sequences.

[0118] In some embodiments, action sequences with a preset step size can be randomly sampled from a preset probability distribution (such as a Gaussian probability distribution with preset parameters); in other embodiments, action sequences with a preset step size can be output using a trained policy network.

[0119] For each candidate action sequence, a trained latent space dynamics model can be used for forward simulation to predict the latent space trajectory generated by executing this action sequence within a preset future step size H, and the currently trained quasi-metric value function can be used to evaluate the final latent space representation of each latent space trajectory The quasi-metric distance of the latent space representation z reaching the target state g of this iteration g That is, calculate as the cost of this latent space trajectory. The lower the cost, the more likely this trajectory is to lead to the target state g. Then, a preset number of elite action sequences with the lowest cost can be selected according to the calculated cost, such as the lowest-cost K sequences as the elite action sequences, 0 < K < N. Finally, based on the selected elite action sequences, the system can update the parameters of the action sampling distribution, such as the mean μ and variance σ of the Gaussian distribution, so that in the next iteration, it is more likely to sample actions similar to these elite action sequences and perform better.

[0120] The above "sampling - simulation - evaluation - update" loop is repeated several times to continuously optimize the action sampling distribution, and finally an optimal action region is obtained. After the iteration ends, usually the mean of the final distribution or the first action in the sequence with the highest score can be selected as the optimal decision output in the current state and executed in the actual environment. The entire MPC-CEM process makes full use of the learned latent space dynamics model and quasi-metric value function to make predictive decisions through internal simulation and optimization.

[0121] Figure 5 Shows a schematic diagram of an online action selection process based on model predictive control and cross-entropy method (MPC-CEM) according to an exemplary embodiment of the present disclosure.

[0122] This process starts with obtaining the current environmental state s t , and using the currently trained characterization function h θconverts it into a corresponding latent space representation z t . The latent space representation z t contains the key information needed for future prediction and is the starting point for the subsequent planning step.

[0123] Next, an iterative optimization loop of MPC-CEM is started. First, N candidate future action sequences are sampled from the Gaussian distribution with superscript j representing the iteration number of the optimization loop inside the Cross-Entropy Method (CEM). Then, the current trained latent space dynamics model is used to perform a forward simulation of each candidate action sequence for H steps into the future to predict the corresponding latent space state trajectory. Next, the trained quasi-metric value function d is used to evaluate the cost of each predicted latent space state trajectory, i.e., to compute the quasi-metric distance of its final predicted state to the goal state as the cost. According to the evaluation results, the K elite action sequences with the lowest cost (i.e., the highest expected return) are selected.

[0124] In some embodiments, an action value function Q * (s, a; g) that satisfies the following approximation relationship can also be derived based on the current trained latent space dynamics model and the quasi-metric value function d:

[0125]

[0126] where d is the quasi-metric value function, z g is the latent space representation of the predicted next state, z is the latent space representation of the goal state g, and r is the reward.

[0127] The derived action value function is used to evaluate the cost of performing different actions for updating the policy network.

[0128] The present embodiments introduce an action value function (Q-function) and approximate it based on the already learned latent space dynamics model and the quasi-metric value function. Through this approximation relationship, the existing model components can be used to evaluate the expected total cost or negative value of performing a certain action in a specific state without the need to learn a complex Q-network independently. The evaluation results can be used to update the explicit policy network. For example, in the Actor-Critic framework, the gradient information of action a with respect to the Q-function, or the advantage function calculated based on the Q-value, can be used to guide the optimization of the policy network (Actor) parameters to make them more inclined to output actions that can bring lower long-term expected cost (i.e., higher Q-value corresponding to negative cost), thereby further improving the learning ability.

[0129] Through the above model construction, joint training, and model-based planning decision process, the multi-objective reinforcement learning method based on quasi-metric and planning proposed by the present disclosure can effectively utilize limited interaction data to learn the environment model (such as the latent space dynamics model) and the goal-oriented cost function (such as the quasi-metric value function), and overcome the challenge of sparse rewards through forward planning, and finally achieve efficient learning and decision-making in multi-objective reinforcement learning tasks.

[0130] The following describes one application example of the multi-objective reinforcement learning method based on quasi-metric and planning proposed by the present disclosure.

[0131] Step one, model construction and joint training.

[0132] In the multi-objective reinforcement learning environment, a quasi-metric-based network model is designed and jointly trained. The network model mainly includes four parts: a representation function, a quasi-metric value function, a latent space dynamics model, and an optional policy network. The transition data sampled from the environment interaction is used to train these models.

[0133] S11, design of quasi-metric value function

[0134] Based on the equivalence between the (negative) optimal state value function and the quasi-metric in the multi-objective reinforcement learning environment, the quasi-metric value function is designed to fit the negative optimal state value function. First, two related functions are defined:

[0135] End-to-end quasi-metric function d θ (s, g): receives the original state s and the goal g, and outputs the final distance, which includes representation and latent space quasi-metric distance calculation in the calculation;

[0136] Latent space quasi-metric distance function d(z1, z2): receives two latent space representations z1, z2, and calculates the quasi-metric distance between them.

[0137] In this example, an interval quasi-metric embedding (IQE) model is used to implement the latent space quasi-metric distance function d(z1, z2).

[0138] S111, representation function (h θ ) construction

[0139] The representation function h θ is usually composed of a basic neural network, which is used to map s, g, etc. in the state space to the latent space to obtain the corresponding latent space representation z s = h θ (s), z g = h θ (g). Its specific structure is selected according to the type of input state. If the state space is a pixel space, h θ is composed of a CNN backbone network (such asFigure 2 (as shown) and FNN (as shown) Figure 3 (as shown) constitutes the state space; if the state space is a vector space, h θ Composed of FNN (e.g.) Figure 3 (As shown).

[0140] S112, IQE Calculation Process

[0141] When using the IQE model, the latent space metric distance function d(z) s ,z g The computation process may include: for the two latent space representations z of the input... s z g The dimensions are divided into multiple categories; for each category, a z-based algorithm is calculated. s z g Find the union of intervals corresponding to the dimension values; sum the lengths of the unions of intervals obtained for all categories to obtain the final metric distance d(z). s ,z g ).

[0142] S113, Optimization objective of the metric function (single step)

[0143] Based on the network model constructed above, and utilizing transfer data sampled from environmental interactions, the end-to-end accuracy metric function d can be trained by optimizing the following single-step optimization objective. θ (including h) θ and d):

[0144]

[0145] The meaning of the parameters can be found in the relevant description above.

[0146] S12, Latent Space Dynamics Model Training (Single Step)

[0147] Using the learned latent space metric distance function d as a standard, a latent space dynamics model is trained in the latent space to fit the environment transition. The true next latent space representation z′ is obtained by the action of the representation function on the next state s′, i.e., z′=h θ (s′). Measures the next latent space representation predicted by the latent space dynamics model. One-step loss function for the difference between the real next hidden space representation z′ and the actual next hidden space representation z′. for:

[0148]

[0149] The meaning of the parameters can be found in the relevant description above.

[0150] S13. Approximation and Application of the Action Value Function (Q-function)

[0151] Based on the learned latent space dynamics model and latent space pseudo-metric distance function d, the action value function Q*(s, a; g) can be approximately calculated:

[0152]

[0153] The parameter meanings in the formula can be referred to the relevant description above. The calculated Q*(s, a; g) is mainly used to update the policy network.

[0154] S14, Multi-step joint training (preferred)

[0155] The training process preferably adopts multi-step joint training. Trajectory segments s t ,s t+1 ,…,s t+H of environment interaction transition data with a step size of H can be sampled from an experience replay buffer.

[0156] S141, Multi-step joint training process and target

[0157] The multi-step joint training process includes: using the representation function to map the trajectory state to the latent space to obtain the mapped latent space representation sequence z t ,z t+1 ,…,z t+H ; using the latent space dynamics model and trajectory action prediction to obtain the predicted latent space representation sequence Then, the following multi-step form of optimization target and loss function is used for joint training:

[0158] Multi-step optimization target of pseudo-metric value function (based on S113 extension):

[0159]

[0160] Multi-step loss function of latent space dynamics model (based on S12 extension):

[0161]

[0162] The parameter meanings can be referred to the relevant description above.

[0163] S142, Policy network update

[0164] If the policy network is included, the gradient based on the action value function Q*(approximated by S13) or other policy optimization methods can be used to update the parameters of the policy network in the joint training (or alternately).

[0165] Step two, model-based planning decision and interaction (MPC-CEM)

[0166] Based on the current trained model, model predictive control (MPC) combined with cross-entropy method (CEM) is used for forward planning and action selection.

[0167] S21, Planning procedure overview

[0168] The current state s t is mapped to a latent space representation z t using the current trained representation function. Figure 5

[0169] S211, Generating candidate action sequences

[0170] N candidate action sequences of length H are generated. They can be randomly sampled from the probability distribution (e.g. Gaussian distribution maintained by CEM; and / or outputted by the current trained policy network.

[0171] S212, Predicting latent space trajectories

[0172] Using the initial latent space representation z t and the N candidate action sequences generated in S211, the current trained latent space dynamics model is iterated H times to obtain N predicted latent space trajectories

[0173] S213, Trajectory cost evaluation

[0174] The terminal predicted latent space representation of each predicted trajectory is calculated using the current trained latent space quasi-metric distance function d to the target latent space representation z g as the cost score of this latent space trajectory. The lower the cost, the better.

[0175] S22, CEM update step

[0176] Based on the trajectory cost scores obtained in step S213, the update step of cross-entropy method (CEM) is performed: select the K elite action sequences with the lowest cost, and use them to update the parameters (e.g. mean and variance of Gaussian distribution) of the action sampling distribution, so that in the next (j+1) iteration, it is more inclined to sample action sequences with lower cost.

[0177] S23, Final action selection

[0178] ​The iterative steps (S211 sampling, S212 prediction, S213 evaluation, S22 update) of CEM are repeated several times. After the iteration, the optimal action sequence is determined according to the final action distribution (for example, the mean) or the sequence with the lowest score, and the first action of the sequence can be output t as a final decision for interacting with the environment.

[0179] Through the implementation of the above steps, the present example can effectively learn and utilize the model to achieve efficient planning and decision-making in a multi-objective environment with sparse rewards.

[0180] Other details and benefits of the present embodiment are described above, and will not be repeated here.

[0181] Although the present specification contains many specific implementation details, these should not be construed as limiting the scope or the scope of protection of any invention, but are mainly used to describe the features of the specific embodiments of the particular invention. Some features described in the present specification in multiple embodiments can also be implemented in a single embodiment. On the other hand, various features described in a single embodiment can also be implemented separately in multiple embodiments or in any suitable sub-combination. In addition, although the features can function as described above in some combinations and even initially claimed as such, one or more features from the claimed combination can be removed from the combination in some cases, and the claimed combination can refer to a sub-combination or a variation of a sub-combination.

[0182] Similarly, although the operations are depicted in a particular order in the drawings, this should not be understood as requiring the operations to be performed in the particular order shown or sequentially, or requiring all of the illustrated operations to be performed to achieve the desired result. In some cases, multi-tasking and parallel processing can be advantageous. In addition, the separation of various system modules and components in the above embodiments should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product, or packaged into multiple software products.

[0183] Thus, particular embodiments of the subject matter have been described. Other embodiments are within the scope of the claims appended hereto. In some cases, the actions recited in the claims can be performed in a different order and still achieve desirable results. In addition, the processes depicted in the drawings do not necessarily require the particular order or sequential order shown to achieve the desired results. In some implementations, multi-tasking and parallel processing can be advantageous.

[0184] The above description is only the preferred embodiment of one or more embodiments of the specification, and is not used to limit one or more embodiments of the specification. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of one or more embodiments of the specification should be included in the protection range of one or more embodiments of the specification.

Claims

1. A quasi-metric and planning based multi-objective reinforcement learning method, characterized in that, Comprising: constructing a representation function, a pseudo-metric value function and a latent space dynamics model, and jointly training the representation function, the pseudo-metric value function and the latent space dynamics model using transition data (s, a, s', r) and a target state g sampled from environment interaction; wherein the representation function is used to map the state sampled from the environment interaction to the corresponding latent space representation; the latent space dynamics model is used to predict the latent space representation of the next state based on the latent space representation of the current state s and the action a applied to the state; the pseudo-metric value function is used to calculate the pseudo-metric distance between the two input latent space representations, and during the training of the latent space dynamics model, it is used as the distance measurement standard between the predicted latent space representation of the next state and the latent space representation of the sampled next state s'; the reward r is used to optimize the pseudo-metric value function; using model predictive control (MPC) and combining cross-entropy method (CEM) to optimize action selection, wherein in the MPC process, the current trained latent space dynamics model is used to predict the future latent space trajectory, the current trained pseudo-metric value function is used to calculate the pseudo-metric distance between the final latent space representation of the predicted latent space trajectory and the latent space representation of the target state as the cost evaluation standard, and the single action applied to the current state is determined according to the lowest cost action planning result obtained by CEM optimization.

2. The method of claim 1, wherein: if the state of the environment is in the form of a pixel space, the representation function is composed of a convolutional neural network (CNN) backbone network and a feedforward neural network (FNN); if the state of the environment is in the form of a vector space, the representation function is composed of a feedforward neural network (FNN).

3. The method of claim 1, wherein, The pseudo-metric value function is implemented using an interval pseudo-metric embedding model.

4. The method of claim 3, wherein, The calculation process of the interval pseudo-metric embedding model includes: for the two input latent space representations, the dimensions of each latent space representation are divided into a plurality of preset categories; for each category, the interval union of the values of the two input latent space representations in the dimensions of the category is calculated; the lengths of the interval unions of all categories are summed up as the pseudo-metric distance between the two input latent space representations.

5. The method of claim 1, wherein, In the joint training, when the reward r is used to optimize the pseudo-metric value function, the optimization target is set as: where θ is the parameter of the quasi-metric value function, λ is the Lagrange multiplier, denotes the mathematical expectation, s ~ p s denotes that the sampling state s conforms to the probability distribution p s , g ~ p g denotes that the target state g conforms to the probability distribution p g , (s, a, s', r) ~ p transition denotes that the transition data (s, a, s', r) is sampled based on the real environment model p transition , d θ is an end-to-end function for calculating the quasi-metric distance, which is used to perform the process of respectively mapping two input states into corresponding hidden space representations and calculating the quasi-metric distance between the two hidden space representations, φ is a pre-selected monotone increasing convex function, ∈ 2 is a constant threshold value.

6. The method of claim 1, wherein, In joint training, the following loss function is used Training the latent space dynamics model: wherein, z' represents the latent space representation of the true next state s', and d is a quasi-metric value function.

7. The method of claim 1, wherein, The joint training is implemented in a multi-step form, wherein: The optimization target of the pseudo-metric value function adopts the following multi-step form: Loss function for latent space dynamics model In the following multi-step form: where θ is the parameter of the quasi-metric value function network, λ is the Lagrange multiplier, H is the preset step size, ρ i is the preset attenuation factor, φ is the pre-selected monotone increasing convex function, d is the quasi-metric value function, z t+i represents the hidden space representation of the t+i-th step state of the sample, z g represents the hidden space representation of the target state g, z t+i+1 represents the hidden space representation of the t+i+1-th step state of the sample, represents the hidden space representation of the predicted t+i+1-th step state, t is the starting time step of the current trajectory segment, r t+i is the immediate reward, ∈ 2 is the constant threshold value.

8. The method according to claim 1 or 7, characterized in that, The joint training is implemented in a multi-step form, including: sampling a trajectory segment of transition data with a preset step length from the experience replay buffer; mapping the states in the trajectory segment to the latent space using the representation function to obtain a sequence of mapped latent space representations; iteratively predicting a sequence of latent space representations of the future preset step length in the latent space using the latent space dynamics model; based on the sequence of mapped latent space representations and the sequence of predicted latent space representations, jointly training the representation function, the pseudo-metric value function and the latent space dynamics model using a multi-step optimization target and a loss function.

9. The method of claim 1, wherein, Using model predictive control (MPC) and combining cross-entropy method (CEM) to optimize action selection, including: The following steps are performed for the current state iteratively until a preset condition is met: N candidate action sequences of a preset step length are sampled from the current action sampling distribution, N being a preset number of candidate action sequences; Based on the current state, the current trained latent space dynamics model is used to predict the latent space trajectory corresponding to each candidate action sequence; The final latent space representation of each predicted latent space trajectory is calculated using the current trained quasi-metric value function, and the quasi-metric distance from the target state is used as the cost; K elite action sequences with the lowest cost are selected according to the calculated cost, K being a preset value, 0 < K < N; The parameters of the action sampling distribution are updated based on the selected elite action sequences.

10. The method of claim 9, wherein, Sampling N candidate action sequences from the current action sampling distribution includes: Randomly sampling an action sequence of a preset step length from a preset probability distribution, or sampling an action sequence of a preset step length using a trained policy network.

11. The method according to claim 1 or 10, characterized in that, The method further includes: Based on the current trained hidden space dynamics model and the quasi-metric value function, an action value function Q is extended to satisfy the following approximate relationship * (s, a; g): where d is a quasi-metric value function, z represents a hidden space representation of the predicted next state, g r is a reward; and g represents a hidden space representation of the target state. The obtained action value function is used to evaluate the cost of performing different actions for updating the policy network.

Images