Multi-target reinforcement learning method based on accuracy and planning

Abstract

The invention discloses a multi-target reinforcement learning method based on quasi-measurement and planning, which comprises the steps of constructing a representation function, a quasi-measurement value function and a hidden space dynamics model, and performing joint training by utilizing transfer data sampled from environment interaction and a target state, the quasi-measurement value function being optimized to fit the cost from the state to the target, and the quasi-measurement value function being optimized to fit the target state; the distance measurement standard is used for optimizing the hidden space dynamic model; during decision making, action selection optimization is carried out by utilizing an MPC framework and combining with a CEM, in the process, a future trajectory is predicted by utilizing a hidden space dynamics model, the cost of the predicted trajectory is evaluated by using a quasi-metric function, and a single action applied to a current state is determined according to a lowest-cost planning result obtained by CEM optimization. According to the method, by learning the quasi-metric cost function and the hidden space model and combining model prediction control planning, sparse rewards can be effectively dealt with, and the sample efficiency and final performance of multi-target reinforcement learning are remarkably improved.

Description

Technical Field

[0001] The present disclosure relates to the field of reinforcement learning of artificial intelligence, and in particular to a multi-objective reinforcement learning method based on quasi-metrics and planning. Background Art

[0002] Artificial intelligence, a key driving force behind the new round of scientific and technological revolution and industrial transformation, is experiencing rapid development driven by explosive growth in data volumes, significant increases in computing power, and continuous breakthroughs in algorithmic models. Reinforcement learning (RL), a key branch of AI, focuses on studying how intelligent agents learn optimal decision-making strategies through autonomous interaction with their environments. In recent years, RL has demonstrated capabilities that surpass human experts in many fields. Examples include the breakthroughs achieved by Google DeepMind's AlphaGo system in Go and the OpenAI Five system developed by the OpenAI team, which achieved state-of-the-art performance in the complex multiplayer online game Dota 2. These achievements demonstrate the enormous potential of RL technology.

[0003] Traditional reinforcement learning algorithms rely heavily on reward functions carefully designed by domain experts based on their specialized knowledge to guide the policy optimization process. However, in many real-world applications, especially in complex robotic systems or interactive tasks, designing a dense reward function that accurately reflects the task objectives and is easy to optimize is often extremely difficult or even impractical.

[0004] In this context, multi-objective reinforcement learning (MORL), especially goal-conditional reinforcement learning (GCL), is particularly important. In a typical multi-objective reinforcement learning environment, the reward function is usually simple to set. The agent may face different goal states in each interaction episode, and the reward signal provided by the environment is often very sparse. Typically, the environment only provides a simple signal to indicate whether the current state has reached the goal state specified in this episode. The agent must use this extremely sparse feedback signal to learn how to explore and master the methods to reach different goal states in a complex environment, which constitutes the core challenge of this field.

[0005] Currently, academia and industry have proposed a number of algorithms to address the challenges of multi-objective reinforcement learning. For example, strategies based on Hindsight Experience Replay (HER) or relabeling treat interaction trajectories that fail to reach the intended goal as successful trajectories that actually reach a certain state (as the subsequent goal) in the successful trajectory. By "creating" successful experiences in this way, the algorithm can learn strategies for transitioning between different states and rely on the generalization ability of neural networks to master universal strategies for reaching different goals. Another type of method draws on the idea of ​​contrastive learning. For example, it treats state pairs collected within the same interaction round as positive samples and state pairs from different rounds as negative samples. In this way, a value function or representation that reflects the reachability between states is learned, thereby guiding policy learning.

[0006] Although the aforementioned methods have alleviated the learning difficulties caused by sparse rewards in multi-objective reinforcement learning to a certain extent, they often suffer from low sample efficiency. In other words, the agent often needs to interact extensively with the environment and collect massive amounts of experience data in order to learn effective strategies. In contrast, model-based reinforcement learning (MBRL) methods have shown potential for improving sample efficiency. By learning an environment model (Dynamics Model) to approximate the state transitions and reward mechanisms of the real environment, the agent can use this model for internal simulation and planning. The model can extract more useful information from limited real-world interaction data and allows the agent to explore and evaluate different strategies "in imagination", thereby more effectively learning strategies even in the absence of immediate reward signals.

[0007] Therefore, how to effectively overcome the defects of sparse rewards and low sample efficiency in multi-objective reinforcement learning is an important and valuable research direction in the current field of reinforcement learning. Summary of the Invention

[0008] This paper proposes a multi-objective reinforcement learning method based on quasi-metrics and planning, which can combine the advantages of the model and effectively overcome the defects of sparse rewards and low sample efficiency in multi-objective reinforcement learning.

[0009] According to one embodiment of the present disclosure, a multi-objective reinforcement learning method based on quasi-metrics and planning is proposed, including:

[0010] Construct a representation function, a quasi-metric function, and a latent space dynamics model, and jointly train the representation function, the quasi-metric function, and the latent space dynamics model using the transition data (s, a, s', r) and the target state g sampled from the environmental interaction; wherein the representation function is used to map the state sampled from the environmental interaction to the corresponding latent space representation; the latent space dynamics model is used to predict the latent space representation of the next state based on the latent space representation of the current state s and the action a applied to the state; the quasi-metric function is used to calculate the quasi-metric distance between the two latent space representations of the input, and is used as a distance metric between the latent space representation of the predicted next state and the latent space representation of the sampled next state s' when training the latent space dynamics model; the reward r is used to optimize the quasi-metric function;

[0011] Model predictive control (MPC) is used in combination with the cross entropy method (CEM) to optimize action selection. In the MPC process, the current trained latent space dynamics model is used to predict the future latent space trajectory, and the current trained quasi-metric value function is used to calculate the quasi-metric distance from the final latent space representation of the predicted latent space trajectory to the latent space representation of the target state as a cost evaluation criterion. Based on the minimum cost action planning result obtained by CEM optimization, a single action applied to the current state is determined.

[0012] In some embodiments, if the state of the environment is in pixel space form, the representation function is composed of a convolutional neural network (CNN) backbone network and a feedforward neural network (FNN); if the state of the environment is in vector space form, the representation function is composed of a feedforward neural network (FNN).

[0013] In some implementations, the quasi-metric value function is implemented using an interval quasi-metric embedding model.

[0014] In some embodiments, the calculation process of the interval quasi-metric embedding model includes:

[0015] For the two latent space representations of the input, the dimensions of each latent space representation are divided into multiple preset categories;

[0016] For each category, calculate the interval union of the values ​​of the two latent spaces representing the input divided into the dimension of the category;

[0017] The length of the union of intervals of all categories is summed up as the quasi-metric distance between the two latent space representations of the input.

[0018] In some embodiments, during joint training, when optimizing the quasi-metric function using the reward r, the optimization objective is set to:

[0019]

[0020] Among them, θ is the parameter of the quasi-metric function, λ is the Lagrange multiplier, represents mathematical expectation, s~p s Indicates that the sampling state s conforms to the probability distribution p s , g~p g Indicates that the target state g conforms to the probability distribution p g ,(s,a,s',r)~p transition Indicates that the transferred data (s,a,s',r) is based on the real environment model p transition Sampling, d θ It is an end-to-end function for calculating the quasi-metric distance, which is used to perform the process of mapping the two input states into corresponding latent space representations and calculating the quasi-metric distance between the two latent space representations. φ is a pre-selected monotonically increasing convex function, ∈ 2 is a constant threshold.

[0021] In some embodiments, during joint training, the following loss function is used: Train the latent space dynamics model:

[0022]

[0023] in, represents the latent space representation of the predicted next state, z′ represents the latent space representation of the true next state s′, and d is the quasi-metric function.

[0024] In some embodiments, joint training is implemented in a multi-step manner, wherein:

[0025] The optimization objective of the quasi-metric function takes the following multi-step form:

[0026]

[0027] Loss function for latent space dynamics models Take the following multi-step form:

[0028]

[0029] Among them, θ is the parameter of the quasi-metric function network, λ is the Lagrange multiplier, H is the preset step size, ρ i is the preset attenuation factor, φ is a pre-selected monotonically increasing convex function, d is the quasi-metric function, z i+i represents the latent space representation of the sampled t+i-th step state, z g represents the latent space representation of the target state g, z t+i+1 represents the latent space representation of the sampled t+i+1th step state, Represents the latent space representation of the predicted t+i+1th step state, t is the starting time step of the current trajectory segment, r t+i is the immediate reward, ∈ 2 is a constant threshold.

[0030] In some embodiments, joint training is implemented in multiple steps, including:

[0031] Sampling trajectory segments of transfer data of a preset step length from the experience replay buffer;

[0032] Use the representation function to map the state in the trajectory segment to the latent space and obtain the mapped latent space representation sequence;

[0033] Iteratively use the latent space dynamics model in the latent space to predict the latent space representation sequence of the future preset step length;

[0034] Based on the mapped latent space representation sequence and the predicted latent space representation sequence, a multi-step optimization objective and loss function are used to jointly train the representation function, quasi-metric function and latent space dynamics model.

[0035] In some embodiments, model predictive control (MPC) is used in conjunction with a cross entropy method (CEM) to optimize action selection, including:

[0036] Iterate the following steps for the current state until the preset conditions are met:

[0037] Sample N candidate action sequences with preset step lengths from the current action sampling distribution, where N is the preset number of candidate action sequences;

[0038] Based on the current state, the currently trained latent space dynamics model is used to predict the latent space trajectory corresponding to each candidate action sequence;

[0039] Use the currently trained quasi-metric value function to calculate the quasi-metric distance from the final latent space representation of each predicted latent space trajectory to the latent space representation of the target state as the cost;

[0040] According to the calculated cost, K elite action sequences with the lowest cost are selected, where K is the preset value, 0 <K<N;

[0041] Parameters of the action sampling distribution are updated based on the selected elite action sequence.

[0042] In some implementations, sampling N candidate action sequences from the current action sampling distribution includes:

[0043] Randomly sample action sequences of preset step lengths from a preset probability distribution, or use a trained policy network to sample action sequences of preset step lengths.

[0044] In some embodiments, the method further comprises:

[0045] Based on the currently trained latent space dynamics model and quasi-metric value function, the action value function Q is expanded to satisfy the following approximate relationship * (s,a;g):

[0046]

[0047] Where d is the quasi-metric function, The latent space representation of the predicted next state, z g represents the latent space representation of the target state g, and r is the reward;

[0048] The obtained action value function is used to evaluate the cost of performing different actions to update the policy network.

[0049] The present disclosure provides a multi-objective reinforcement learning method based on quasi-metrics and planning. By combining model construction and training with model-based planning and decision-making, the method achieves significant benefits. First, by constructing and jointly training a representation function, a quasi-metric value function, and a latent space dynamics model, the present disclosure effectively addresses the reward sparsity problem commonly found in multi-objective reinforcement learning. The quasi-metric value function directly fits the distance from the state to the goal, fully utilizing the immediate rewards (typically sparse) provided by the environment and learning in conjunction with a specific optimization objective. This allows the construction of an internal cost d that reflects the goal's reachability. Simultaneously, the learned latent space dynamics model enables the agent to predict future state transitions in the latent space. Second, through model-based planning, decision-making, and interaction, forward simulation prediction using the trained model and evaluation based on the learned quasi-metric cost d, sample efficiency can be significantly improved. At each decision step, the agent can explore multiple future trajectories through internal simulation and evaluate their pros and cons based on the quasi-metric value function, thereby making more predictive decisions, reducing reliance on large amounts of real-world interaction data and accelerating the learning process of effective strategies. This paper proposes a framework that combines learning and planning, providing an effective way to perform efficient multi-objective reinforcement learning in sparse reward environments.

[0050] Furthermore, in different embodiments of the present disclosure, by adopting the interval quasimetric embedding (IntervalQuasimetric Embeddings) model as the implementation method of the quasimetric value function, a good foundation is provided for learning the cost function that satisfies the quasimetric property; by adopting specific optimization objectives and loss functions for joint training, especially adopting a specially designed multi-step form, it is possible to more stably and effectively propagate value information and learn dynamic models, further improving the performance and robustness of learning, especially when dealing with long time-dependent tasks; in the planning stage, optimization algorithms such as the cross entropy method (CEM) are adopted to effectively search and find the optimal action sequence among multiple candidate trajectories, thereby improving the quality of planning decisions.

[0051] Further details and advantages of the present disclosure are detailed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the specification and, together with the description, serve to explain the principles of the specification.

[0053] Figure 1 A flowchart of a multi-objective reinforcement learning method based on quasi-metrics and planning according to an embodiment of the present disclosure is shown.

[0054] Figure 2 A schematic diagram of a CNN backbone network for implementing a characterization function model according to an exemplary embodiment of the present disclosure is shown.

[0055] Figure 3 A schematic diagram of an FNN for implementing a characterization function model according to an exemplary embodiment of the present disclosure is shown.

[0056] Figure 4 A schematic diagram of an end-to-end model structure based on quasi-metrics according to an exemplary embodiment of the present disclosure is shown.

[0057] Figure 5 A schematic diagram of a planning process based on model predictive control and cross entropy method (MPC-CEM) according to an exemplary embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0058] Exemplary embodiments will be described in detail herein, with examples illustrated in the accompanying drawings. In the following description, when referring to the drawings, identical numerals in different figures represent identical or similar elements, unless otherwise indicated. The embodiments described in the following exemplary embodiments are not intended to represent all possible embodiments consistent with the present disclosure. Rather, they are merely examples of apparatus and methods consistent with certain aspects of the present disclosure, as detailed in the appended claims.

[0059] This disclosure provides a multi-objective reinforcement learning method based on quasi-metrics and planning, which is used to solve the problems of sparse rewards and low sample efficiency that are prevalent in this field. Figure 1 FIG. 1 shows a flowchart of a multi-objective reinforcement learning method based on quasi-metrics and planning according to an embodiment of the present disclosure. Figure 1 As shown in Figure 1, the method mainly includes two core steps: model construction and training (step 1) and model-based planning decision-making and interaction (step 2).

[0060] Step 1: Construct a representation function, a quasi-metric function, and a latent space dynamics model, and jointly train the representation function, the quasi-metric function, and the latent space dynamics model using the transfer data (s, a, s', r) and the target state g sampled from the environmental interaction; wherein the representation function is used to map the state sampled from the environmental interaction to the corresponding latent space representation; the latent space dynamics model is used to predict the latent space representation of the next state based on the latent space representation of the current state s and the action a applied to the state; the quasi-metric function is used to calculate the quasi-metric distance between the two latent space representations of the input, and when training the latent space dynamics model, it is used as a distance metric between the latent space representation of the predicted next state and the latent space representation of the sampled next state s'; the reward r is used to optimize the quasi-metric function.

[0061] In reinforcement learning, "environment" refers to the system in which the agent resides and interacts. The agent influences the environment by performing actions and receives feedback in the form of the next state s' and reward r. This continuous process of action and feedback is called environment interaction. "State" is a specific description of the environment at a specific moment, containing the relevant information required for the agent's decision-making. "State sampled from environment interaction" refers to the environment state s at a specific moment experienced and recorded during the agent's actual interaction with the environment. This state data forms the basis for subsequent random sampling (sampling) for training and learning. Reward r is a scalar feedback signal provided by the environment itself according to preset rules after the agent performs an action a to transition from the current state s to the next state s'.

[0062] In step 1, a series of interrelated neural network models are first constructed and jointly trained using interaction data with the environment. The following describes the representation function, quasi-metric function, and latent space dynamics model constructed according to this embodiment.

[0063] The constructed representation function is used to map the states sampled from the environment interaction to the corresponding latent space representation.

[0064] Characterization function H θIt is used to map the original state s sampled by the agent from the environment interaction and the target state g of the current task to a more information-dense latent space representation z s and z g The latent space representation is used to capture features that are important for subsequent tasks such as cost estimation and dynamics prediction.

[0065] In some embodiments, if the state of the environment is in pixel space, the representation function is composed of a convolutional neural network (CNN) backbone network and a feedforward neural network (FNN).

[0066] For example, if the input is an image or video frame from a camera, corresponding to an image-based environment, the representation function can be composed of a convolutional neural network backbone network CNN and a feedforward neural network FNN. Figure 2 The convolutional layer shown in Figure 1 is responsible for extracting spatial features from high-dimensional pixel data, and its output feature map is then flattened and passed through one or more fully connected layers (such as Figure 3 The FNN shown in the figure is further processed to finally generate a low-dimensional latent space vector representation.

[0067] In some embodiments, if the state of the environment is in a vector space form, the representation function is formed by a feedforward neural network (FNN).

[0068] For example, if the input is a feature vector consisting of robot joint angles, velocities, object coordinates, etc., corresponding to a vector-based environment, the representation function can be directly constructed by a feedforward neural network (FNN) (e.g. Figure 3 The FNN is shown in Figure 1). The FNN transforms and abstracts the input feature vector through a series of fully connected layers and nonlinear activation functions (such as ReLu), extracts higher-level features, and outputs the final latent space vector representation.

[0069] The characterization function H constructed by the above implementation method θ , which can effectively process different types of input states and provide high-quality, unified format latent space representation for subsequent quasi-metric function calculation and latent space dynamics modeling.

[0070] The constructed quasi-metric value function is used to calculate the quasi-metric distance between the two latent space representations of the input.

[0071] Based on the theory of equivalence between (negative) optimal value function and quasi-metric in multi-objective reinforcement learning, this paper proposes to construct a quasi-metric value function d θ The function receives two inputs (e.g. z s and z g ), and outputs a scalar value (e.g. d(z s ,zs )), which represents the quasi-metric distance between the two inputs. It can be combined with the characterization function H θ and the quasi-metric function d, we get the end-to-end function d that calculates the quasi-metric distance from one state to another θ .d θ (s,g) is used to fit the expected cost of reaching goal g from state s.

[0072] In a possible implementation, the quasi-metric value function is implemented using an Interval Quasi-metric Embeddings (IQE) model.

[0073] In some examples, the calculation process of the interval quasi-metric embedding model includes:

[0074] For the two latent space representations of the input, the dimensions of each latent space representation are divided into multiple preset categories;

[0075] For each category, calculate the interval union of the values ​​of the two latent spaces representing the input divided into the dimension of the category;

[0076] The length of the union of intervals of all categories is summed up as the quasi-metric distance between the two latent space representations of the input.

[0077] For example, when calculating the interval quasi-metric embedding model, we can first put two inputs (such as z s and z g ) are divided into several categories. For example, some dimensions may describe position information, so they are classified into one category; some describe speed information, so they are classified into another category, and so on. Then, for each category of feature description, s and z g The range of the values ​​​​in the dimension that are classified into this category is merged to obtain the interval union of this category. For example, for z s and z g The range of values ​​on the dimensions that are classified into the location information category is merged, and the z s and z g The ranges spanned by the values ​​in the dimensions that are classified as speed information are merged, and so on. Finally, the sum of the lengths of the union of intervals corresponding to all categories is calculated, for example, the length of the union of intervals of the position information category, the length of the union of intervals of the speed information category, and so on is added together to obtain the quasi-metric distance.

[0078] Figure 4 A schematic diagram of an end-to-end model structure based on quasi-metrics according to an exemplary embodiment of the present disclosure is shown. As shown in the figure, the end-to-end model includes:

[0079] Input and encoding: The current state s and the target state g are mapped to their respective latent space representations z through encoders (i.e., representation functions) s and z g ;

[0080] IQE model: includes the Quasimetric head and its subsequent modules. Latent space representation z s and z g It is sent to the quasi-metric head module and passes through the IQE model to obtain two latent space representations z s and z g quasi-metric distance between

[0081] Output d θ (s, g): After being processed by the characterization function and the IQE model, a scalar value d is finally output. θ (s,g), that is, d(z s ,z g ). This value is the quasi-metric distance from the state s to the target g calculated sequentially through the characterization function and the IQE model.

[0082] According to the above implementation, the IQE model is used to implement a quasi-metric value function. Combined with its specific interval arithmetic calculation process, it provides a structured approach to defining the distance from a state to a target. Interval representation and calculations can better capture complex relationships, uncertainty, or range information in the state space. Furthermore, by adopting the above IQE model to satisfy its quasi-metric properties, it facilitates the learning of a better cost evaluation function, thereby enabling better action planning and improving system performance.

[0083] The constructed latent space dynamics model is used to receive the latent space representation of a state and the action applied to the state to predict the latent space representation of the next state.

[0084] The latent space dynamics model receives the latent space representation of the state (e.g. z s ) and the action selected by the agent (e.g. action a) as input, and predict the latent space representation of the state at the next moment after executing the action (e.g. ), thereby learning the state transition rules of the environment.

[0085] The three models (representation function, quasi-metric function, latent space dynamics model) are jointly trained using the transfer data (state s, action a, next state s', reward r) sampled from the interaction with the environment and the corresponding target state g. This is the key link in model learning in this embodiment. In order to solve the problem of sparse rewards, in some embodiments, a simpler reward r can be set. For example, in most time steps, the reward r may be a fixed negative constant (such as -1), representing the basic cost of performing an action step; only when the next state s' just reaches or is very close to the target state g specified in the current round, the reward r can be changed to a different value (such as 0 or a positive value), which is used as a sparse signal of goal achievement to optimize the quasi-metric function.

[0086] According to this embodiment, the goal of training the latent space dynamics model is to minimize the latent space representation of the predicted next state The difference between the latent space representation z′ of the true next state, in particular, using the end-to-end quasi-metric function d currently being learned θ This difference is calculated as a distance metric.

[0087] In some embodiments, during joint training, when optimizing the quasi-metric function using the reward r, the optimization objective is set to:

[0088]

[0089] Among them, θ is the parameter of the quasi-metric function, λ is the Lagrange multiplier, represents mathematical expectation, s~p s Indicates that the sampling state s conforms to the probability distribution p s , g~p g Indicates that the target state g conforms to the probability distribution p g ,(s,a,s',r)~p transition Indicates that the transferred data (s,a,s',r) is based on the real environment model p transition Sampling, d θ It is an end-to-end function for calculating the quasi-metric distance, which is used to perform the process of mapping the two input states into corresponding latent space representations and calculating the quasi-metric distance between the two latent space representations. φ is a pre-selected monotonically increasing convex function (such as a softplus function), ∈ 2 is a constant threshold.

[0090] This embodiment proposes a specific form of optimization objective to optimize the end-to-end quasi-metric function d θ The optimization objective adopts the form of minimum-maximum (min-max), and introduces the Lagrange multiplier λ to deal with the constrained optimization problem. On the one hand, term, encouraging the learned end-to-end function d θ It can effectively distinguish different state-target pairs, making the cost estimation from state to target discriminable; on the other hand, through the constraint Make the model meet local consistency. By setting the above optimization goal, we can ensure that the learned end-to-end function d θ It not only reflects the global cost from state to target, but also coordinates with its local changes in state transfer and the actual cost r, thereby providing a cost evaluation basis that is both discriminative and highly reliable for subsequent planning.

[0091] In some embodiments, during joint training, the following loss function is used: Train the latent space dynamics model:

[0092]

[0093] in, represents the latent space representation of the predicted next state, z′ represents the latent space representation of the true next state s′, and d is the quasi-metric function.

[0094] According to this embodiment, a loss function for training the latent space dynamics model in joint training is further proposed. The specific form of this loss function is used to measure and minimize the latent space representation of the next moment predicted by the latent space dynamics model. The difference between the actual next-moment latent space representation z′ is measured not using standard metrics such as the Euclidean distance, but rather using the quasi-metric function d learned in this embodiment. By using d as the distance metric, the learning process of the latent space dynamics model satisfies the geometric structure and cost relationship of the latent space defined by the quasi-metric function, ensuring that the learned state transition model is consistent with the task objective (i.e., reaching the goal at the lowest cost), thereby improving the effectiveness of subsequent model-based planning decisions. The formula includes the sum of the squared distances in both directions to further account for and accommodate the possible asymmetry of the quasi-metric function.

[0095] Joint training ensures that the parameters of each model can be optimized collaboratively, learning latent space representations and dynamic models that are beneficial for both predicting transfers and estimating costs.

[0096] Joint training can be implemented in a multi-step format to improve stability and efficiency. In some embodiments, in a multi-step joint training:

[0097] The optimization objective of the quasi-metric function takes the following multi-step form:

[0098]

[0099] Loss function for latent space dynamics models Take the following multi-step form:

[0100]

[0101] Among them, θ is the parameter of the quasi-metric function network, λ is the Lagrange multiplier, H is the preset step size, ρ i is a preset attenuation factor, φ is a pre-selected monotonically increasing convex function (such as a softplus function), d is a quasi-metric function used to calculate the quasi-metric distance between the two latent space representations of the input, z t+i represents the latent space representation of the sampled t+i-th step state, z g represents the latent space representation of the target state g, z t+i+1 represents the latent space representation of the sampled t+i+1th step state, Represents the latent space representation of the predicted t+i+1th step state, t is the starting time step of the current trajectory segment, r t+i is the immediate reward,∈ 2 is a constant threshold.

[0102] The above-mentioned embodiment further proposes a specific multi-step implementation form of the quasi-metric value function optimization objective and the latent space dynamics model loss function. The multi-step implementation has significant advantages over relying solely on single-step information. For the optimization of the quasi-metric value function, the multi-step form takes into account the cumulative cost on the future H-step trajectory and the consistency of multi-step transfers, so that time credit can be allocated more effectively, and long-term target information can be back-propagated faster, thereby improving the ability to learn cost functions in sparse rewards or long-term tasks; for the training of the latent space dynamics model, the multi-step form makes the model not only accurate in single-step predictions, but also able to maintain a low cumulative error in the prediction of consecutive H steps, which helps to learn more robust and long-term consistent environmental transfer laws and reduce the problem of error accumulation in model predictions. Therefore, the use of the multi-step form proposed in this embodiment for joint training can promote more stable and efficient learning, and provide a more reliable model foundation for subsequent planning decisions that rely on multi-step predictions.

[0103] In some embodiments, joint training is implemented in a multi-step manner, which may specifically include:

[0104] Sampling trajectory segments of transfer data of a preset step length from the experience replay buffer;

[0105] Use the representation function to map the state in the trajectory segment to the latent space and obtain the mapped latent space representation sequence;

[0106] Iteratively use the latent space dynamics model in the latent space to predict the latent space representation sequence of the future preset step length;

[0107] Based on the mapped latent space representation sequence and the predicted latent space representation sequence, a multi-step optimization objective and loss function are used to jointly train the representation function, quasi-metric function and latent space dynamics model.

[0108] This implementation further proposes a specific implementation process for multi-step joint training. This approach directly incorporates multi-step dependency information and the model's long-term predictive capability into training, enabling gradient propagation based on longer temporal information. This helps alleviate short-sightedness and enables the learning of more accurate and predictive latent space dynamics models and quasi-metric functions, thereby enhancing the learning efficiency and ultimate performance of the overall method.

[0109] Step 2: Use model predictive control (MPC) in combination with the cross entropy method (CEM) to optimize action selection. In the MPC process, the currently trained latent space dynamics model is used to predict the future latent space trajectory, and the currently trained quasi-metric value function is used to calculate the quasi-metric distance from the final latent space representation of the predicted latent space trajectory to the latent space representation of the target state as a cost evaluation criterion. Based on the minimum cost action planning result obtained by CEM optimization, a single action applied to the current state is determined.

[0110] After the model is trained or during training, when the agent needs to make a decision, the model-based planning process can be executed. It should be noted that reinforcement learning is typically a process of continuous learning and interaction with the environment. The agent needs to make decisions and take actions while learning to collect new experience data to further improve its model or strategy. It usually cannot wait until the model is "fully trained" before taking action. Therefore, step 2 does not strictly occur after step 1 is fully completed, but can be interleaved with step 1.

[0111] In some embodiments, using model predictive control (MPC) in combination with a cross entropy method (CEM) to optimize action selection may include:

[0112] Iterate the following steps for the current state until the preset conditions are met:

[0113] Sample N candidate action sequences with preset step lengths from the current action sampling distribution, where N is the preset number of candidate action sequences;

[0114] Based on the current state, the currently trained latent space dynamics model is used to predict the latent space trajectory corresponding to each candidate action sequence;

[0115] Calculate the quasi-metric distance between the final latent space representation of each predicted latent space trajectory and the latent space representation of the target state using the currently trained quasi-metric value function as the cost;

[0116] Select K elite action sequences with the lowest cost according to the calculated cost, where K is a preset value;

[0117] Update the parameters of the action sampling distribution based on the selected elite action sequences.

[0118] In some embodiments, action sequences of a preset step size can be randomly sampled from a preset probability distribution (such as a Gaussian probability distribution with preset parameters); in other embodiments, action sequences of a preset step size can be output using a trained policy network.

[0119] For each candidate action sequence, a trained latent space dynamics model can be used for forward simulation to predict the latent space trajectory generated by executing this action sequence within a preset future step size H, and the currently trained quasi-metric value function can be used to evaluate the final latent space representation of each latent space trajectory The quasi-metric distance of the latent space representation z reaching the target state g of this iteration g That is, calculate as the cost of this latent space trajectory. The lower the cost, the more likely this trajectory is to lead to the target state g. Then, a preset number of elite action sequences with the lowest cost can be selected according to the calculated cost, such as the K sequences with the lowest cost as the elite action sequences, 0 < K < N. Finally, based on the selected elite action sequences, the system can update the parameters of the action sampling distribution, such as the mean μ and variance σ of the Gaussian distribution, so that in the next iteration, it is more inclined to sample actions similar to these elite action sequences and perform better.

[0120] The above "sampling - simulation - evaluation - update" loop is repeated several times to continuously optimize the action sampling distribution, and finally the optimal action region is obtained. After the iteration ends, usually the mean of the final distribution or the first action in the sequence with the highest score can be selected as the optimal decision output in the current state and executed in the actual environment. The entire MPC-CEM process makes full use of the learned latent space dynamics model and quasi-metric value function to make predictive decisions through internal simulation and optimization.

[0121] Figure 5 Shows a schematic diagram of an online action selection process based on model predictive control and cross-entropy method (MPC-CEM) according to an exemplary embodiment of the present disclosure.

[0122] This process starts with obtaining the current environmental state s t , and using the currently trained characterization function h θConvert it into the corresponding latent space representation z t The latent space represents z t It contains the key information needed to make future forecasts and is the starting point for subsequent planning steps.

[0123] Next, the iterative optimization loop of MPC-CEM is started. First, the Gaussian distribution N candidate future action sequences are sampled, where the superscript j represents the number of iterations of the internal optimization loop of the cross-entropy method (CEM). Then, using the currently trained latent space dynamics model, a forward simulation is performed for each candidate action sequence H steps into the future, predicting N corresponding future latent space state trajectories. Next, the cost of each predicted latent space state trajectory is evaluated using the trained quasi-metric function d. This cost is calculated as the quasi-metric distance from the final predicted state to the target state. Based on the evaluation results, K elite action sequences with the lowest cost (i.e., the highest expected reward) are selected.

[0124] In some embodiments, the action value function Q that satisfies the following approximate relationship can be obtained based on the currently trained latent space dynamics model and the quasi-metric value function: * (s,a;g):

[0125]

[0126] Where d is the quasi-metric function, The latent space representation of the predicted next state, z g represents the latent space representation of the target state g, and r is the reward;

[0127] The obtained action value function is used to evaluate the cost of performing different actions to update the policy network.

[0128] This embodiment introduces an action value function (Q-function) and approximates the action value function based on the learned latent space dynamics model and quasi-metric function. Through this approximate relationship, it is possible to use existing model components to evaluate the expected total cost or negative value of performing an action in a specific state without having to independently learn a complex Q network. The evaluation results can be used to update the explicit policy network. For example, in the Actor-Critic framework, the gradient information of the Q function for action a, or the advantage function calculated based on the Q value, can be used to guide the optimization of the policy network (Actor) parameters, so that it is more inclined to output actions that can bring lower long-term expected costs (i.e., negative costs corresponding to higher Q values) to further improve learning capabilities.

[0129] Through the above-mentioned model construction, joint training and model-based planning and decision-making process, the multi-objective reinforcement learning method based on quasi-metrics and planning proposed in the present disclosure can effectively utilize limited interactive data to learn environment models (such as latent space dynamics models) and goal-oriented cost functions (such as quasi-metric value functions), and overcome the challenges brought by sparse rewards through forward planning, and ultimately achieve efficient learning and decision-making in multi-objective reinforcement learning tasks.

[0130] The following introduces an application example of the multi-objective reinforcement learning method based on quasi-metrics and planning proposed in this disclosure.

[0131] Step 1: Model construction and joint training.

[0132] In a multi-objective reinforcement learning environment, we design and jointly train a quasi-metric-based network model. This network model consists of four main components: a representation function, a quasi-metric value function, a latent space dynamics model, and an (optional) policy network. These models are trained using transition data sampled from environmental interactions.

[0133] S11. Design of quasi-metric function

[0134] Based on the equivalence between the (negative) optimal state value function and the quasi-metric in the multi-objective reinforcement learning environment, this example designs a quasi-metric value function to fit the negative optimal state value function. First, define two related functions:

[0135] End-to-end quasi-metric function d θ (s, g): receives the original state s and the target g, and outputs the final distance, which is calculated by the representation and latent space quasi-metric distance calculation;

[0136] Latent space quasi-metric distance function d(z1,z2): receives two latent space representations z1 and z2 and calculates the quasi-metric distance between them.

[0137] In this example, the interval quasi-metric embedding (IQE) model is used to implement the latent space quasi-metric distance function d(z1,z2).

[0138] S111, characterization function (h θ )Build

[0139] Characterization function h θ It is usually composed of a basic neural network, which is used to map the state space s, g, etc. to the latent space to obtain the corresponding latent space representation z s =h θ (s), z g =h θ (g). Its specific structure is selected according to the input state type. If the state space is pixel space, h θ By CNN backbone network (such as Figure 2 As shown) and FNN (as Figure 3 As shown); if the state space is a vector space, h θ Composed of FNN (such as Figure 3 shown).

[0140] S112, IQE calculation process

[0141] When using the IQE model, the latent space quasi-metric distance function d(z s ,z g ) may include: for the two latent space representations z of the input s 、z g , divide its dimensions into multiple categories; for each category, calculate the z s 、z g In the interval union of the corresponding dimension value; sum the lengths of the interval unions obtained for all categories to obtain the final quasi-metric distance d(z s ,z g ).

[0142] S113, Quasi-metric function optimization objective (single step)

[0143] Based on the network model constructed above, using the transfer data sampled from the environment interaction, the end-to-end quasi-metric function d can be trained by optimizing the following single-step optimization objective: θ (including h θ and d):

[0144]

[0145] The meaning of the parameters can be found in the relevant description above.

[0146] S12, Latent Space Dynamics Model Training (Single Step)

[0147] Using the learned latent space quasi-metric distance function d as a metric, the latent space dynamics model is trained in the latent space to fit the environment transition. The true next latent space representation z′ is obtained by the representation function acting on the next state s′, that is, z′=h θ (s′). Measures the next latent space representation predicted by the latent space dynamics model The single-step loss function of the difference between the true next latent space representation z′ for:

[0148]

[0149] The meaning of the parameters can be found in the relevant description above.

[0150] S13. Action-value function (Q-function) approximation and application

[0151] Based on the learned latent space dynamics model and latent space quasi-metric distance function d, the action value function Q*(s, a; g) can be approximately calculated:

[0152]

[0153] The meaning of the parameters in the formula can be found in the relevant description above. The calculated Q*(s, a; g) is mainly used to update the policy network.

[0154] S14, multi-step joint training (preferred)

[0155] The training process preferably adopts multi-step joint training. The trajectory segments s of the environment interaction transfer data with a step size of H can be sampled from the experience replay buffer. t ,s t+1 ,…,s t+H and related actions, rewards, and goals g.

[0156] S141. Multi-step joint training process and goals

[0157] The multi-step joint training process includes: using the representation function to map the trajectory state to the latent space, and obtaining the mapped latent space representation sequence z t ,z t+1 ,…,z t+H ; Using the latent space dynamics model and trajectory action prediction to obtain the predicted latent space representation sequence Then, the following multi-step optimization objective and loss function are used for joint training:

[0158] Multi-step optimization objective of the quasi-metric function (based on S113 extension):

[0159]

[0160] Multi-step loss function for latent space dynamics models (Based on S12 extension):

[0161]

[0162] The meaning of the parameters can be found in the relevant description above.

[0163] S142, Policy Network Update

[0164] If a policy network is included, the parameters of the policy network can be updated in joint training (or alternatively) using gradient-based action-value function Q* (approximated by S13) or other policy optimization methods.

[0165] Step 2: Model-based planning, decision-making and interaction (MPC-CEM)

[0166] Based on the currently trained model, model predictive control (MPC) combined with the cross entropy method (CEM) is used for forward planning and action selection.

[0167] S21. Overview of the planning process

[0168] The current state s t Use the currently trained representation function to map to the latent space representation z t Taking this as a starting point, forward planning is performed in the latent space: N candidate action sequences are generated, and the corresponding N future H-step latent space trajectories are predicted using the currently trained latent space dynamics model, and then the cost of each trajectory is evaluated (e.g. Figure 5 shown).

[0169] S211: Generate candidate action sequence

[0170] Generate N candidate action sequences of length H. The candidate action sequences can be selected from the probability distribution maintained by CEM (e.g. Gaussian distribution j is the number of iterations within the CEM); and / or use the currently trained policy network to output the action sequence.

[0171] S212. Predicting latent space trajectories

[0172] Using the initial latent space to represent z t And the N candidate action sequences generated in S211, the currently trained latent space dynamics model is iterated H times to obtain N predicted latent space trajectories

[0173] S213, trajectory cost evaluation

[0174] Use the currently trained latent space quasi-metric distance function d to calculate the terminal prediction latent space representation of each predicted trajectory To the target latent space representation z g Quasi-metric distance As the cost score of the Cain space trajectory. The lower the cost, the better.

[0175] S22, CEM update steps

[0176] According to the trajectory cost score obtained in step S213, the update step of the cross entropy method (CEM) is performed: the K elite action sequences with the lowest cost are selected and used to update the parameters of the action sampling distribution (such as the mean and variance of the Gaussian distribution) so that in the next (j+1) iteration, action sequences with lower sampling costs are more likely to be sampled.

[0177] S23, final action selection

[0178] Repeat the CEM iteration steps (S211 sampling, S212 prediction, S213 evaluation, S22 update) several times. After the iteration is completed, the optimal action sequence is determined based on the final action distribution (such as mean) or the sequence with the lowest score. Usually, the first action a in the sequence is output. t As the final decision, used to interact with the environment.

[0179] By implementing the steps above, this example is able to effectively learn and utilize a model for efficient planning and decision-making in a multi-objective environment with sparse rewards.

[0180] For other details and beneficial effects of this embodiment, please refer to the relevant introduction above and will not be repeated here.

[0181] Although this specification includes many specific implementation details, these should not be interpreted as limiting the scope of any invention or the scope of protection claimed, but are mainly used to describe the features of specific embodiments of specific inventions. Certain features described in multiple embodiments within this specification may also be implemented in combination in a single embodiment. On the other hand, the various features described in a single embodiment may also be implemented separately in multiple embodiments or in any suitable sub-combination. In addition, although features may work in certain combinations as described above and even initially claimed as such, one or more features from the claimed combination may be removed from the combination in some cases, and the claimed combination may point to a sub-combination or a variation of the sub-combination.

[0182] Similarly, although operations are depicted in a particular order in the accompanying drawings, this should not be understood as requiring that these operations be performed in the particular order shown or performed sequentially, or that all illustrated operations be performed to achieve the desired results. In some cases, multitasking and parallel processing may be advantageous. In addition, the separation of various system modules and components in the above-described embodiments should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product, or packaged into multiple software products.

[0183] Thus, specific embodiments of the subject matter have been described. Other embodiments are within the scope of the following claims. In some cases, the actions recited in the claims can be performed in a different order and still achieve the desired results. Furthermore, the processes depicted in the accompanying drawings do not necessarily require the particular order shown or sequential sequence to achieve the desired results. In some implementations, multitasking and parallel processing may be advantageous.

[0184] The above description is merely a preferred embodiment of one or more embodiments of this specification and is not intended to limit one or more embodiments of this specification. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of one or more embodiments of this specification shall be included in the scope of protection of one or more embodiments of this specification.

Claims

1. A multi-objective reinforcement learning method based on quasi-metrics and planning, characterized in that: include: Construct a representation function, a quasi-metric function, and a latent space dynamics model, and jointly train the representation function, the quasi-metric function, and the latent space dynamics model using the transition data (s, a, s', r) and the target state g sampled from the environmental interaction; wherein the representation function is used to map the state sampled from the environmental interaction to the corresponding latent space representation; the latent space dynamics model is used to predict the latent space representation of the next state based on the latent space representation of the current state s and the action a applied to the state; the quasi-metric function is used to calculate the quasi-metric distance between the two latent space representations of the input, and is used as a distance metric between the latent space representation of the predicted next state and the latent space representation of the sampled next state s' when training the latent space dynamics model; the reward r is used to optimize the quasi-metric function; Model predictive control (MPC) is used in combination with the cross entropy method (CEM) to optimize action selection. In the MPC process, the current trained latent space dynamics model is used to predict the future latent space trajectory, and the current trained quasi-metric value function is used to calculate the quasi-metric distance from the final latent space representation of the predicted latent space trajectory to the latent space representation of the target state as a cost evaluation criterion. Based on the minimum cost action planning result obtained by CEM optimization, a single action applied to the current state is determined.

2. The method according to claim 1, wherein: If the state of the environment is in pixel space, the representation function is composed of a convolutional neural network (CNN) backbone network and a feedforward neural network (FNN); If the state of the environment is in vector space form, the representation function is formed by a feedforward neural network (FNN).

3. The method according to claim 1, characterized in that The quasi-metric value function is implemented using the interval quasi-metric embedding model.

4. The method according to claim 3, characterized in that The calculation process of the interval quasi-metric embedding model includes: For the two latent space representations of the input, the dimensions of each latent space representation are divided into multiple preset categories; For each category, calculate the interval union of the values ​​of the two latent spaces representing the input divided into the dimension of the category; The length of the union of intervals of all categories is summed up as the quasi-metric distance between the two latent space representations of the input.

5. The method according to claim 1, wherein In joint training, when optimizing the quasi-metric function using the reward r, the optimization objective is set to: Among them, θ is the parameter of the quasi-metric function, λ is the Lagrange multiplier, represents mathematical expectation, s~p s Indicates that the sampling state s conforms to the probability distribution p s , g~p g Indicates that the target state g conforms to the probability distribution p g ,(s,a,s',r)~p transition Indicates that the transferred data (s,a,s',r) is based on the real environment model p transition Sampling, d θ It is an end-to-end function for calculating the quasi-metric distance, which is used to perform the process of mapping the two input states into corresponding latent space representations and calculating the quasi-metric distance between the two latent space representations. φ is a pre-selected monotonically increasing convex function, ∈ 2 is a constant threshold.

6. The method according to claim 1, wherein In joint training, the following loss function is used Train the latent space dynamics model: in, represents the latent space representation of the predicted next state, z′ represents the latent space representation of the true next state s′, and d is the quasi-metric function.

7. The method according to claim 1, characterized in that Joint training is implemented in a multi-step format, where: The optimization objective of the quasi-metric function takes the following multi-step form: Loss function for latent space dynamics models Take the following multi-step form: Among them, θ is the parameter of the quasi-metric function network, λ is the Lagrange multiplier, H is the preset step size, ρ i is the preset attenuation factor, φ is a pre-selected monotonically increasing convex function, d is the quasi-metric function, z t+i represents the latent space representation of the sampled t+i-th step state, z g represents the latent space representation of the target state g, z t+i+1 represents the latent space representation of the sampled t+i+1th step state, Represents the latent space representation of the predicted t+i+1th step state, t is the starting time step of the current trajectory segment, r t+i is the immediate reward, ∈ 2 is a constant threshold.

8. The method according to claim 1 or 7, characterized in that Joint training is implemented in multiple steps, including: Sampling trajectory segments of transfer data of a preset step length from the experience replay buffer; Use the representation function to map the state in the trajectory segment to the latent space and obtain the mapped latent space representation sequence; Iteratively use the latent space dynamics model in the latent space to predict the latent space representation sequence of the future preset step length; Based on the mapped latent space representation sequence and the predicted latent space representation sequence, a multi-step optimization objective and loss function are used to jointly train the representation function, quasi-metric function and latent space dynamics model.

9. The method according to claim 1, characterized in that Utilize Model Predictive Control (MPC) combined with the Cross Entropy Method (CEM) for action selection optimization, including: Iterate the following steps for the current state until the preset conditions are met: Sample N candidate action sequences with preset step lengths from the current action sampling distribution, where N is the preset number of candidate action sequences; Based on the current state, the currently trained latent space dynamics model is used to predict the latent space trajectory corresponding to each candidate action sequence; Use the currently trained quasi-metric value function to calculate the quasi-metric distance from the final latent space representation of each predicted latent space trajectory to the latent space representation of the target state as the cost; According to the calculated cost, K elite action sequences with the lowest cost are selected, where K is the preset value, 0 <K<N; Parameters of the action sampling distribution are updated based on the selected elite action sequence.

10. The method according to claim 9, characterized in that Sampling N candidate action sequences from the current action sampling distribution includes: Randomly sample action sequences of preset step lengths from a preset probability distribution, or use a trained policy network to sample action sequences of preset step lengths.

11. The method according to claim 1 or 10, characterized in that The method further comprises: Based on the currently trained latent space dynamics model and quasi-metric value function, the action value function Q is expanded to satisfy the following approximate relationship * (s,a;g): Where d is the quasi-metric function, The latent space representation of the predicted next state, z g represents the latent space representation of the target state g, and r is the reward; The obtained action value function is used to evaluate the cost of performing different actions to update the policy network.

Images