Implicit average flow policy off-line reinforcement learning method for robot autonomous control

Abstract

The present application relates to a kind of implicit average flow strategy offline reinforcement learning methods for robot autonomous control, comprising: obtaining the offline data set for robot;Variational autoencoder is trained based on offline data set;Establish the average flow matching strategy guided by latent behavior distribution as priori;Iterative training is carried out in combination with the learning mode of behavior cloning and reinforcement learning, and the target strategy for deployment in robot is generated by distillation optimization to average flow matching strategy, for robot in target environment based on target strategy executes autonomous control.This scheme not only realizes efficient strategy optimization at algorithm level, but also provides low-cost, high-performance and high-stability solution for autonomous control of robot in high-dimension, multi-task and complex environment at application level.

Description

Technical Field

[0001] This invention relates to the field of machine learning, and more particularly to an offline reinforcement learning method for implicit average current policy for autonomous robot control. Background Technology

[0002] Offline Reinforcement Learning (Offline RL) aims to learn policies using a fixed dataset without interaction in a real-world environment. A core problem in Offline RL is distribution shift, which is the mismatch between the distribution of the training data and the state-action distribution induced by the learned policy. This distribution shift can lead to performance degradation in real-world applications, making it a key research focus in Offline RL. To address this challenge, existing methods mainly fall into two categories: policy regularization methods and pessimistic value function methods. Policy regularization methods mitigate the impact of distribution shift by constraining the learned policy to closely approximate the behavior policy of the collected data. For example, TD3+BC introduces a behavior cloning-based regularization term, achieving a simple yet effective policy constraint; BCQ uses a Conditional Variational Autoencoder (CVAE) to approximate the behavior policy and improves policy performance through perturbation models; other methods limit the degree to which the policy deviates from the behavior policy through methods such as KL divergence, Maximum Mean Discrepancy (MMD), or Fisher divergence. Pessimistic value function methods mitigate the overestimation of Q-values ​​caused by distribution bias by learning conservative Q-values ​​from out-of-distribution (OOD) actions not seen in offline data. For example, CQL directly learns conservative Q-values ​​to ensure policy safety, while IQL and IDQL estimate the lower bound of Q-values ​​through expected regression, thus achieving pessimistic policy updates.

[0003] In recent years, diffusion models and flow matching models have been increasingly introduced into offline RL policy learning due to their powerful expressive capabilities and ability to effectively fit complex behavioral policies in offline data. Some methods directly approximate behavioral policies using diffusion models and then select candidate actions by evaluating the value of in-sample actions; Diffusion-QL uses the diffusion model as the policy and trains it using an objective similar to TD3+BC. However, diffusion models typically require multiple sampling steps, resulting in high inference costs. To reduce sampling overhead, EDP proposes using approximate actions to reduce inference time, while SRPO and DTQL distill the multi-step diffusion process using a single-step generator, but these methods still have limited policy expressive capabilities. Flow matching methods have also been applied to offline RL. For example, FQL approximates behavioral policies using flow matching policies and distills flow policies to achieve efficient policy learning and inference; FlowQ learns flow policies by modeling energy-guided probability paths through conditional velocity fields. However, these methods generally rely on multi-step sampling, incurring additional computational overhead, and require training via time backpropagation, which may lead to training instability.

[0004] In offline reinforcement learning, existing techniques primarily mitigate distribution shift problems through policy regularization or pessimistic value functions. However, these methods still have significant limitations. First, while policy regularization methods such as TD3+BC and BCQ improve learning stability by constraining policies to approximate behavioral policies, they lack the ability to express complex behavioral policies and struggle to fully capture the diversity of behaviors in the data. Second, policy distillation methods based on diffusion models or flow matching, such as Diffusion-QL and FQL, while possessing stronger expressive power, generally rely on multi-step sampling for policy generation, leading to low inference efficiency. Furthermore, they require backpropagation for training, which not only increases computational overhead but may also cause training instability. In addition, existing methods largely rely on standard Gaussian prior initialization and cannot directly learn the latent distribution of behavioral policies, thus limiting the expressive power and performance improvement of policies. Summary of the Invention

[0005] The technical problem to be solved by this invention is to provide an offline reinforcement learning method for implicit average current policy for autonomous robot control.

[0006] To achieve the above-mentioned objectives, this invention provides an offline reinforcement learning method for implicit average current policy in robot autonomous control, comprising the following steps: S1. Obtain an offline dataset for the robot, wherein the offline dataset includes multiple state-action pairs; S2. A variational autoencoder is trained based on an offline dataset to capture a potential behavior distribution that follows a normal distribution from diverse robot actions. S3. Establish an average flow matching strategy guided by the potential behavior distribution as a priori. The obtained potential behavior distribution is used as a priori guide, and the constructed average motion speed replaces the instantaneous speed modeled in the flow matching strategy to form an optimized path for robot behavior from the initial state to the target state, thereby realizing motion generation that conforms to the potential behavior distribution. S4. Iterative training is performed by combining behavioral cloning and reinforcement learning to distill and optimize the average flow matching strategy to generate a target strategy for deployment in the robot, which is used by the robot to perform autonomous control based on the target strategy in the target environment.

[0007] According to one aspect of the present invention, step S2, which involves training a variational autoencoder based on an offline dataset to capture a potential behavior distribution that follows a normal distribution from diverse robot actions, includes: set up For continuous latent variables sampled from the learned distribution of latent behaviors, it is represented as: ; in, Represents the distribution of potential behaviors. Indicates the robot's state. Indicates the robot's actions, subscript This represents the network parameters of the encoder in a variational autoencoder; Then, the latent behavior distribution that follows a normal distribution is represented as: ; in, Indicates a normal distribution. and denoted as the mean and standard deviation predicted by the variational autoencoder, respectively.

[0008] According to one aspect of the present invention, step S2, the step of training a variational autoencoder based on an offline dataset, includes: A variational autoencoder is constructed for learning the distribution of latent behaviors, wherein the variational autoencoder includes an encoder part and a decoder part, the encoder part being used to map state-action pairs to latent variables. The decoder section is used to extract latent variables Restructuring action; The variational autoencoder is jointly trained based on maximizing the lower bound of evidence, and during the training process, the latent variables... The variational autoencoder is optimized by sampling the input state-action pairs from the learned latent behavior distribution and maximizing the following lower bound of evidence: ; in, The log-likelihood represents the conditional probability, i.e., the probability under a given state. Under the conditions, the action The logarithm of the probability of occurrence. Indicates in latent variables Given the potential behavioral distribution, the expected likelihood of action reconstruction. This represents the likelihood of reconstructing the action using a variational autoencoder. This represents the distribution of the latent behavior output by the variational autoencoder and the latent variables used as priors. KL divergence between them Denotes the variational posterior distribution, given an action and state In the case of inferring latent variables The distribution, This represents the prior distribution, which in a variational autoencoder is the given state. Latent variables Distribution, subscript This represents the network parameters of the decoder in a variational autoencoder.

[0009] According to one aspect of the invention, in the step of jointly training the variational autoencoder based on maximizing the lower bound of evidence, during the training process, the latent variables are adjusted from the learned latent behavior distribution according to the input state-action pairs. During sampling, a reparameterization technique is employed to achieve a differentiable sampling process, which supports gradient optimization to obtain latent variables. The optimized form is expressed as: ; in, This indicates sampling from a standard Gaussian distribution. Represents the identity matrix.

[0010] According to one aspect of the present invention, step S3, the step of establishing an average flow matching strategy guided by the potential behavior distribution as a priori, includes: Construct the average motion speed, where the time point is used as the metric. t and r The average motion velocity is constructed by the ratio of the spatial displacement to the time interval of the action. ; ; in, Represents the average motion velocity field. Indicates the robot's time t The action, Indicates the robot's state. Represents the instantaneous velocity function. The variable representing integration represents the time interval. At some point in time, This represents the velocity field based on set conditions, i.e., in the state. Next action Over time t The probability flow direction and speed; The average action speed is used to replace the instantaneous speed modeled in the flow matching strategy, and the average flow matching network is constructed. Using the obtained potential behavior distribution as a priori guide, the constructed average flow matching network is trained to form an average flow matching strategy guided by the potential behavior distribution as a priori.

[0011] According to one aspect of the present invention, during the training of the average flow matching network, the learned latent behavior distribution is used as a priori guidance, and the average flow matching network is optimized based on a pre-constructed average action speed optimization objective and a corresponding loss function. The average action speed optimization objective takes the following form: ; ; in, This represents the target velocity field, which is the optimization objective for the average motion velocity. Indicates the action The partial derivatives describe the sensitivity of the velocity field to changes in motion. Indicates time The partial derivatives describe the rate of change of the velocity field with time. This represents the average motion velocity field predicted by the average flow matching network, and is an intermediate function. loss function usage Norm measures the average motion velocity field predicted by the average flow matching network. With the target velocity field The difference, and the target velocity field Using the stop-gradient operation ensures that the gradient exists only on the network parameters. The above calculation shows that the loss function can be expressed as: ; in, This represents the training loss of the average flow matching network, whose goal is to improve the average motion velocity field predicted by the average flow matching network. Approaching the target velocity field The specified monitoring signal, This represents the average action velocity field predicted by the average flow matching network. The expression, To express a joint expectation, i.e. based on the sampling process , , And generate network input actions through linear interpolation. Calculate the average motion velocity field predicted by the average flow matching network. With the target velocity field The expected value of the difference between them. Indicates time and time From uniform distribution Mid-sampling, Representing state-action pairs From empirical datasets Mid-sampling, This indicates that the latent variables are distributed from the latent behavior distribution that follows a normal distribution. Sampling.

[0012] According to one aspect of the present invention, step S4, the step of iterative training combining behavioral cloning and reinforcement learning, includes: A target strategy for deployment in robots is constructed based on the average flow matching strategy; Acquire offline data from diverse scenarios and obtain the latent behavior distribution based on the trained variational autoencoder; Based on the latent behavioral distribution, samples were taken at time points. t latent variables and observational data based on robot interaction with diverse scenarios. The average flow matching strategy is trained using a behavior cloning loss to ensure that the average flow matching strategy mimics the robot's behavior in diverse offline data scenarios; among which, the observation data Includes robot status information; Based on the latent behavioral distribution, samples were taken at time points. t latent variables and observational data based on robot interaction with diverse scenarios. The target policy is trained using reinforcement learning to maximize the value function while simultaneously generating the target policy by regularizing the knowledge of the average flow matching policy through distillation loss.

[0013] According to one aspect of the present invention, in the step of constructing a target strategy for deployment in a robot based on an average flow matching strategy, the target strategy is a finite-step strategy obtained based on the average flow matching strategy, wherein, for the sampling operation of the average flow matching strategy, the time integral is replaced with the average motion speed, thereby completing the conversion of the average flow matching strategy into a finite-step strategy; wherein, the average motion speed that replaces the time integral is expressed as: ; in, This means transforming the average flow matching strategy into the average action speed of the replacement time integral in a finite step strategy.

[0014] According to one aspect of the invention, in the step of constructing a target policy for deployment in a robot based on an average flow matching strategy, the target policy is a one-step policy obtained based on the average flow matching strategy, wherein the average motion velocity, replacing the time integral, is expressed as: ; in, This means transforming the average flow matching strategy into the average action speed of the replacement time integral in a one-step strategy. Indicates average motion speed The average speed of movement with a one-step interval.

[0015] According to one aspect of the present invention, in step S4, the loss function for the one-step policy in the step of iterative training combining behavior cloning and reinforcement learning is expressed as: ; ; in, The loss function represents the policy. Network parameters representing the policy, Indicates the data status Make expectations This represents offline data from diverse scenarios, i.e., experience datasets. This represents the value function for reinforcement learning training of the target policy. This represents the robot action generated by the target policy. This represents the robot action generated by the average flow strategy. For latent variables sampled from the latent behavior distribution, Represents the distillation loss function. To express a joint expectation, that is, for two random variables... and After sampling according to their respective distributions, the quantities in parentheses are averaged. This indicates that the gradient operation is stopped, i.e., the stop-gradient operation.

[0016] According to one aspect of the present invention, this scheme achieves efficient approximation and single-step generation of behavioral policies by introducing a MeanFlow Policy. This significantly reduces the computational overhead of multi-step sampling while maintaining policy expressiveness, thereby improving the stability and efficiency of policy learning. Furthermore, this scheme initializes the policy generation process by learning the latent behavior distribution, further enhancing the policy's adaptability and performance, effectively overcoming the shortcomings of existing technologies in complex behavioral policy modeling and efficient reasoning.

[0017] According to one aspect of the present invention, a latent behavior distribution is learned through a variational autoencoder (VAE), which is then used as a priori for subsequent policy learning optimization. Based on this, the present invention further proposes a latent distribution-guided meanflow policy, which effectively models behavioral policies and captures complex behavior distributions. Finally, a meanflow distillation policy optimization method is proposed to improve the policy, achieving efficient policy learning. This method balances the behavior of the meanflow distillation policy with policy improvement, and compared to iterative flow matching methods, it achieves more expressive policy learning results while ensuring sample efficiency.

[0018] According to one aspect of the present invention, this approach significantly enhances the effectiveness of flow-based policies by using a latent behavior prior distribution learned through a variational autoencoder. This eliminates reliance on standard Gaussian priors and instead learns a data-driven latent behavior distribution that captures richer information about the behavior policy. Furthermore, this latent behavior distribution can be used as a prior to guide the training of the target policy, thereby achieving more efficient and rational action generation.

[0019] According to one aspect of the present invention, this approach is based on the proposed average flow matching strategy guided by latent behavior distribution. The action distribution of the average flow strategy can be initialized using the latent behavior distribution, thereby achieving efficient strategy learning and action generation, which is effectively applicable to the autonomous control of complex robot tasks.

[0020] According to one aspect of the present invention, this approach can fully support low-step sampling through a finite-step strategy, providing greater flexibility and control accuracy for robot motion generation.

[0021] According to one aspect of the present invention, this approach, based on a distillation-average flow matching strategy, can fully utilize the powerful expressive capabilities of the average flow strategy to capture the complex motion distribution of the robot, while avoiding directly using the flow strategy as the final deployment strategy. This approach ensures the control accuracy of the robot and significantly reduces computational overhead, providing an effective technical approach for the autonomous control of robots in complex, multi-dimensional tasks.

[0022] According to one aspect of the present invention, this approach can be further extended to more complex tasks of higher dimensions, improving robustness in uncertain environments and enabling efficient policy transfer across tasks and environments, thereby promoting the further development and application of robot autonomous control technology.

[0023] According to one aspect of the present invention, an average flow strategy guided by latent distribution is introduced, enabling efficient single-step action generation under the knowledge condition of the latent prior distribution. Through this design, the average flow strategy not only characterizes the complex multimodal behavior distribution in the dataset but also, combined with latent behavioral priors, provides more informative guidance for policy learning. Based on this, the present invention further proposes an average flow distillation strategy optimization method. This method, by combining Q-value maximization with average flow constraints, transfers the effective knowledge from the average flow strategy to the final target policy. This distillation process ensures both the effectiveness and sample efficiency of policy improvement while avoiding the high computational overhead and potential instability caused by backpropagation of numerical ordinary differential equations (ODEs), thus possessing stronger feasibility and stability in practical robot autonomous control tasks. Attached Figure Description

[0024] Figure 1 This is a flowchart illustrating the steps of the offline reinforcement learning method for implicit average current policy for autonomous robot control according to the present invention. Figure 2 The flowchart shows the offline reinforcement learning method for implicit average current policy for robot autonomous control according to the present invention. Figure 3 The diagram shows the experimental environment in the embodiments of the present invention, wherein (a) represents the experimental environment of a quadruped robot maze, (b) represents the experimental environment of a humanoid robot maze, (c) represents the experimental environment of a quadruped robot soccer stadium, (d) represents the experimental environment of a quadruped robot small maze, (e) represents the experimental environment of a quadruped robot medium maze, (f) represents the experimental environment of a quadruped robot large maze, (g) represents the experimental environment of a single block operation, (h) represents the experimental environment of a double block operation, (i) represents the experimental environment of an object retrieval scene interaction task, (j) represents the experimental environment of an object placement scene interaction task, (k) represents the experimental environment of a 3×3 puzzle solving task, and (l) represents the experimental environment of a 4×4 puzzle solving task. Figure 4 The following are learning curves comparing different methods in the embodiments of the present invention, wherein (a) represents the learning curve of the quadruped robot's soccer stadium navigation task, (b) represents the learning curve of the humanoid robot's medium-sized maze navigation task, (c) represents the learning curve of the 3×3 puzzle solving task, (d) represents the learning curve of the scene interaction task, (e) represents the learning curve of the single-block operation task, (f) represents the learning curve of the 4×4 puzzle solving task, (g) represents the learning curve of the quadruped robot's large maze navigation task, (h) represents the learning curve of the double-block operation task, (i) represents the learning curve of the quadruped robot's maze navigation task, (j) represents the learning curve of the humanoid robot's large maze navigation task, and (k) represents the learning curve of the quadruped robot's giant maze navigation task. Figure 5 The following is a comparison chart of the comprehensive performance indicators of different methods in the embodiments of the present invention, wherein (a) represents the comparison chart of indicators about the median, (b) represents the comparison chart of indicators about the interquartile mean, (c) represents the comparison chart of indicators about the mean, and (d) represents the comparison chart of indicators about the optimal gap. Detailed Implementation

[0025] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the embodiments will be briefly described below. Obviously, the drawings described below are merely some embodiments of the present invention, and those skilled in the art can obtain other drawings based on these drawings without creative effort.

[0026] In describing embodiments of the present invention, the terms "longitudinal," "lateral," "upper," "lower," "front," "rear," "left," "right," "vertical," "horizontal," "top," "bottom," "inner," and "outer" express orientations or positional relationships based on the orientations or positional relationships shown in the relevant drawings. They are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, the above terms should not be construed as limitations on the present invention.

[0027] The present invention will now be described in detail with reference to the accompanying drawings and specific embodiments. The embodiments cannot be described in detail here, but the embodiments of the present invention are not limited to the following embodiments.

[0028] Combination Figure 1 and Figure 2 As shown, according to one embodiment of the present invention, an offline reinforcement learning method for implicit average current policy for robot autonomous control includes the following steps: S1. Obtain an offline dataset for the robot, wherein the offline dataset includes multiple state-action pairs; S2. A variational autoencoder is trained based on an offline dataset to capture a potential behavior distribution that follows a normal distribution from diverse robot actions. S3. Establish an average flow matching strategy guided by the potential behavior distribution as a priori. The obtained potential behavior distribution is used as a priori guide, and the constructed average motion speed replaces the instantaneous speed modeled in the flow matching strategy to form an optimized path for robot behavior from the initial state to the target state, thereby realizing motion generation that conforms to the potential behavior distribution. S4. Iterative training is performed by combining behavioral cloning and reinforcement learning to distill and optimize the average flow matching strategy to generate a target strategy for deployment in the robot, which is used by the robot to perform autonomous control based on the target strategy in the target environment.

[0029] According to one embodiment of the present invention, step S2, which involves training a variational autoencoder based on an offline dataset to capture a potential behavior distribution that follows a normal distribution from diverse robot actions, includes: set up For continuous latent variables sampled from the learned distribution of latent behaviors, it is represented as: ; in, Represents the distribution of potential behaviors. Indicates the robot's state. Indicates the robot's actions, subscript This represents the network parameters of the encoder in a variational autoencoder; Then, the latent behavior distribution that follows a normal distribution is represented as: ; in, Indicates a normal distribution. and denoted as the mean and standard deviation predicted by the variational autoencoder, respectively.

[0030] According to one embodiment of the present invention, step S2, the step of training a variational autoencoder based on an offline dataset, includes: A variational autoencoder (VAE) is constructed to learn the distribution of latent behaviors. The VAE consists of an encoder part and a decoder part. The encoder part is used to map state-action pairs to latent variables. The decoder part is used to extract latent variables Restructuring action; The variational autoencoder is jointly trained based on the Evidence Lower Bound (ELBO) lower bound, and during the training process, the latent variables... The variational autoencoder is optimized by sampling the input state-action pairs from the learned latent behavior distribution and maximizing the following lower bound of evidence: ; in, The log-likelihood represents the conditional probability, i.e., the probability under a given state. Under the conditions, the action The logarithm of the probability of occurrence. Indicates in latent variables Given the potential behavioral distribution, the expected likelihood of action reconstruction. This represents the likelihood of reconstructing the action using a variational autoencoder. This represents the distribution of the latent behavior output by the variational autoencoder and the latent variables used as priors. KL divergence between them Denotes the variational posterior distribution, given an action and state In the case of inferring latent variables The distribution, This represents the prior distribution, which in a variational autoencoder is the given state. Latent variables Distribution, subscript This represents the network parameters of the decoder in a variational autoencoder.

[0031] like Figure 1 As shown, according to one embodiment of the present invention, in the step of jointly training the variational autoencoder based on maximizing the lower bound of evidence, during the training process, the latent variables are adjusted from the learned latent behavior distribution according to the input state-action pairs. During sampling, a reparameterization technique is employed to achieve a differentiable sampling process, which supports gradient optimization to obtain latent variables. The optimized form is expressed as: ; in, This indicates sampling from a standard Gaussian distribution. Represents the identity matrix.

[0032] like Figure 1 As shown, according to one embodiment of the present invention, step S3, the step of establishing an average flow matching strategy guided by the potential behavior distribution as a priori, includes: Construct the average motion speed, where the time point is used as the metric. t andr The average motion velocity is constructed by the ratio of the spatial displacement to the time interval of the action. ; ; in, Represents the average motion velocity field. Indicates the robot's time t The action, Indicates the robot's state. Represents the instantaneous velocity function. The variable representing integration represents the time interval. At some point in time, This represents the velocity field based on set conditions, i.e., in the state. Next action Over time t The probability flow direction and speed; In this embodiment, the average motion speed is obtained based on the following steps, wherein, based on the definition of average motion speed: using time points... t and r The ratio of the spatial displacement to the time interval between actions yields a general expression for the average motion velocity, which is expressed as: ; Furthermore, the general expression for average motion velocity is related to... Differentiate, and If we treat it as a constant, then we get: ; .

[0033] The average action speed is used to replace the instantaneous speed modeled in the flow matching strategy, and the average flow matching network is constructed. Using the obtained latent behavior distribution as prior guidance, the constructed average flow matching network is trained to form an average flow matching strategy guided by the latent behavior distribution. Specifically, during the training process of the average flow matching network, the learned latent behavior distribution is used as prior guidance, and the average flow matching network is optimized based on a pre-constructed average action speed optimization objective and a corresponding loss function. In this embodiment, an objective function for optimizing training is provided for the constructed average action speed, and is expressed as: ; in, This represents the average action velocity field predicted by the average flow matching network. Action The partial derivatives, This represents the average action velocity field predicted by the average flow matching network. Regarding time The partial derivatives; Furthermore, latent variables are sampled from the latent behavioral distribution. And it is represented as: ; Furthermore, based on the reparameterization method of flow matching, Replace with ,default ,Right now Therefore, the average motion speed optimization objective is obtained in the following form: ; ; in, This represents the target velocity field, which is the optimization objective for the average motion velocity. Indicates the action The partial derivatives describe the sensitivity of the velocity field to changes in motion. Indicates time The partial derivatives describe the rate of change of the velocity field with time. This represents the average motion velocity field predicted by the average flow matching network, and is an intermediate function. Furthermore, along the vector The directional derivative can be calculated using the Jacobian-Vector Product (JVP) of the Jacobian matrix and the direction vector. This represents the input tangent vector, along which the function is computed. Total derivative: where Representing latent variables Relative to time rate of change, Indicates time (i.e., the starting point of integration) is a constant that does not change with time. This represents the rate at which time changes with itself. This calculation is a highly optimized basic operation in modern deep learning frameworks, such as torch.func.jvp in PyTorch or jax.jvp in JAX.

[0034] Furthermore, this scheme introduces MeanFlow to learn the average velocity of robot movements. For this purpose, the loss function used for the MeanFlow matching network is... Norm measures the average motion velocity field predicted by the average flow matching network. With the target velocity field The difference, and the target velocity field The stop-gradient operation (i.e., sg operation) is used to ensure that the gradient exists only on the network parameters. The calculation is performed on top of the target velocity field, without propagating to the step of generating the target velocity field. Therefore, the loss function is expressed as: ; in, This represents the training loss of the average flow matching network, whose goal is to improve the average motion velocity field predicted by the average flow matching network. Approaching the target velocity field The specified monitoring signal, This represents the average action velocity field predicted by the average flow matching network. The expression, To express a joint expectation, i.e. based on the sampling process , , And generate network input actions through linear interpolation. Calculate the average motion velocity field predicted by the average flow matching network. With the target velocity field The expected value of the difference between them. Indicates time and time From uniform distribution Mid-sampling, Representing state-action pairs From empirical datasets Mid-sampling, This indicates that the latent variables are distributed from the latent behavior distribution that follows a normal distribution. Sampling.

[0035] Therefore, the process of selecting training samples is: state-action pairs From empirical datasets Mid-sampling, latent variables Sampling from the potential behavior distribution, time points and time From uniform distribution Mid-sampling, and then generating network input actions through linear interpolation. Therefore, the loss function is the average motion velocity field predicted by the average flow matching network based on the above sampling process. With the target velocity field The expected value of the difference between them.

[0036] The goal of training the average flow matching network is to minimize the loss function of the average flow matching network.

[0037] According to one embodiment of the present invention, step S4, which involves iterative training combining behavioral cloning and reinforcement learning, includes: A target policy for deployment in a robot is constructed based on the average flow matching strategy. The target policy is a finite-step policy obtained based on the average flow matching strategy. Specifically, for the sampling operation of the average flow matching strategy, the time integral is replaced with the average motion velocity, completing the transformation from the average flow matching strategy to the finite-step policy. The average motion velocity, where the time integral is replaced, is expressed as: ; in, This indicates the average action speed in the substitution time integral of the average flow matching strategy, which is transformed into a finite-step strategy. (Subscript) r Indicates a point in time.

[0038] Acquire offline data from diverse scenarios and obtain the latent behavior distribution based on the trained variational autoencoder; Based on the latent behavioral distribution, samples were taken at time points. t latent variables and observational data based on robot interaction with diverse scenarios. The average flow matching strategy is trained using a behavior cloning loss to ensure that the average flow matching strategy mimics the robot's behavior in diverse offline data scenarios; among which, the observation data Includes robot status information; Based on the latent behavioral distribution, samples were taken at time points. t latent variables and observational data based on robot interaction with diverse scenarios. The target policy is trained using reinforcement learning to maximize the value function while simultaneously generating the target policy by regularizing the knowledge of the average flow matching policy through distillation loss.

[0039] According to one embodiment of the present invention, in the step of constructing a target strategy for deployment in a robot based on an average flow matching strategy, the target strategy is a one-step strategy obtained based on the average flow matching strategy, wherein the average motion speed of the replacement time integral is expressed as: ; in, This means transforming the average flow matching strategy into the average action speed of the replacement time integral in a one-step strategy. Indicates average motion speed The average speed of movement with a step interval, and It is obtained by sampling based on the potential prior distribution.

[0040] According to one embodiment of the present invention, in step S4, the loss function used for the one-step policy in the step of iterative training combining behavior cloning and reinforcement learning is expressed as: ; ; in, The loss function represents the policy. Network parameters representing the policy, Indicates the data status Make expectations This represents offline data from diverse scenarios, i.e., experience datasets. This represents the value function for reinforcement learning training of the target policy. This represents the robot action generated by the target policy. This represents the robot action generated by the average flow strategy. For latent variables sampled from the latent behavior distribution, Represents the distillation loss function. To express a joint expectation, that is, for two random variables... and After sampling according to their respective distributions, the quantities in parentheses are averaged. This indicates that the gradient operation is stopped, i.e., the stop-gradient operation.

[0041] To further illustrate this plan, further examples will be provided.

[0042] Example First, the experimental setup is introduced. The experiments are based on two benchmark suites, OGBench and D4RL, covering a total of 68 tasks. Then, three classes of ten representative policy learning methods are introduced as baselines for comparison, and systematically compared with the proposed Latent Mean Flow Policy Optimization (LaMPO) offline reinforcement learning method for robot autonomous control. To ensure the reliability of the statistical results, the normalized comprehensive index is further tested, demonstrating its ability to support the learning and deployment of robot autonomous control policies in complex scenarios.

[0043] Benchmarking The method was evaluated on two widely used benchmark sets: OGBench and D4RL, covering 68 tasks across 12 different categories, such as... Figure 3 As shown.

[0044] OGBench includes a diverse range of complex tasks, covering robot motion control and manipulation, with a complexity significantly higher than the standard D4RL benchmark (10 task classes). Although OGBench was originally designed for offline objective-conditional reinforcement learning, its reward-based single-task variant (denoted as "-singletask") can be used in experiments to maintain consistency with the standard reward maximization setting. Specifically, five motion control environments and five manipulation environments from OGBench are used, each containing five different tasks, for a total of 50 tasks. This approach was also evaluated on the AntMaze and Adroit tasks of the D4RL benchmark (LaMPO), using the "-v2" version of the dataset.

[0045] The AntMaze task, characterized by sparse rewards and long-term trajectories, requires controlling an 8-DOF Ant robot to navigate mazes of three different sizes (umaze, medium, and large) and reach a designated goal. This type of task is also extremely challenging for both online and offline reinforcement learning (RL) algorithms.

[0046] The Adroit suite includes high-dimensional robotic manipulation tasks based on a 24-DOF Shadow Hand, such as hammering nails, opening doors, spinning pens, and moving spheres. These tasks offer limited expert demonstrations and sparse rewards, making them an important benchmark for evaluating generalization capabilities in complex manipulation scenarios.

[0047] Baseline method This approach (LaMPO) is compared with three strong baselines: flow-based strategies, diffusion-based strategies, and Gaussian-based strategies.

[0048] Flow strategy Since only a few methods have explicitly used streaming policies previously, this comparison introduces several streaming policy variants, including different schemes based on policy extraction, weighted behavior cloning, parameterized policy gradients, and rejection sampling. The main comparisons focus on FQL, FAWAC, FBRAC, and IFQL. FQL trains a one-step policy using constrained flow behavior, rather than directly iteratively optimizing the streaming policy using the Q-value function, making it the closest streaming matching baseline to our proposed scheme (LaMPO). FAWAC is a version of the streaming policy based on advantage-weighted Actor-Critic; FBRAC is a version of the streaming policy based on behavior-regularized Actor-Critic; and IFQL combines the in-sample Q-learning critic and the rejection sampling mechanism of IDQL, and then uses FQL's policy optimization method. The main difference between FAWAC and FBRAC lies in their policy extraction schemes, but their overall architecture is similar to our proposed scheme (LaMPO).

[0049] diffusion strategy Three diffusion strategy baselines were compared: IDQL, SRPO, and Consistency-AC (CAC).

[0050] IDQL combines a diffusion model with rejection sampling and uses Q-values ​​for sample selection, making it an in-sample policy learning method. SRPO uses the score function of the diffusion behavior model to regularize the policy gradient during optimization. CAC introduces a consistency model as a more expressive policy and achieves policy improvement through the Actor-Critic algorithm. Notably, SRPO and CAC, like LaMPO, employ a distillation-based policy extraction scheme.

[0051] Gaussian strategy Compared to the classic Gaussian policy baseline, this includes BC, IQL, and ReBRAC. BC is a direct imitation learning method based on behavioral cloning. IQL uses in-sample Q-learning to avoid overestimating the Q-values ​​of out-of-distribution actions during policy evaluation. ReBRAC incorporates regularization on both the actor and critic, including policy constraints and conservative Q-value estimation.

[0052] Experimental results The learning curves for the 68 tasks in OGBench and D4RL are shown below. Figure 4 All methods were trained within a fixed number of steps and evaluated under eight different random seeds. Baseline results were derived directly from publicly available experimental results for each method. The same network size and discount factor were used to ensure fair comparison. Results show that our proposed approach (LaMPO) significantly outperforms other methods on both benchmarks.

[0053] In addition, to ensure the rigor and reproducibility of the statistical results, Figure 5 The paper further demonstrates four types of normalized comprehensive performance metrics: median, interquartile range mean (IQM), mean, and optimal gap. These results are all calculated based on the final performance score of the normalization and obtained through multi-random seed experiments. Figure 5 The horizontal error bars represent standard errors, and the methods are ranked by median performance. This evaluation provides a comprehensive view of the overall performance and consistency of the methods. The results show that the proposed scheme (LaMPO) has significant advantages in both overall performance and stability.

[0054] Experimental results show that the present invention achieves consistently superior performance across 68 tasks in the OGBench and D4RL benchmark tests, with an average improvement of approximately 19% compared to the existing best methods. This demonstrates that the present invention not only enables efficient and highly expressive policy learning but also significantly improves the policy optimization performance of offline reinforcement learning in complex environments, fully validating the effectiveness and advancement of the proposed method in offline reinforcement learning.

[0055] According to this invention, the proposed solution no longer relies on instantaneous velocity fields in traditional flow matching methods. Instead, it introduces a modeling approach based on average motion velocity and initializes it in conjunction with latent behavior distributions. This approach can more accurately characterize complex motion distributions during robot autonomous control, avoiding the high noise and instability issues caused by simply relying on instantaneous velocity, thereby improving the stability of policy learning.

[0056] According to the present invention, the proposed latent distribution guidance mechanism enables the strategy to generate single-step actions under the constraints of a latent prior distribution. Through this design, the robot can utilize the prior information about the environment and actions contained in the latent distribution when performing tasks, thereby achieving efficient modeling of complex, multimodal distributions. This improvement significantly enhances the expressive power and generalization ability of the strategy, enabling the robot to better adapt to complex scenarios.

[0057] Furthermore, this invention introduces an average flow distillation optimization method in the policy distillation stage. By combining Q-value maximization with average flow constraints, complex flow policy knowledge is transferred to a more computationally efficient one-step policy. This process avoids dependence on solving numerical ordinary differential equations, thereby effectively reducing computational overhead and backpropagation instability during training, making the training process more efficient and easier to deploy in practical robot control systems.

[0058] According to the present invention, this solution not only achieves efficient strategy optimization at the algorithm level, but also provides a low-cost, high-performance and high-stability solution for the autonomous control of robots in high-dimensional, multi-task and complex environments at the application level.

[0059] The above description is merely an example of a specific solution of the present invention. For any devices and structures not described in detail herein, it should be understood that they are implemented using common devices and methods already available in the art.

[0060] The above description is merely one embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the invention by those skilled in the art. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. An offline reinforcement learning method for implicit average current policy in robot autonomous control, characterized in that, Includes the following steps: S1. Obtain an offline dataset for the robot, wherein the offline dataset includes multiple state-action pairs; S2. A variational autoencoder is trained based on an offline dataset to capture a normally distributed latent behavior distribution from diverse robot actions; this includes: set up For continuous latent variables sampled from the learned distribution of latent behaviors, it is represented as: in, Represents the distribution of potential behaviors. Indicates the robot's state. Indicates the robot's actions, subscript This represents the network parameters of the encoder in a variational autoencoder; Then, the latent behavior distribution that follows a normal distribution is represented as: in, Indicates a normal distribution. and represents the mean and standard deviation predicted by the variational autoencoder, respectively; S3. Establish an average flow matching strategy guided by the potential behavior distribution as a priori. This strategy utilizes the obtained potential behavior distribution as a priori guidance, replacing the instantaneous velocity modeled in the flow matching strategy with the constructed average motion velocity to form an optimized path for robot behavior from the initial state to the target state, thereby achieving motion generation that conforms to the potential behavior distribution. This includes: Construct the average motion speed, where the time point is used as the metric. t and r The average motion velocity is constructed by the ratio of the spatial displacement to the time interval of the action. in, Represents the average motion velocity field. Indicates the robot's time t The action, Indicates the robot's state. Represents the instantaneous velocity function. The variable representing integration represents the time interval. At some point in time, This represents the velocity field based on set conditions, i.e., in the state. Next action Over time t The probability flow direction and speed; The average action speed is used to replace the instantaneous speed modeled in the flow matching strategy, and the average flow matching network is constructed. Using the obtained latent behavior distribution as prior guidance, the constructed average flow matching network is trained to form an average flow matching strategy guided by the latent behavior distribution. Specifically, during the training of the average flow matching network, the learned latent behavior distribution is used as prior guidance, and the average flow matching network is optimized based on a pre-constructed average action speed optimization objective and a corresponding loss function. The form of the average action speed optimization objective is as follows: in, This represents the target velocity field, which is the optimization objective for the average motion velocity. Indicates the action The partial derivatives describe the sensitivity of the velocity field to changes in motion. Indicates time The partial derivatives describe the rate of change of the velocity field with time. This represents the average motion velocity field predicted by the average flow matching network, and is an intermediate function. loss function usage Norm measures the average motion velocity field predicted by the average flow matching network. With the target velocity field The difference, and the target velocity field Using the stop-gradient operation ensures that the gradient exists only on the network parameters. The above calculation shows that the loss function can be expressed as: in, This represents the training loss of the average flow matching network, whose goal is to improve the average motion velocity field predicted by the average flow matching network. Approaching the target velocity field The specified monitoring signal, This represents the average action velocity field predicted by the average flow matching network. The expression, To express a joint expectation, i.e. based on the sampling process , , And generate network input actions through linear interpolation. Calculate the average motion velocity field predicted by the average flow matching network. With the target velocity field The expected value of the difference between them. Indicates time and time From uniform distribution Mid-sampling, Representing state-action pairs From empirical datasets Mid-sampling, This indicates that the latent variables are distributed from the latent behavior distribution that follows a normal distribution. Mid-sampling; S4. Iterative training is performed by combining behavioral cloning and reinforcement learning to distill and optimize the average flow matching strategy to generate a target strategy for deployment in the robot, which is used by the robot to perform autonomous control based on the target strategy in the target environment.

2. The offline reinforcement learning method for implicit average current policy for robot autonomous control according to claim 1, characterized in that, Step S2, the step of training the variational autoencoder based on the offline dataset, includes: A variational autoencoder is constructed for learning the distribution of latent behaviors, wherein the variational autoencoder includes an encoder part and a decoder part, the encoder part being used to map state-action pairs to latent variables. The decoder section is used to extract latent variables Restructuring action; The variational autoencoder is jointly trained based on maximizing the lower bound of evidence, and during the training process, the latent variables... The variational autoencoder is optimized by sampling the input state-action pairs from the learned latent behavior distribution and maximizing the following lower bound of evidence: in, The log-likelihood represents the conditional probability, i.e., the probability under a given state. Under the conditions, the action The logarithm of the probability of occurrence. Indicates in latent variables Given the potential behavioral distribution, the expected likelihood of action reconstruction. This represents the likelihood of reconstructing the action using a variational autoencoder. This represents the distribution of the latent behavior output by the variational autoencoder and the latent variables used as priors. KL divergence between them Denotes the variational posterior distribution, given an action and state In the case of inferring latent variables The distribution This represents the prior distribution, which in a variational autoencoder is the given state. Latent variables Distribution, subscript This represents the network parameters of the decoder in a variational autoencoder.

3. The offline reinforcement learning method for implicit average current policy for robot autonomous control according to claim 2, characterized in that, In the step of jointly training the variational autoencoder based on maximizing the lower bound of evidence, during the training process, the latent variables are adjusted from the learned latent behavior distribution according to the input state-action pairs. During sampling, a reparameterization technique is employed to achieve a differentiable sampling process, which supports gradient optimization to obtain latent variables. The optimized form is expressed as: in, This indicates sampling from a standard Gaussian distribution. Represents the identity matrix.

4. The offline reinforcement learning method for implicit average current policy for robot autonomous control according to claim 3, characterized in that, Step S4, which combines behavioral cloning and reinforcement learning for iterative training, includes: A target strategy for deployment in robots is constructed based on the average flow matching strategy; Acquire offline data from diverse scenarios and obtain the latent behavior distribution based on the trained variational autoencoder; Based on the latent behavioral distribution, samples were taken at time points. t latent variables and observational data based on robot interaction with diverse scenarios. The average flow matching strategy is trained using a behavior cloning loss to ensure that the average flow matching strategy mimics the robot's behavior in diverse offline data scenarios; among which, the observation data Includes robot status information; Based on the latent behavioral distribution, samples were taken at time points. t latent variables and observational data based on robot interaction with diverse scenarios. The target policy is trained using reinforcement learning to maximize the value function while simultaneously generating the target policy by regularizing the knowledge of the average flow matching policy through distillation loss.

5. The offline reinforcement learning method for implicit average current policy for robot autonomous control according to claim 4, characterized in that, In the step of constructing a target policy for deployment in a robot based on the average flow matching strategy, the target policy is a finite-step policy obtained based on the average flow matching strategy. Specifically, for the sampling operation of the average flow matching strategy, the time integral is replaced with the average motion speed, completing the transformation from the average flow matching strategy to a finite-step policy. The average motion speed, where the time integral is replaced, is expressed as: in, This means transforming the average flow matching strategy into the average action speed of the replacement time integral in a finite step strategy.

6. The offline reinforcement learning method for implicit average current policy for robot autonomous control according to claim 5, characterized in that, In the step of constructing a target policy for deployment in the robot based on the average flow matching strategy, the target policy is a one-step policy obtained based on the average flow matching strategy, wherein the average motion speed with replacement time integral is expressed as: in, This means transforming the average flow matching strategy into the average action speed of the replacement time integral in a one-step strategy. Indicates average motion speed The average speed of movement with a one-step interval.

7. The offline reinforcement learning method for implicit average current policy for robot autonomous control according to claim 6, characterized in that, In step S4, the loss function used for the one-step policy in the iterative training step that combines behavior cloning and reinforcement learning is expressed as: in, The loss function representing the policy. Network parameters representing the policy, Indicates the data status Make expectations This represents offline data from diverse scenarios, i.e., experience datasets. This represents the value function for reinforcement learning training of the target policy. This represents the robot action generated by the target policy. This represents the robot action generated by the average flow strategy. For latent variables sampled from the latent behavior distribution, Represents the distillation loss function. To express a joint expectation, that is, for two random variables... and After sampling according to their respective distributions, the quantities in parentheses are averaged. This indicates that the gradient operation is stopped, i.e., the stop-gradient operation.

Images