Nuclear density estimation driven diffusion reinforcement learning method and system, and intelligent agent
By introducing kernel density estimation and Beta kernel density estimation in diffusion reinforcement learning, combining multi-sampling mechanism and dual Q network structure, the strategy update and value estimation stability problems of existing diffusion reinforcement learning methods in online learning environments are solved, and more efficient strategy optimization and more stable value estimation are achieved.
Patent Information
- Application Number
- CN202510326381.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-03-19
- Publication Date
- 2025-07-01
AI Technical Summary
The existing diffusion reinforcement learning methods are difficult to directly improve strategies through behavioral cloning losses in online learning environments, and problems such as policy expression limitations, instability in Q function estimation and low computational efficiency affect their application efficiency in complex environments.
The diffusion reinforcement learning method driven by kernel density estimation is used to calculate action entropy through Beta kernel density estimation, update the action network parameters and entropy weights, and combine the multi-sampling mechanism and dual Q network structure to improve the stability of value estimation and the efficiency of strategy optimization.
It significantly improves the efficiency of strategy optimization, reduces decision time, enhances the stability of value estimation, and improves the system's real-time response and work efficiency.
Smart Images

Figure CN120235211A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the application of artificial intelligence technology in the field of agent technology. Specifically, it relates to a diffusion reinforcement learning method and system, and an agent driven by kernel density estimation. Background Art
[0002] Reinforcement learning is an important branch in the field of artificial intelligence and has wide applications in fields such as robot control, visual processing, and industrial systems. In the field of quadruped robot control, traditional reinforcement learning methods usually use deterministic neural networks or single Gaussian distributions to parameterize policies. This simplification makes it difficult to adapt to the multi-modal motion requirements under complex terrains. For example, when a quadruped robot needs to switch between actions such as walking, jumping, and climbing according to ground hardness, obstacle distribution, and body dynamics characteristics, the deterministic policy is prone to generating mechanized fixed gait patterns, resulting in low motion efficiency and even the risk of imbalance.
[0003] In recent years, diffusion models have received extensive attention due to their successful applications in the fields of image and video generation. Diffusion models learn data distributions by modeling the forward diffusion process and the reverse generation process. Among them, the forward process gradually adds noise to the data in T steps to generate a series of latent variables. This process is usually defined as a Markov process, where the sampling at each step is based on the state of the previous step and finally converges to an isotropic Gaussian distribution. The reverse process reconstructs the original data distribution by iterative denoising, and this process learns parameters by minimizing the variational lower bound. This property has been successfully applied to the terrain adaptive control of advanced quadruped robots such as MIT Cheetah.
[0004] In the field of reinforcement learning, diffusion models were initially applied to offline reinforcement learning and imitation learning. In offline reinforcement learning, by using a pre-collected static data set to avoid interaction with the environment, diffusion models are used to reconstruct actions or perform behavior cloning, thereby enriching policy diversity under multi-modal distributions. However, these methods often have difficulty adapting to real-time feedback and rely too much on fixed data sets. Imitation learning focuses on directly learning policies from expert data and maintains certain advantages in sparse or missing reward scenarios. However, its dependence on data quality and diversity makes it difficult to adapt to changing environments or new goals.
[0005] In contrast, online reinforcement learning can potentially achieve better final performance by continuously interacting with the environment to collect richer samples. However, directly integrating diffusion models into online learning still faces the following main challenges:
[0006] 1. Diffusion Model Training Dependency: The variational lower bound used to train diffusion models depends on the availability of "good" samples and cannot improve the policy solely through optimization. In an online learning environment, due to the lack of a prior dataset, diffusion models cannot directly improve the policy through behavior cloning loss, making it more challenging to update diffusion-based policies in online reinforcement learning.
[0007] 2. Policy Expression Limitations: Due to the lack of an analytical density function, existing diffusion-based policies cannot introduce the principle of maximum entropy for optimization, forcing them to rely on a large number of diffusion steps to achieve sufficient exploration, which results in excessive decision-making time. Some current methods attempt to estimate the action entropy generated by diffusion policies through the expectation-maximization algorithm and Gaussian mixture models, or increase action randomness by adding uniformly distributed actions to cloned actions, but the implicit entropy regularization terms of these methods cannot be directly used for policy updates.
[0008] 3. Unstable Value Estimation: The accuracy of Q-function estimation fluctuates with policy updates, and the multimodality of diffusion policies may amplify this effect, thus hindering the smooth convergence of the value function. This problem is particularly prominent in methods that require accurate learning of the Q-function gradient, as the alignment process may introduce biases, leading to suboptimal algorithm performance.
[0009] Computational Efficiency Issues: Existing diffusion policy methods usually need to learn optimal actions and uniform actions simultaneously, which results in high computational overhead. In addition, to achieve sufficient exploration, these methods often require a large number of diffusion steps, which significantly increases the decision-making time.
[0010] From a geometric perspective, some methods utilize the score structure of diffusion models to align the gradient of the Q-function with the score of the diffusion model, introducing the Q-score matching method. However, this method requires accurate learning of the Q-function gradient, and the alignment process may introduce biases. At the same time, due to the lack of an analytical expression for diffusion policies, these methods cannot introduce entropy loss, resulting in poor exploration performance.
[0011] Therefore, there is an urgent need for a new technical solution to solve the above problems, especially while maintaining the advantages of diffusion models, effectively addressing key challenges such as computational efficiency, policy expression, and value estimation stability. Summary of the Invention
[0012] The objective of the present invention is to provide a kernel density estimation-driven diffusion reinforcement learning method, system, and agent to solve the following problems in the prior art:
[0013] 1. The lack of an analytical expression for diffusion policies makes it difficult to introduce the principle of maximum entropy for optimization;
[0014] 2. Traditional methods require a large number of diffusion steps to achieve sufficient exploration, resulting in overly long decision-making times.
[0015] 3. The Q-function estimation fluctuates with policy updates, affecting the stability of the value function convergence.
[0016] 4. Existing methods have high computational costs, affecting the efficiency of practical applications.
[0017] The present invention is achieved through the following technical solutions:
[0018] In a first aspect, the present invention provides a diffusion reinforcement learning method driven by kernel density estimation, including:
[0019] Obtain the value network, diffusion policy, experience replay pool, and learning rate parameters of the control system within the intelligent agent, and initialize them.
[0020] Obtain the current state and sample new actions according to the diffusion policy.
[0021] Interact with the current environment to obtain new states and rewards, generate samples with the new states and rewards, and store the samples in the experience replay pool.
[0022] Sample N actions for each state, calculate the values of several batches of samples, and update the value network parameters.
[0023] Generate M actions through the diffusion policy, calculate the action entropy according to the Beta kernel density estimation, and update the action network parameters and entropy weights.
[0024] And output the updated action network parameters and entropy weights to the control system of the intelligent agent.
[0025] Preferably, the sampling of N actions includes:
[0026] Generate a diffusion policy, and the generation of the diffusion policy includes initializing Gaussian noise and constructing a reverse process mean function.
[0027] Set an iterative sampling function, and perform sampling through the iterative sampling function and the reverse process mean function to output N actions.
[0028] Preferably, the construction of the reverse process mean function includes:
[0029]
[0030] where μ θ (a K , s, k) is a parameterized mean function, a K is the action vector at the k-th step, s is the current state vector, k is the current diffusion step, and α k = 1 - β kis the noise scheduling parameter, is the cumulative noise parameter, β k is the noise variance parameter, a θ is the parameterized action prediction network, is the product noise parameter at the (k - 1)-th step.
[0031] Preferably, the setting iterative sampling function includes:
[0032]
[0033] where a k-1 is the action vector at the (k - 1)-th step, is the random noise for reparameterized sampling, is the product noise parameter at the (k - 1)-th step.
[0034] Preferably, the calculating the values of several batches of samples includes:
[0035] Form an action set from the N actions collected for each state and perform target calculation of Q value;
[0036]
[0037] where is the loss function of the Q function, E is the expectation, D is the experience replay buffer, s ′ is the next state, a is the current state, a ′ n is the next action sampled at the n-th time, r(s,a) is the reward function, γ is the discount factor, Q φ is the current Q network, is the target Q network, φ is the parameter of the Q network.
[0038] Preferably, the calculating the action entropy according to the Beta kernel density estimation includes:
[0039] Select the Beta kernel function and perform density estimation through the cross-reference strategy;
[0040]
[0041] where K B represents the Beta kernel function, x is the input vector, x i is the reference point vector, h is the bandwidth parameter, K is the dimension of the action space, x d is the value at the d-th dimension, α d is the first shape parameter of the Beta distribution, β d is the second shape parameter of the Beta distribution, B(α d ,β d ) is the Beta function;
[0042]
[0043] Wherein, is the estimated probability density, a i is the action for which the density is to be estimated, M is the number of samples, a m is the m-th reference sample.
[0044] Preferably, the updating of the action network parameters and entropy weights includes:
[0045] Calculating the average Q-value of a number of samples;
[0046]
[0047] Estimating the entropy value of the current policy;
[0048]
[0049] Optimizing the objective by minimizing the KL divergence;
[0050]
[0051] Updating the entropy weight;
[0052]
[0053] Wherein, is the average Q-value of a number of samples, s t is the state at time t, and are the 0-m actions sampled at time t, respectively, and are the double Q networks, H(π θ ) is the entropy of the policy, π θ is the parameterized policy, D KL is the KL divergence, λ is the weight coefficient of the uniform distribution, U is the uniform distribution, η is the entropy weight coefficient, and α is the learning rate.
[0054] In a second aspect, the present invention further provides a diffusion reinforcement learning system driven by kernel density estimation, including:
[0055] A diffusion policy module, configured to obtain the value network, diffusion policy, experience replay pool and learning rate parameters of the control system in the intelligent body and initialize them; obtain the current state, sample a new action according to the diffusion policy; interact with the current environment to obtain a new state and reward, generate a sample with the new state and reward, and store the sample in the experience replay pool;
[0056] The kernel density estimation and entropy optimization module is configured to sample N actions for each state, calculate the value of several batches of samples, and update the value network parameters; generate M actions through the diffusion strategy, calculate the action entropy according to the Beta kernel density estimation, and update the action network parameters and entropy weights;
[0057] The Q-function update module is configured to output the updated action network parameters and entropy weights to the control system of the intelligent agent;
[0058] The main control module is connected to the diffusion strategy module, the kernel density estimation and entropy optimization module, and the Q-function update module, and is used to execute the above-mentioned diffusion reinforcement learning method driven by kernel density estimation.
[0059] In a third aspect, the present invention also provides an intelligent agent, including a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the computer program, the above-mentioned diffusion reinforcement learning method driven by kernel density estimation is implemented.
[0060] In a fourth aspect, the present invention also provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the above-mentioned diffusion reinforcement learning method driven by kernel density estimation is implemented.
[0061] The technical solution of the present invention has at least the following advantages and beneficial effects:
[0062] The present invention realizes reinforcement learning by combining kernel density estimation with a diffusion model, and has significant advantages in terms of technical principles and application effects. At the basic technical level, the present invention solves the technical problem that the existing diffusion strategy lacks an analytical expression. By adopting the Beta kernel density estimation, an accurate estimation of the policy distribution is realized, the policy optimization process is made analytically tractable, and theoretical convergence guarantees are provided. At the same time, the present invention overcomes the technical defect that a large number of diffusion steps are required in traditional methods. Through an improved reverse process design, the required diffusion steps are significantly reduced, the decision-making time for action generation is greatly reduced, the real-time response ability of the system is improved, and the working efficiency and response time of the intelligent agent in practical applications are improved. In addition, the present invention also solves the technical problem of unstable Q-function estimation. The innovative multiple sampling mechanism combined with the double Q-network structure effectively improves the stability of value estimation, reduces the overestimation risk, and enhances the convergence performance of the algorithm.
[0063] In terms of specific technical advantages, the present invention has achieved significant improvements in the process of policy generation. The improved diffusion policy can not only generate actions more efficiently, but also provide more stable policy output, while effectively reducing the consumption of computing resources. In terms of distribution estimation, the present invention provides a more accurate policy distribution estimation through the Beta kernel density estimation method, better handles the particularity of the bounded action space, and significantly enhances the generalization ability of the algorithm. In terms of exploration efficiency, the present invention realizes more effective exploration of the state space through an optimized entropy adjustment mechanism, provides a better exploration-exploitation balance, and enhances the ability to discover the global optimal solution.
[0064] In terms of application value, the present invention significantly improves the generality of the reinforcement learning algorithm. The improved algorithm is not only applicable to various continuous control tasks, but also can well support multi-modal action distributions, and has good task adaptability. In terms of practicality, the present invention reduces the complexity of system implementation, improves the adjustability of the algorithm, and enhances the flexibility of actual deployment. In terms of performance, through a series of technical innovations, the present invention significantly improves the quality of task completion, enhances the stability of the system, and improves the long-term operation performance.
[0065] In terms of innovative contributions, the present invention proposes a new policy optimization framework at the theoretical level, establishes a complete theoretical analysis system, and expands the technical boundaries of reinforcement learning. At the method level, the present invention innovatively combines kernel density estimation with diffusion models, proposes a new entropy optimization mechanism, and designs an efficient multi-sampling strategy. At the application level, the present invention creates a new algorithm application paradigm, expands the technical application scenarios, and provides more implementation possibilities for related fields. BRIEF DESCRIPTION OF THE DRAWINGS
[0066] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following will briefly introduce the drawings required for the embodiments. It should be understood that the following drawings only show some embodiments of the present invention, and therefore should not be regarded as limiting the scope. For those of ordinary skill in the art, without creative efforts, other related drawings can also be obtained based on these drawings.
[0067] Figure 1 It is a schematic flowchart of the present invention;
[0068] Figure 2 It is the global optimal solution that the system of the present invention can reach;
[0069] Figure 3 It is all the peak points identified by the system of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS
[0070] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the following will clearly and completely describe the technical solutions in the embodiments of the present invention with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are some, but not all, of the embodiments of the present invention. The components of the embodiments of the present invention usually described and illustrated in the accompanying drawings here can be arranged and designed in various different configurations.
[0071] In the division of modules in this application, it is a logical division. In actual implementation, there may be other division methods. For example, multiple modules can be combined or integrated into another system, or some features can be ignored or not executed.
[0072] The independently described modules or sub-modules can be physically separated or not: they can be implemented in software or in hardware, and some modules or sub-modules can be implemented in software, and the processor calls the software to implement the functions of this part of the modules or sub-modules, and other parts of the modules or sub-modules are implemented in hardware, such as through a hardware circuit. In addition, some or all of the modules can be selected according to actual needs to achieve the purpose of the solution of this application.
[0073] Please refer to Figures 1 - 3 , a diffusion reinforcement learning method driven by kernel density estimation provided by the present invention includes:
[0074] S101: Obtain the value network, diffusion strategy, experience replay pool, and learning rate parameter of the control system in the intelligent body, and initialize them;
[0075] The value network is a neural network used to evaluate the long-term expected return of a state or a state-action pair. It includes a state value function: evaluating the quality of the current state. An action value function (Q value): evaluating the expected return of performing an action in a certain state. It acts as an "evaluator" in reinforcement learning, guiding policy optimization. Helping the algorithm quickly screen high-value paths in a complex environment and reducing the search space.
[0076] The diffusion strategy generates an action sequence by gradually denoising, and is suitable for high-dimensional continuous action spaces (such as robot control). The diffusion model was originally used to generate high-quality images and was later introduced into the field of reinforcement learning to generate more robust action policies. It processes complex action distributions, generates smoother and safer actions (such as robotic arm grasping), and is highly robust to noise and environmental changes.
[0077] The experience replay pool is a buffer for storing historical data (state, action, reward, next state) of the interaction between the intelligent body and the environment, and is used for random sampling during subsequent training. The learning rate parameter is a hyperparameter that controls the update step size of the neural network parameters and determines the amplitude of the adjustment of the model parameters during each gradient descent.
[0078] In this embodiment, the applied value network, diffusion policy, experience replay pool, and learning rate parameters all serve the control of the agent, i.e., the robot.
[0079] S102: Obtain the current state and sample a new action according to the diffusion policy;
[0080] In this example, the state can be sensor observations, such as robot joint angles, camera images, etc. A neural network (such as UNet) is used to predict the noise. The input includes the noisy action and the current state. Starting from random noise, the noise is gradually denoised to generate the action. The current state is used as the conditional input, and the reverse denoising process is called in the policy to generate the action under the current state.
[0081] For example, for the robot grasping task:
[0082] State: Manipulator joint angles + camera image;
[0083] Action: Target position of the end effector;
[0084] The diffusion policy generates a smooth and safe grasping trajectory.
[0085] S103: Interact with the current environment to obtain a new state and reward, generate a sample with the new state and reward, and store the sample in the experience replay pool;
[0086] S104: Sample N actions for each state, calculate the values of several batches of samples, and update the value network parameters;
[0087] S105: Generate M actions through the diffusion policy, calculate the action entropy according to the Beta kernel density estimation, update the action network parameters and the entropy weight; and output the updated action network parameters and entropy weight to the control system of the agent.
[0088] In the specific algorithm execution process, the system first enters the diffusion policy generation stage. This stage starts by sampling the initial noise from the standard Gaussian distribution and gradually transforms the noise into meaningful actions through a carefully designed reverse diffusion process. In this process, the application of the reparameterization trick significantly improves the sampling efficiency while ensuring the differentiability of the gradient. Subsequently, the system enters the Q-function update stage, where multiple action samples are sampled to enhance the stability of the value estimation. This multi-sampling mechanism not only provides a more accurate value function estimation but also effectively alleviates the overestimation problem in traditional methods.
[0089] After completing the initial action generation and value estimation, the system enters the density estimation phase. In this phase, through the precise calculation of the Beta kernel function and the combination of cross-reference estimation strategies, the system can accurately grasp the characteristics of the action distribution. This density estimation method based on the Beta kernel is particularly suitable for dealing with bounded action spaces and provides more accurate estimation results than traditional Gaussian kernels. Finally, in the policy optimization phase, the system realizes continuous improvement of the policy through steps such as calculating the batch average Q-value, estimating the policy entropy, optimizing the KL divergence, and updating the entropy weight.
[0090] In an exemplary embodiment of the present invention, the sampling of N actions includes:
[0091] Generating a diffusion policy, where the generating of the diffusion policy includes initializing Gaussian noise and constructing a reverse process mean function; setting an iterative sampling function, and sampling through the iterative sampling function and the reverse process mean function to output N actions.
[0092] Specifically, the generation of the diffusion policy is the first core step of the present invention;
[0093] Initializing Gaussian noise:
[0094]
[0095] where: p(a T ) represents the action vector at the T-th step of the diffusion process, represents a multi-dimensional Gaussian distribution with a mean of 0 and a covariance matrix of the identity matrix, and T represents the total number of steps of the diffusion process.
[0096] Specifically, constructing the reverse process mean function includes:
[0097]
[0098] In the formula, μ θ (a K , s, k) is a parameterized mean function, a K is the action vector at the k-th step, s is the current state vector, k is the current diffusion step, α K = 1 - β K is the noise scheduling parameter, is the cumulative noise parameter, β K is the noise variance parameter, a θ is a parameterized action prediction network, is the cumulative noise parameter at the (k - 1)-th step.
[0099] This function directly predicts the action mean through a parameterized network instead of noise prediction in traditional diffusion models, and this improvement significantly improves the generation efficiency.
[0100] Specifically, setting the iterative sampling function includes:
[0101]
[0102] where a K-1 is the action vector at the (k - 1)-th step, is the random noise for reparameterized sampling, is the product noise parameter at the (k - 1)-th step.
[0103] Through this iterative process, the initial noise is gradually transformed into meaningful actions while ensuring the stability of the sampling process.
[0104] In an exemplary embodiment of the present invention, after obtaining the actions generated by the policy, it is necessary to evaluate the value of these actions. The present invention adopts an innovative multi-sampling mechanism to improve the evaluation accuracy: First, sample multiple actions for each state to form an action set. Then, calculate the Q-value through an enhanced TD target.
[0105] Specifically, calculating the value of several batches of samples includes:
[0106] Form an action set from the N actions collected for each state and calculate the Q-value for the target;
[0107]
[0108] where is the loss function of the Q function, E is the expectation, D is the experience replay buffer, s ′ is the next state, a is the current state, a ′ n is the next action sampled at the n-th time, r(s, a) is the reward function, γ is the discount factor, Q φ is the current Q network, is the target Q network, and φ is the parameter of the Q network.
[0109] This multi-sampling mechanism effectively alleviates the instability of value estimation, especially when dealing with multi-modal action distributions.
[0110] In an exemplary embodiment of the present invention, calculating the action entropy according to the Beta kernel density estimation includes:
[0111] Select the Beta kernel function and perform density estimation through cross-reference strategy;
[0112]
[0113] where K B represents the Beta kernel function, x is the input vector, x iis the reference point vector, h is the bandwidth parameter, K is the dimension of the action space, and x d is the value of the d-th dimension, and α d is the first shape parameter of the Beta distribution, and β d is the second shape parameter of the Beta distribution, and B(α d , β d ) is the Beta function;
[0114]
[0115] In the formula, is the estimated probability density, a i is the action for which the density is to be estimated, M is the number of samples, and a m is the m-th reference sample.
[0116] Adopting the above method not only considers the boundary constraints of the action space but also provides a more accurate density estimate.
[0117] An exemplary embodiment of the present invention for updating the action network parameters and entropy weights includes:
[0118] Calculating the average Q-value of a number of samples;
[0119]
[0120] Estimating the entropy value of the current policy;
[0121]
[0122] Optimizing the objective by minimizing the KL divergence;
[0123]
[0124] Updating the entropy weight;
[0125]
[0126] In the formula, is the average Q-value of a number of samples, s t is the state at time t, and are the 0-m actions sampled at time t respectively, and are the double Q-networks, H(π θ ) is the entropy of the policy, π θ is the parameterized policy, D KL is the KL divergence, λ is the weight coefficient of the uniform distribution, U is the uniform distribution, η is the entropy weight coefficient, and α is the learning rate.
[0127] Through the close cooperation of these four steps, the present invention achieves an efficient policy learning process. In particular, the dynamic entropy weight adjustment ensures an appropriate balance between exploration and exploitation, while the KL divergence optimization helps the policy converge to high-value regions while maintaining diversity.
[0128] The reinforcement learning system provided by the present invention mainly consists of four core modules. Among them, the diffusion policy module is responsible for generating action samples and includes three key components: an action generator, a parameter optimizer, and a noise scheduler. The action generator directly outputs actions based on an improved diffusion process, the parameter optimizer is responsible for updating network parameters to improve performance, and the noise scheduler dynamically controls the noise level in the diffusion process. The kernel density estimation module jointly completes the accurate estimation of the policy distribution through its kernel function calculation unit, density estimator, and cross-validation unit. The entropy optimization module includes a KL divergence calculator, a weight adjuster, and an exploration controller for achieving the dynamic balance between exploration and exploitation. The Q-function update module ensures the stability and accuracy of the value function estimation through its double Q-network structure, sampling controller, and target network updater.
[0129] A diffusion reinforcement learning system driven by kernel density estimation, comprising:
[0130] A diffusion policy module, configured to obtain the value network, diffusion policy, experience replay pool, and learning rate parameters of the control system within the intelligent agent and initialize them; obtain the current state, sample new actions according to the diffusion policy; interact with the current environment to obtain new states and rewards, generate samples with the new states and rewards, and store the samples in the experience replay pool;
[0131] A kernel density estimation and entropy optimization module, configured to sample N actions for each state, calculate the value of several batches of samples, and update the value network parameters; generate M actions through the diffusion policy, calculate the action entropy according to the Beta kernel density estimation, and update the action network parameters and entropy weights;
[0132] A Q-function update module, configured to output the updated action network parameters and entropy weights to the control system of the intelligent agent;
[0133] A main control module, connected to the diffusion policy module, the kernel density estimation and entropy optimization module, and the Q-function update module, for executing the above-mentioned reinforcement learning method based on a diffusion model and a kernel density estimation.
[0134] To verify the effectiveness of the present invention, a typical multi-modal Gaussian test environment was designed. In this environment, the task of the agent is to start from the origin in a 10×10 two-dimensional grid space and search for and reach four Gaussian peak points with different reward values. These peak points are located at coordinates (-4, -4), (-5, 5), (5, -5), and (4, 4) respectively. Among them, the reward values of two peak points are 150, and the other two are 100. This asymmetric reward setting increases the difficulty of the task and provides a good verification environment for the performance test of the algorithm.
[0135] During the experiment, the system was first initialized, including the initialization of network parameters, the setting of diffusion steps, the adjustment of the learning rate, and the determination of the discount factor. In the training stage, we set reasonable episode limits and the total number of training episodes, and used an experience replay buffer large enough to store historical experiences. Through this setting, it is ensured that the algorithm can fully learn and adapt to the environmental characteristics. During the training process, the system demonstrated strong exploration ability and learning efficiency, successfully discovered all Gaussian peak points, and was able to accurately identify the global optimal solution. Figure 2 Indicates that the system can reach the global optimal solution, Figure 3 Indicates that the system has identified all peak points.
[0136] In addition, for the purpose of applying to the final agent of this solution, this embodiment also gives an example of applying to a quadruped robot.
[0137] 1. Dynamic gait generation for complex terrains
[0138] When a quadruped robot walks in an unstructured environment (such as a pile of rocks, a slope, or a muddy road surface), it needs to generate multi-modal gaits to balance stability and mobility. The diffusion strategy module of the present invention can optimize gait control in the following ways:
[0139] Multi-modal action generation: The improved reverse diffusion process directly generates joint angle sequences, adapting the selection of supporting legs and the swing amplitude for different terrains.
[0140] Real-time decision optimization: By reducing the number of diffusion steps to 10 - 20 steps (the traditional method requires 50 - 100 steps), the action generation delay is reduced to the 10ms level, meeting the real-time control requirements.
[0141] Constraint handling: Beta kernel density estimation can accurately model the bounded constraints of joint angles (such as the ±90° rotation limit of the hip joint), avoiding mechanical damage.
[0142] 2. Dynamic obstacle avoidance and path planning
[0143] In a narrow space or a dynamic environment (such as post-disaster ruins search and rescue), the quadruped robot needs to adjust its motion strategy online:
[0144] Entropy-driven Exploration: Through the KL divergence optimization mechanism, the system actively generates diverse actions (such as jumping and side-shifting) in unknown areas, improving the environmental coverage efficiency.
[0145] Multi-objective Balance: The double Q-network and multi-sampling mechanism can simultaneously evaluate the safety (such as collision risk) and efficiency (such as path length) of actions, achieving dynamic priority adjustment.
[0146] In addition, in each embodiment of the present invention, each functional unit can be integrated into a processing unit, or each unit can exist physically alone, or two or more units can be integrated into one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of a software functional unit.
[0147] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. This computer software product is stored in a storage medium and includes several instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods in each embodiment of the present invention. The foregoing storage medium includes: various media such as USB flash drives, mobile hard disks, read-only memories (ROMs), random access memories (RAMs), magnetic disks, or optical discs that can store program codes.
[0148] The above are only the preferred embodiments of the present invention and are not used to limit the present invention. For those skilled in the art, the present invention can have various changes and modifications. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included within the protection scope of the present invention.
Claims
1. A diffusion reinforcement learning method driven by kernel density estimation, characterized in that, including: Obtain the value network, diffusion strategy, experience replay pool, and learning rate parameters of the intelligent body's control system, and initialize them; Obtain the current state and sample new actions according to the diffusion strategy; Interact with the current environment to obtain new states and rewards, generate samples with the new states and rewards, and store the samples in the experience replay pool; Sample N actions for each state, calculate the values of several batches of samples, and update the value network parameters; Generate M actions through the diffusion strategy, calculate the action entropy according to the Beta kernel density estimation, and update the action network parameters and entropy weights; And output the updated action network parameters and entropy weights to the control system of the intelligent body.
2. The diffusion reinforcement learning method driven by kernel density estimation according to claim 1, wherein The sampling of N actions includes: Generate a diffusion strategy, and the generation of the diffusion strategy includes initializing Gaussian noise and constructing a reverse process mean function; Set an iterative sampling function, and sample through the iterative sampling function and the reverse process mean function to output N actions.
3. A diffusion reinforcement learning method driven by kernel density estimation according to claim 2, characterized in that, The construction of the reverse process mean function includes: where μ θ (a k , s, k) is a parameterized mean function, a k is the action vector at the k-th step, s is the current state vector, k is the current diffusion step, α k = 1 - β k is the noise schedule parameter, is the cumulative noise parameter, β k is the noise variance parameter, a θ is the parameterized action prediction network, is the product noise parameter at the (k - 1)-th step.
4. A diffusion reinforcement learning method driven by kernel density estimation according to claim 3, characterized in that The setting of the iterative sampling function includes: where a k-1 is the action vector at the (k - 1)-th step, is the random noise for reparameterized sampling.
5. A diffusion reinforcement learning method driven by kernel density estimation, characterized in that The calculation of the values of several batches of samples includes: The N actions collected for each state form an action set, and calculate the target Q value; In the formula, is the loss function of the Q function, E is the expectation, D is the experience replay buffer, s ′ is the next state, a is the current state, a ′ n is the next action of the nth sample, r(s,a) is the reward function, γ is the discount factor, Q φ is the current Q network, is the target Q network, and φ is the parameter of the Q network.
6. A diffusion reinforcement learning method driven by kernel density estimation, characterized in that, The calculation of the action entropy according to the Beta kernel density estimation includes: Select a Beta kernel function and perform density estimation through a cross-reference strategy; Where K B represents the Beta kernel function, x is the input vector, x i is the reference point vector, h is the bandwidth parameter, K is the dimension of the action space, x d is the value of the d-th dimension, α d is the first shape parameter of the Beta distribution, β d is the second shape parameter of the Beta distribution, B(α d , β d ) is the Beta function; wherein, is the estimated probability density, a i is the action of the density to be estimated, M is the number of samples, and a m is the m-th reference sample.
7. A diffusion reinforcement learning method driven by kernel density estimation according to claim 5, characterized in that, The update of the action network parameters and entropy weights includes: Calculate the average Q value of several samples; Estimate the entropy value of the current strategy; Optimize the target by minimizing the KL divergence; Update the entropy weight; Wherein, is the average Q value of a number of samples, s t is the state at time t, and are the 0-m actions sampled at time t respectively, and are the double Q networks, H(π θ ) is the entropy of the policy, π θ is the parameterized policy, D KL is the KL divergence, λ is the weight coefficient of the uniform distribution, U is the uniform distribution, η is the entropy weight coefficient, and α is the learning rate.
8. A diffusion reinforcement learning system driven by kernel density estimation, characterized in that, including: A diffusion strategy module, configured to obtain the value network, diffusion strategy, experience replay pool, and learning rate parameters of the intelligent body's control system, and initialize them; Obtain the current state, sample new actions according to the diffusion strategy; interact with the current environment to obtain new states and rewards, generate samples with the new states and rewards, and store the samples in the experience replay pool; A kernel density estimation and entropy optimization module, configured to sample N actions for each state, calculate the values of several batches of samples, and update the value network parameters; generate M actions through the diffusion strategy, calculate the action entropy according to the Beta kernel density estimation, and update the action network parameters and entropy weights; A Q function update module, configured to output the updated action network parameters and entropy weights to the control system of the intelligent body; A main control module, connected to the diffusion strategy module, the kernel density estimation and entropy optimization module, and the Q function update module, for executing a diffusion reinforcement learning method driven by kernel density estimation according to any one of claims 1-7.
9. An agent, comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements a diffusion reinforcement learning method driven by kernel density estimation according to any one of claims 1-7.
10. A computer-readable storage medium, characterized in that, A computer program is stored on the computer-readable storage medium, and when the computer program is executed by a processor, it implements a diffusion reinforcement learning method driven by kernel density estimation according to any one of claims 1-7.
