A robot skill learning method based on fusion algorithm

Abstract

The application discloses a robot skill learning method based on a fusion algorithm, comprising the following steps: constructing a robot experience replay pool and an expert experience pool in advance, and initializing an action policy network and a state value network; further comprising the following steps: storing experience data into the robot experience replay pool by interacting with an environment, and cooperatively updating the action policy network and the state value network based on mixed batch data sampled from the robot experience replay pool and the expert experience pool; and outputting and executing actions by the robot according to a perceived environment state by using the trained action policy network. The application constructs a double experience pool structure of the expert experience pool and the robot experience replay pool, adopts a mixed batch sampling and cooperative updating mechanism, rapidly improves the strategy performance by using expert demonstration, guarantees the training stability, independently explores and optimizes the strategy by using reinforcement learning, and finally realizes efficient and stable convergence of sample robot skill learning.

Description

Technical Field

[0001] This invention belongs to the field of robot skill learning technology, specifically relating to a robot skill learning method based on a fusion algorithm. Background Technology

[0002] As industrial manufacturing evolves towards smaller batches, more diverse varieties, and greater flexibility and intelligence, collaborative robots, as core equipment in intelligent manufacturing, are expected to perform diverse tasks in complex, dynamic, and unstructured environments. Traditional industrial robots rely on teach-and-playback or offline programming, resulting in fixed operating skills that are difficult to adapt to changes in task parameters, let alone learn and improve autonomously through interaction with the environment.

[0003] Therefore, the field of robot skill learning is increasingly adopting artificial intelligence-based methods, especially deep reinforcement learning. Deep reinforcement learning enables robots to learn optimal strategies by allowing them to interact with their environment through trial and error, aiming to maximize cumulative rewards, thus giving robots the potential to cope with complex scenarios.

[0004] However, applying deep reinforcement learning directly to physical robots faces severe challenges. First, its sample efficiency is extremely low, requiring massive amounts of interactive data, which is costly in terms of time and money on physical platforms. Second, the stochastic exploration of the strategy in the early stages of training leads to a slow learning process. Furthermore, training based on random initialization has poor stability and is difficult to converge.

[0005] To overcome the inherent limitations of deep reinforcement learning, imitation learning has been introduced as a supplement. By directly mimicking demonstration data from human experts, imitation learning can quickly initialize policies, avoiding blind exploration in the early stages of training. However, simple imitation learning also has limitations: firstly, its performance is severely restricted by the quality and scale of the expert data, making it difficult to surpass the level of the demonstrator; secondly, in dynamic environments, it lacks the ability to optimize and adapt through autonomous interaction.

[0006] Existing technologies have attempted to combine reinforcement learning with imitation learning, such as simply mixing expert data into the experience replay pool of reinforcement learning. However, this approach does not fundamentally solve the problem of efficiently integrating the two learning paradigms.

[0007] Specifically, designing a mechanism that allows robots to fully utilize expert data guidance in the early stages of training to rapidly improve strategy performance, while also smoothly transitioning in the later stages to leverage the autonomous exploration and optimization capabilities of reinforcement learning, and ensuring the stability of the entire training process, remains a pressing technical challenge in this field. The lack of this deep integration means that existing robot skill learning methods have not achieved ideal results in terms of efficiency, stability, and final performance, thus hindering further improvements in the intelligence level of collaborative robots. Summary of the Invention

[0008] This invention provides a robot skill learning method based on a fusion algorithm, which can efficiently and stably obtain operation strategies adapted to complex scenarios through the collaborative optimization mechanism of deep reinforcement learning and imitation learning. It effectively solves the technical problems in the prior art, such as low efficiency of deep reinforcement learning samples, high exploration risk, and imitation learning performance limited by expert data, which lead to low robot skill learning efficiency, unstable training process, and difficulty in surpassing the level of expert demonstration.

[0009] The technical solution adopted in this invention is as follows: A robot skill learning method based on a fusion algorithm includes: Pre-build the robot experience playback pool and expert experience pool, and initialize the action policy network and state value network; The method also includes, The robot stores experiential data in the robot's experiential replay pool by interacting with the environment, and updates the action policy network and the state value network collaboratively based on mixed batch data sampled from the robot's experiential replay pool and the expert experience pool. The trained action policy network enables the robot to output and execute actions based on the perceived environmental state.

[0010] The robot skill learning method based on fusion algorithm disclosed in this invention also has the following additional technical features: Collaboratively updating the action policy network includes: Based on data sampled from the robot's experience replay pool, a first gradient is obtained to indicate the update direction of the action policy network; Based on the data sampled from the expert experience pool, a mimicry loss is obtained, and a second gradient is obtained based on the mimicry loss; The first gradient and the second gradient are fused together to update the parameters of the action policy network.

[0011] When imitation loss is obtained, a selective imitation strategy is executed: For an expert state-action pair sampled from the expert experience pool, the difference between the output action of the current action policy network in the expert state and the expert action is obtained; When the estimated value of the state-action value function corresponding to the expert action is higher than the estimated value of the state-action value function corresponding to the output action, the difference is included in the imitation loss.

[0012] The first gradient and the second gradient are fused together, specifically as follows: The first gradient and the second gradient are fused by weighted summation. The weight coefficient of the second gradient decreases dynamically as the training period increases.

[0013] Constructing mixed batch data, specifically: Hybrid batch data is constructed by sampling from the robot experience replay pool and the expert experience pool. The proportion of data sampled from the expert experience pool decreases dynamically as the training cycle increases.

[0014] The state value network is updated collaboratively, specifically as follows: The target temporal difference error is obtained by calculating the target network. The state value network is updated using gradient descent based on the temporal difference error target.

[0015] The target network also includes: After the action policy network and the state value network are updated, the target network is softly updated. The soft update parameter ranges from 0.001 to 0.01.

[0016] The robot experience replay pool and the expert experience pool are specifically as follows: The experience data stored in the robot's experience replay pool is in the form of a quadruple, including the current state, the action performed, the immediate reward, and the next state; The data stored in the expert experience pool is in the form of state-action pairs, including the current state and the action to be performed.

[0017] The present invention also provides a robot skill learning system for implementing any one of the methods described above, the system comprising: One or more processors; Memory; And one or more computer programs, which are stored in the memory and configured to be executed by the one or more processors, the one or more computer programs including instructions for performing any of the methods.

[0018] The present invention further provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements any one of the methods described herein.

[0019] Due to the adoption of the above technical solution, the beneficial effects achieved by this invention are as follows: 1. In this invention, by introducing an expert experience pool and using its data for network updates, high-quality behavioral demonstrations are provided in the early stages of training. This greatly reduces the large number of invalid or dangerous interactions caused by random exploration in the early stages of deep reinforcement learning, thereby significantly reducing the number of samples and time costs required for training. This makes direct or simulated training on physical robots more economical and feasible, and significantly improves the efficiency and feasibility of skill learning.

[0020] Furthermore, expert data, as prior knowledge, provides a clear and reliable initial direction for the optimization of the policy network, effectively preventing the policy network from falling into local optima or completely failing due to poor initialization and early exploration, thereby improving the stability of the training process. Simultaneously, since policy learning begins with imitating expert behavior, the policy performance during training is consistently maintained above a high baseline level, effectively ensuring the stability of the training process and the lower bound of policy performance.

[0021] Moreover, this method does not simply superimpose two learning paradigms. Instead, it uses mixed batch data to simultaneously perform imitation learning and reinforcement learning within a single training batch, achieving a complementary advantage between the two and generating optimized policies that surpass expert levels. This collaborative update mechanism enables the policy network not only to quickly master expert skills but also to explore behavioral strategies that are superior and more adaptive than expert demonstrations through continuous interaction with the environment, thus overcoming the bottleneck of performance limitations imposed by simple imitation learning on expert data. Attached Figure Description

[0022] The accompanying drawings, which are included to provide a further understanding of the invention and form part of this invention, illustrate exemplary embodiments of the invention and are used to explain the invention, but do not constitute an undue limitation of the invention. In the drawings: Figure 1 This is a flowchart illustrating the robot skill learning method based on a fusion algorithm according to one embodiment of the present invention. Figure 2 This is a system architecture diagram of the robot skill learning method based on fusion algorithm according to one embodiment of the present invention. Detailed Implementation

[0023] To more clearly illustrate the overall concept of the present invention, a detailed description will be provided below with reference to the accompanying drawings and examples.

[0024] Many specific details are set forth in the following description in order to provide a full understanding of the invention. However, the invention may also be practiced in other ways different from those described herein, and therefore the scope of protection of the invention is not limited to the specific embodiments disclosed below.

[0025] like Figure 1 and Figure 2 As shown, a robot skill learning method based on a fusion algorithm includes: S000: Pre-build the robot experience playback pool and expert experience pool, and initialize the action policy network and state value network.

[0026] The main purpose of this step is to establish a structured data storage framework and a trainable parameterized model for subsequent fusion learning, laying the physical and algorithmic foundation for achieving deep collaboration between imitation learning and reinforcement learning.

[0027] First, two experience pools need to be constructed that are independent in data structure and function but complementary to each other. The robot experience replay pool is used to dynamically store the experience data generated by the robot through trial and error interaction in real or simulated environments. This pool is empty at the beginning, and its contents will gradually accumulate as the training process progresses. It is the core data source for reinforcement learning components to evaluate and optimize policies.

[0028] In parallel, an expert experience pool needs to be constructed to pre-store state-action pair data generated by human experts through teaching devices (such as teach pendants, motion capture systems, etc.). The separate design of these two pools ensures that data from different sources (autonomous exploration and expert demonstration) are logically and clearly isolated, providing structural convenience and assurance for subsequent implementation of mixed sampling according to specific strategies, and avoiding data confusion.

[0029] At the same time, it is necessary to initialize the deep neural network model, specifically including the action policy network (Actor) and the state value network (Critic). The action policy network takes the environmental state as input and outputs the robot's actions; its function is to represent and execute skill policies.

[0030] State-value networks take states and actions as joint inputs and output a scalar value to evaluate the long-term expected value of performing an action in a given state.

[0031] During network initialization, its parameters are typically initialized using a random initialization strategy, such as Xavier initialization or He initialization, to ensure the stability of gradient flow in the early stages of training. Furthermore, based on deep reinforcement learning, a target network corresponding to the aforementioned network structure can be initialized simultaneously, with its parameters initially set to be the same as the main network. This aims to improve the stability of the learning process through a soft update strategy during subsequent training.

[0032] By designing a physically separate but logically connected dual experience pool, a clear and independent data source is provided for the subsequent fusion algorithm. This enables the combination of rapid guidance in imitation learning and autonomous exploration in reinforcement learning, avoiding policy learning objective conflicts caused by mixed data sources in a single experience pool.

[0033] Furthermore, through the systematic initialization of neural networks, a nonlinear mapping capability from high-dimensional perceptual input to complex action output was established, providing an optimizable function approximator for skill learning and generalization.

[0034] The method also includes, S100: By interacting with the environment, experience data is stored in the robot experience playback pool, and the action policy network and the state value network are collaboratively updated based on the mixed batch data sampled from the robot experience playback pool and the expert experience pool.

[0035] The main purpose of this step is to achieve a dynamic balance and deep integration between rapid guidance of imitation learning and autonomous exploration of reinforcement learning by designing a structured data utilization and network optimization mechanism.

[0036] First, environmental interaction and data collection are performed. The robot agent operates in a simulated or real environment, based on the current action policy network μ. p The output action (with a small amount of random noise added to encourage exploration) interacts with the environment. The environment then returns an immediate reward r and a new state s. next This interactive experience is represented as a tuple (s, a, r, s next The data is stored in the robot's experience replay pool in the form of [database name missing]. This process ensures that the agent's real-world interaction experiences gained through trial and error are continuously recorded, providing a dynamically updated data foundation for reinforcement learning components.

[0037] Subsequently, a hybrid training batch is constructed. N samples are randomly selected from the robot's experience replay pool. N empirical data points, and N random samples from the expert experience pool. The state-action pairs together constitute a batch of size N. +N Mixed batches.

[0038] Then, a collaborative network update is performed. Using the constructed mixed batch data, the action policy network and the state value network are collaboratively optimized. For the state value network Q... v The updated target y combines the immediate reward with the target network's estimate of future value. The accuracy of its value estimation is improved by minimizing the mean squared error loss between the current value network output and the target y, and by using gradient descent to update the parameters v.

[0039] For the action policy network μ p Its update gradient consists of two parts working together: one part originates from the reinforcement learning objective, which aims to select the state-value network Q that makes the Q-value network more efficient. vOne part is based on maximizing the output action; the other part is based on imitation learning, which calculates the output action of the policy network and the expert action 'a' in the mixed batch. j The differences between the two gradients (such as mean squared error) constitute the imitation loss. Finally, the two gradients are weighted and fused to jointly update the policy network parameters p, ensuring that the policy both learns from expert demonstrations and autonomously evolves in the direction of maximizing long-term rewards.

[0040] This training loop continues until the policy network performance converges or the preset number of iterations is reached.

[0041] By constructing hybrid batches from dual-source experience pools and utilizing both types of data simultaneously in a single update step, imitation learning and reinforcement learning are integrated at the gradient level in an instant and dynamic manner, rather than simply switching stages or mixing data, thus fundamentally promoting the complementary advantages of the two paradigms.

[0042] Furthermore, the collaborative update mechanism ensures that value assessment and strategy improvement always refer to both expert knowledge and practical interaction experience, jointly guiding the network parameters to evolve towards a better solution space, effectively improving the overall learning efficiency, stability, and the upper limit of the final strategy's performance.

[0043] S200: Using the trained action policy network, the robot outputs and executes actions based on the perceived environmental state.

[0044] The main purpose of this step is to deploy and execute the optimal skill strategies obtained during the training phase in the actual physical system, thereby achieving a connection from learning to application and ultimately completing the autonomous task.

[0045] First, deploy the policy network. This involves deploying the fully trained action-policy network μ that meets performance requirements. p The parameters are fixed and loaded into the computing unit (such as an industrial computer, embedded GPU, or dedicated neural computing accelerator) in the robot control system. This deployment process ensures that the knowledge gained from training is completely transferred to the execution end, providing core intelligence for the robot's online decision-making.

[0046] Subsequently, the system enters an autonomous operation cycle. The robot perceives and analyzes the current environmental state s in real time through its multimodal sensing system (such as 3D vision sensors, six-dimensional force sensors, joint encoders, etc.). This state s is then fed into the deployed action policy network μ. p The network uses forward reasoning to instantly output the optimal action 'a' corresponding to the current state, i.e., a = μ, without needing to interact with the environment through trial and error. p(s). The motion command (such as the joint target angle, end effector target pose or velocity, etc.) is sent to the robot's underlying motion controller to drive the robotic arm, gripper and other actuators to accurately complete the specified operation, such as precision assembly, workpiece gripping or trajectory tracking.

[0047] This perception-decision-execution loop will continue to run until the entire task is completed. During this process, the strategies acquired in the training phase enable the robot to cope with dynamically changing environmental conditions during the task, demonstrating complex decision-making abilities and operational skills.

[0048] This step achieves the final closed loop between the artificial intelligence model and the physical robot system, applying complex skills learned offline or in simulation. The deployed system involves only efficient forward computation of the policy network during runtime, exhibiting strong real-time decision-making capabilities and low computational resource consumption. It is suitable for operation on edge devices with limited computing power, meeting the stringent real-time requirements of industrial applications.

[0049] Moreover, since the actions performed are based on stable strategies optimized through extensive simulations and expert data, the predictability, reliability, and security of the operation process are greatly improved compared to traditional programming or online learning, avoiding the uncertainty and potential risks caused by exploration during the execution process.

[0050] In a preferred embodiment of the present invention, the robot experience replay pool and the expert experience pool are specifically as follows: The experience data stored in the robot's experience replay pool is in the form of a quadruple, including the current state, the action performed, the immediate reward, and the next state; The data stored in the expert experience pool is in the form of state-action pairs, including the current state and the action to be performed.

[0051] The experience data stored in the robot experience replay pool is in the form of a quadruple, specifically containing four basic elements: current state s (representing instantaneous information about the robot and the environment), action a (the specific operation taken by the robot according to the policy), immediate reward r (the immediate evaluation feedback of the environment on the state-action pair), and next state s. next (The new state obtained after the action is performed). This quadruple (s, a, r, s) next It fully describes a complete decision-making process unit.

[0052] Correspondingly, the data stored in the expert experience pool takes the form of state-action pairs, specifically containing two core elements: the current state s (the environmental state during the expert's demonstration) and the action a (the optimal or teaching action demonstrated by the expert in this state). This state-action pair (s, a) centrally embodies the expert's decision-making knowledge in a specific state.

[0053] By designing differentiated data structures for two different data sources, a clear definition of data source and purpose is achieved. The four-tuple structure of the robot experience replay pool fully preserves the time-series information and reward signals necessary for policy evaluation and iterative optimization in reinforcement learning, forming the basis for calculating the temporal difference error objective. Meanwhile, the two-tuple structure of the expert experience pool focuses on extracting the core knowledge of expert decision-making (the mapping from state to action), providing the most direct and pure learning samples for imitation learning.

[0054] As a preferred embodiment of the present invention, the mixed batch data is constructed as follows: Hybrid batch data is constructed by sampling from the robot experience replay pool and the expert experience pool. The proportion of data sampled from the expert experience pool decreases dynamically as the training cycle increases.

[0055] The core objective of this implementation is to achieve an adaptive ratio of imitation learning and reinforcement learning data sources in a single training batch, thereby dynamically balancing the relationship between expert knowledge guidance and autonomous environment exploration throughout the entire training process.

[0056] When constructing each training batch, the system randomly samples from two experience pools in parallel: a certain number (denoted as N) is drawn from the robot's experience replay pool. The empirical quadruple (s, a, r, s) next At the same time, a certain number (denoted as N) are drawn from the expert experience pool. The state-action pairs (s, a) are formed, and the final mixed batch size is N. +N .

[0057] The proportion of expert data in the batch λ (i.e., λ=N) / (N +N The value is not fixed, but rather decreases dynamically as the training cycle (or number of iterations) increases.

[0058] In the initial training phase, a relatively high initial ratio λ is set. initial (e.g., 0.5 or higher), which makes policy updates in the early stages of training heavily reliant on high-quality expert demonstrations, enabling the rapid initialization of a high-performance, safe, and reliable policy, effectively avoiding the inefficiency and high risk caused by random exploration in the initial stage of pure reinforcement learning.

[0059] As the number of training epochs E increases, this proportion λ gradually decreases according to a predetermined rule, for example, using a linear decay strategy: λ = max(λ min ,λinitial -α·E), where α is the attenuation coefficient, and λ is the attenuation coefficient. min A minimum set ratio (which can be 0) is used to ensure that the strategy does not completely forget expert knowledge in the later stages of training. Ultimately, the learning focus smoothly transitions from imitating experts to relying on the robot's own experience of interacting with the environment for reinforcement learning optimization.

[0060] This dynamic sampling ratio mechanism allows the agent to focus on imitation in the early stages of training, and then gradually engage in autonomous learning. This greatly optimizes the overall learning path and improves training efficiency, thereby resolving the potential conflict between imitation learning and reinforcement learning in terms of objectives. The early high proportion of expert data utilization ensures rapid policy improvement and security, while the subsequent decay avoids the agent overfitting to expert behavior, creating conditions for it to break through the expert performance ceiling and discover better strategies. This achieves a seamless connection and complementary advantages between the two learning paradigms.

[0061] As a preferred embodiment of the present invention, the collaborative updating of the action policy network includes: Based on data sampled from the robot's experience replay pool, a first gradient is obtained to indicate the update direction of the action policy network; Based on the data sampled from the expert experience pool, a mimicry loss is obtained, and a second gradient is obtained based on the mimicry loss; The first gradient and the second gradient are fused together to update the parameters of the action policy network.

[0062] The core objective of this step is to enable the parameter updates of the action policy network to be guided by both reinforcement learning objectives and imitation learning objectives through a structured gradient fusion mechanism, thereby achieving deep synergy between the two learning paradigms at the parameter optimization level.

[0063] First, a first gradient is obtained based on data sampled from the robot's experience replay pool. This gradient is calculated to update the policy in the direction of maximizing long-term cumulative reward. Specifically, this gradient can be calculated as: , in, This indicates the first gradient that needs to be calculated; Represents the expected value; It is a state-value network (Parameters are) The output of the action The gradient; It is an action policy network (Parameters are) ) for its own parameters The gradient; This represents the state sampled from the robot's experience replay pool. This gradient indicates the direction of policy improvement from only the perspective of long-term rewards.

[0064] Secondly, the imitation loss is derived based on data sampled from the expert experience pool, and then the second gradient is obtained. This process aims to make the policy mimic expert behavior. Imitation loss function: , in, This represents behavioral cloning loss, also known as imitation loss. This indicates the number of human expert data points in the current batch; Action Policy Network , represents the current strategy of the collaborative robot, and its parameters are (equivalent to) ); and These represent the states sampled from the expert experience pool and their corresponding expert actions, respectively. The loss function measures the difference between the current policy output action and the expert action. The second gradient is the imitation loss. Policy network parameters gradient: .

[0065] Finally, the first gradient and the second gradient are fused together to update the parameters of the action policy network.

[0066] This collaborative update mechanism achieves a deep integration of imitation learning and reinforcement learning at the core parameter update level of the algorithm, rather than a simple external switching or data mixing, ensuring that the optimization signals provided by the two paradigms can guide the policy evolution in a timely and joint manner.

[0067] Through gradient fusion, the policy network can simultaneously perform reinforcement learning and imitation learning in the same training iteration, achieving a balance between exploration and exploitation, and between short-term efficiency and long-term optimality. It should be noted that by adjusting the weight coefficients... Alternatively, by controlling the proportion of expert data in mixed batches, the degree of dependence on expert knowledge during the learning process can be finely adjusted to adapt to different task requirements and expert data quality, thereby fundamentally improving the robustness and generalization ability of the method.

[0068] As one embodiment of this implementation, when the imitation loss is obtained, a selective imitation strategy is executed: For an expert state-action pair sampled from the expert experience pool, the difference between the output action of the current action policy network in the expert state and the expert action is obtained; When the estimated value of the state-action value function corresponding to the expert action is higher than the estimated value of the state-action value function corresponding to the output action, the difference is included in the imitation loss.

[0069] The core objective of this embodiment is to introduce a value judgment mechanism during imitation learning, thereby intelligently filtering high-quality expert demonstration data and avoiding the negative impact of imperfect or suboptimal expert behavior on policy learning.

[0070] For each expert state-action pair (s) sampled from the expert experience pool j ,a j First, the difference between the output action of the current action policy network in the expert state and the expert action is calculated. This difference is a fundamental element of imitation learning, and its calculation formula is: ||μ|| p (s j )-a j ‖, where μ p (s j ) represents the action policy network with parameter p in expert state s. j The action to output.

[0071] Secondly, key value judgment and selective inclusion steps are performed. The estimated state-action value function corresponding to the expert's action, i.e., Q, is calculated. v (s j ,a j ), where Q v This represents a state-value network with parameter v. Simultaneously, it calculates the estimated state-action value function (Q) corresponding to the current policy's output action. v (s j ,μ p (s j )).

[0072] Then compare and judge: if and only if Q v (s j ,a j Q v (s j ,μ p (s j Only when the expected long-term value of the expert action is higher than the expected long-term value of the current strategy action will the system obtain the action difference ||μ. p (s j )-a j || Included in the final imitation loss middle.

[0073] This selective mechanism addresses the performance degradation that can result from inconsistent expert data quality. It ensures that the policy network only learns and imitates expert behaviors validated as superior by the value function, automatically filtering out suboptimal or erroneous expert demonstrations, thus significantly improving the robustness and reliability of imitation learning.

[0074] From a computational efficiency perspective, this strategy only performs loss calculations when the value judgment is valid, avoiding computational overhead for invalid samples. At the same time, guided by the value network, the imitation learning process becomes more accurate and efficient, accelerating the convergence of the strategy and improving its performance.

[0075] As another embodiment of this implementation, the first gradient and the second gradient are combined, specifically as follows: The first gradient and the second gradient are fused by weighted summation. The weight coefficient of the second gradient decreases dynamically as the training period increases.

[0076] The core objective of this embodiment is to dynamically adjust the relative weights of imitation learning and reinforcement learning on policy updates during training through an adaptive gradient fusion strategy, thereby achieving a smooth transition from imitation-guided learning to autonomous optimization.

[0077] The fusion is achieved through a weighted sum. Total gradient The calculation formula is as follows: , in, Indicates the network parameters used for the final update of the action policy. The total gradient; It is the first gradient calculated based on the robot's own interaction experience; The second gradient, or imitation loss, is calculated based on expert data. Policy network parameters The gradient; This refers to the weight coefficients of the second gradient (imitative learning gradient).

[0078] Weighting coefficient It is not a fixed value, but rather varies with the training cycle (or number of training steps). The decay occurs dynamically as the amount of decay increases. For example, a linear decay strategy can be used: ,in These are the initial weights. It is the preset minimum weight (which can be zero). It is the attenuation rate. This is the current training cycle number. Alternatively, exponential decay or other similar methods can be used.

[0079] This dynamic decay strategy constructs an inherent learning mechanism. In the early stages of training, a larger [set value] is used. The value allows the imitation learning gradient to dominate the update process, quickly guiding the policy to a high-performance region near the expert policy, greatly improving sample efficiency and stability in the early stages of training.

[0080] As training progresses, gradually reduce This strengthens the influence of the learning gradient, encouraging agents to explore and optimize more deeply based on environmental feedback, thus potentially discovering and learning better strategies than those demonstrated by experts.

[0081] This mechanism effectively reconciles the inherent conflicts that may exist between imitation learning and reinforcement learning in terms of objectives. By allocating the dominance of the two in the time dimension, it avoids the phenomenon that strategies may waver between imitation and exploration under fixed weights, guiding the learning process to converge in an orderly and stable manner. By designing different decay curves, it can flexibly adapt to the different requirements of different tasks on the degree of dependence on prior knowledge, enhancing the versatility and robustness of the method.

[0082] As a preferred embodiment of the present invention, the collaborative updating of the state value network specifically involves: The target temporal difference error is obtained by calculating the target network. The state value network is updated using gradient descent based on the temporal difference error target.

[0083] The core objective of this implementation is to calculate a stable temporal difference error target by introducing a target network mechanism, thereby achieving more accurate and stable training of the state value network and providing a reliable value estimation basis for policy evaluation.

[0084] First, the target time-series difference error is calculated using the target network. The specific form of this target value is as follows: , in, The target value for time-series difference error is the target value for updating the state-value network. Indicates the state Next action Immediate rewards obtained from the environment; It is a discount factor, with a value between 0 and 1, used to weigh the importance of current rewards against future rewards; This represents the target state value network's response to the next state. The target policy network outputs the following actions. The long-term value estimate. It should be noted that here... and Specifically, the target network has its parameters kept asynchronous with the main network through a soft update strategy, thereby providing a stable learning target.

[0085] Then, based on the temporal difference error target, the state-value network is updated using gradient descent. Specifically, the loss function of the state-value network is calculated: , in, The loss function representing a state-value network; Indicating in strategy The following expectations; This represents the current state value network (its parameters are...). State-action pairs The value prediction. This loss function measures the difference between the network's predicted value and the target value. The mean squared error between the state-value network parameters is calculated using this loss function. gradient And use gradient descent algorithms (such as the Adam optimizer) to update the parameters. This allows the predictions of the state value network to gradually approach the more stable target value calculated by the target network.

[0086] By introducing a target network to compute the temporal difference objective, the correlation caused by the bootstrapping method in value estimation is effectively broken, the instability and divergence problems in the value function training process are significantly alleviated, and a more reliable value benchmark is provided for the policy gradient.

[0087] Moreover, this mechanism achieves stable convergence of deep reinforcement learning algorithms. By decoupling the update of the target value from the update of the current value network, it significantly reduces the variance of value estimation, thereby accelerating the convergence process of training and improving the performance of the finally learned policy. This design enables the algorithm to maintain robust learning capabilities even when facing complex, high-dimensional state spaces.

[0088] Specifically, the target network also includes: After the action policy network and the state value network are updated, the target network is softly updated. The soft update parameter ranges from 0.001 to 0.01.

[0089] The core objective of this embodiment is to ensure that the target network parameters can smoothly and stably track the evolution of the main network through a progressive parameter update strategy, thereby providing a continuous and reliable value estimation target for temporal difference learning.

[0090] The soft update operation is performed after the action policy network and the state value network have completed parameter updates. Its update rule follows the formula below: , in, and These represent the parameters of the target state value network and the target action policy network, respectively. and These represent the parameters of the currently updated master state value network and master action policy network, respectively. This is the soft update coefficient, and its value is strictly limited to between 0.001 and 0.01.

[0091] The key to this mechanism lies in the soft update coefficient. The parameters of the target network take tiny values. With each update, the parameters change only by a very small amount ( Some) moved towards the new parameters of the main network, while the vast majority ( Some of them still retain their original parameters.

[0092] This soft-update coefficient strategy greatly enhances the stability of the training process. Because the target value... The computation relies on target network parameters that change slowly, effectively avoiding drastic fluctuations in the value estimation target, thus preventing the main value network and even the entire policy learning process from diverging due to target instability. Furthermore, ensuring successful algorithm convergence, by achieving a delicate balance between maintaining relative target stability and tracking the learning progress of the main network, provides a reliable learning foundation for deep reinforcement learning based on function approximation.

[0093] limited Within a small range of 0.001 to 0.01, the gradual and smooth updating of the target network parameters is ensured. This not only reflects the learning results of the main network in a timely manner, but also avoids introducing noise due to excessively rapid changes. This is a key hyperparameter setting to ensure the stable and efficient execution of complex skill learning tasks.

[0094] The present invention also provides a robot skill learning system for implementing any one of the methods described above, the system comprising: One or more processors; Memory; And one or more computer programs, which are stored in the memory and configured to be executed by the one or more processors, the one or more computer programs including instructions for performing any of the methods.

[0095] Therefore, it is possible to achieve any effect in robot skill learning methods based on fusion algorithms, which will not be elaborated here.

[0096] The present invention further provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements any one of the methods described herein.

[0097] Therefore, it is possible to achieve any effect in robot skill learning methods based on fusion algorithms, which will not be elaborated here.

[0098] For any parts not mentioned in this invention, existing technologies can be used or referenced.

[0099] The various embodiments in this specification are described in a progressive manner. The same or similar parts between the various embodiments can be referred to each other. Each embodiment focuses on describing the differences from other embodiments.

[0100] The above description is merely an embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principle of the present invention should be included within the scope of the claims of the present invention.

Claims

1. A robot skill learning method based on a fusion algorithm, characterized in that, include: Pre-build the robot experience playback pool and expert experience pool, and initialize the action policy network and state value network; The method also include, The robot stores experiential data in the robot's experiential replay pool by interacting with the environment, and updates the action policy network and the state value network collaboratively based on mixed batch data sampled from the robot's experiential replay pool and the expert experience pool. The trained action policy network enables the robot to output and execute actions based on the perceived environmental state.

2. The method according to claim 1, characterized in that, Collaboratively updating the action policy network includes: Based on data sampled from the robot's experience replay pool, a first gradient is obtained to indicate the update direction of the action policy network; Based on the data sampled from the expert experience pool, a mimicry loss is obtained, and a second gradient is obtained based on the mimicry loss; The first gradient and the second gradient are fused together to update the parameters of the action policy network.

3. The method according to claim 2, characterized in that, When imitation loss is obtained, a selective imitation strategy is executed: For an expert state-action pair sampled from the expert experience pool, the difference between the output action of the current action policy network in the expert state and the expert action is obtained; When the estimated value of the state-action value function corresponding to the expert action is higher than the estimated value of the state-action value function corresponding to the output action, the difference is included in the imitation loss.

4. The method according to claim 2, characterized in that, The first gradient and the second gradient are fused together, specifically as follows: The first gradient and the second gradient are fused by weighted summation. The weight coefficient of the second gradient decreases dynamically as the training period increases.

5. The method according to claim 1, characterized in that, Constructing mixed batch data, specifically: Hybrid batch data is constructed by sampling from the robot experience replay pool and the expert experience pool. The proportion of data sampled from the expert experience pool decreases dynamically as the training cycle increases.

6. The method according to claim 1, characterized in that, The state value network is updated collaboratively, specifically as follows: The target temporal difference error is obtained by calculating the target network. The state value network is updated using gradient descent based on the temporal difference error target.

7. The method according to claim 6, characterized in that, The target network also includes: After the action policy network and the state value network are updated, the target network is softly updated. The soft update parameter ranges from 0.001 to 0.

01.

8. The method according to claim 1, characterized in that, The robot experience replay pool and the expert experience pool are specifically as follows: The experience data stored in the robot's experience replay pool is in the form of a quadruple, including the current state, the action performed, the immediate reward, and the next state; The data stored in the expert experience pool is in the form of state-action pairs, including the current state and the action to be performed.

9. A robot skill learning system, characterized in that, The system for implementing the method as described in any one of claims 1 to 8 includes: One or more processors; Memory; And one or more computer programs, said one or more computer programs being stored in said memory and configured to be executed by said one or more processors, said one or more computer programs including instructions for performing the method as claimed in any one of claims 1 to 8.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1 to 8.

Images