Robot skill learning method, apparatus, device, and storage medium

Abstract

The application is suitable for the technical field of robot motion control, and provides a robot skill learning method, which comprises the following steps: generating an auxiliary reward component function by using a large language model according to learning prompt information; constructing a multi-branch value network according to a reward component function set composed of a plurality of reward component functions in the learning prompt information and the auxiliary reward component function; performing iterative policy training on a policy network by using the multi-branch value network based on weight coefficients of each value branch until a target function of the policy network converges; and guiding robot skill learning by using the policy network, so that the intelligent decision-making ability of the large language model and the fine-grained feedback of the multi-branch value network are combined, the policy learning efficiency and overall performance in reinforcement learning are improved, different skill acquisition requirements in the policy optimization process can be adapted to, different environments and complex robot tasks can be coped with, and the accuracy, precision and efficiency of task execution are improved.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of robot motion control, and particularly relates to a robot skill learning method, device, equipment and storage medium. BACKGROUND

[0002] A body-aware robot collects interaction data through interaction with the environment, aiming to learn the skills expected by humans to perform specific actions or tasks. However, with the increasing freedom of robots and the increasing complexity of dynamic environments, the implementation of this task becomes more difficult. This complexity requires the imposition of additional constraints as optimization targets to ensure that the robot can efficiently and accurately complete the task. Reinforcement Learning (RL) is an effective policy optimization method that optimizes the policy by maximizing cumulative rewards, effectively converting each robot constraint into a reward component. This approach has been successful in various robot tasks, such as leg motion control, dexterous hand operation, and overall robot manipulation. However, current RL methods optimize the policy by adding all reward components, which means that the robot needs to learn multiple optimization targets simultaneously in parallel, which is difficult for the robot. This approach makes learning inefficient.

[0003] Van et al. proposed a Hybrid Reward Architecture (HRA) that is applied in reinforcement learning to improve sample efficiency and learning effectiveness by designing fine-grained reward functions. Specifically, HRA decomposes complex reward functions into multiple simple sub-reward components, each corresponding to a different task goal or event, thereby guiding the robot to learn more effectively. This approach has been proven to significantly improve learning efficiency, as it allows the robot to focus more on the specific learning needs of each sub-reward component. Further, Huang et al. introduced the Hybrid Dynamic Policy Gradient (HDPG) method to optimize this process, which dynamically adjusts the contribution (i.e., weight) of each reward branch in the policy optimization process, achieving intelligent guidance of the robot learning process. HDPG technology enables the robot to prioritize learning components that can quickly accumulate rewards, allowing the robot to focus on "simpler" components before dealing with more "challenging" components. However, it is worth noting that the effectiveness of the HDPG method largely depends on the dynamic weight calculation rules designed by human experts, which may not always be applicable or guarantee optimal performance in different robot tasks. Therefore, the widespread application and performance optimization of the HDPG method still face challenges. SUMMARY

[0004] The present application aims to provide a robot skill learning method, device, equipment and storage medium, aiming at solving the problem of poor learning effect of robot skill learning for complex tasks and environment caused by the prior art.

[0005] Poor learning effect for complex tasks and environment

[0006] In one aspect, the present application provides a robot skill learning method, comprising the following steps:

[0007] Based on the preset learning prompt information, an auxiliary reward component function is generated using a large language model, and a reward component function set is formed by a plurality of reward component functions in the learning prompt information and the auxiliary reward component function;

[0008] A multi-branch value network is constructed according to the reward component function set, wherein each value branch in the multi-branch value network corresponds to each reward component function in the reward component function set;

[0009] Based on the weight coefficients of each value branch, the multi-branch value network is used for iterative policy training of a policy network until the objective function of the policy network converges;

[0010] The policy network obtained at the end of training is used to guide robot skill learning.

[0011] Preferably, the step of using the multi-branch value network to perform iterative policy training of a policy network based on the weight coefficients of each value branch comprises:

[0012] During the policy training process, the weight coefficients of each value branch are updated according to a preset weight update strategy.

[0013] Preferably, the step of updating the weight coefficients of each value branch according to a preset weight update strategy comprises:

[0014] During the policy training process, when a preset weight update frequency is reached, a weight adjustment rule most suitable for the current policy is selected from a pre-constructed rule library using the large language model based on the current policy performance and the historical policy performance, wherein the current policy is a policy adopted by the policy network after the current training round for selecting the optimal action;

[0015] Based on the current policy performance, the weight adjustment rule is used to update the weight coefficients of each value branch.

[0016] Preferably, before the step of using the large language model to select the weight adjustment rule most suitable for the current policy from the pre-constructed rule library, the method further comprises:

[0017] According to the learning prompt information and a preset rule example, the rule library for updating weights is constructed by using the large language model.

[0018] Preferably, based on the weight coefficients of each value branch, the step of iteratively training the policy network by using the multi-branch value network further comprises:

[0019] After each round of policy training, the advantage value of each value branch under the corresponding reward component function is weighted and fused according to the weight coefficient of each value branch corresponding to the current value branch, to obtain a total advantage value under the overall target corresponding to the set of reward component functions;

[0020] According to the target function, the network parameters of the policy network are learned and updated by using the total advantage value, so as to maximize the target function.

[0021] On the other hand, the present application provides a robot skill learning device, which comprises:

[0022] A component set construction unit is configured to generate an auxiliary reward component function by using a large language model based on preset learning prompt information, and to form a reward component function set by using a plurality of reward component functions in the learning prompt information and the auxiliary reward component function.

[0023] A value network construction unit is configured to construct a multi-branch value network according to the reward component function set, wherein each value branch in the multi-branch value network corresponds to each reward component function in the reward component function set.

[0024] A policy network training unit is configured to iteratively train a policy network by using the multi-branch value network based on the weight coefficients of each value branch, until a target function of the policy network converges.

[0025] A skill learning guidance unit is configured to guide robot skill learning by using the policy network obtained after training.

[0026] Preferably, the policy network training unit comprises:

[0027] A weight coefficient updating unit is configured to update the weight coefficients of each value branch according to a preset weight updating strategy during the policy training process.

[0028] Preferably, the weight coefficient updating unit comprises:

[0029] a rule selection unit configured to, during the policy training process, select a weight adjustment rule most suitable for a current policy from a pre-constructed rule library based on a current policy performance and a historical policy performance when a preset weight update frequency is reached, wherein the current policy is a policy adopted by the policy network after a current training round for selecting an optimal action;

[0030] a weight updating sub-unit configured to update weight coefficients of each of the value branches based on the current policy performance and the weight adjustment rule.

[0031] In another aspect, the present application also provides an agent device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the robot skill learning method as described above when executing the computer program.

[0032] In another aspect, the present application also provides a computer readable storage medium, which stores a computer program, wherein the computer program is executed by a processor to implement the steps of the robot skill learning method as described above.

[0033] According to the learning prompt information, the large language model is used to generate an auxiliary reward component function, a multi-branch value network is constructed according to a reward component function set composed of a plurality of reward component functions in the learning prompt information and the auxiliary reward component function, the policy network is iteratively trained based on weight coefficients of each value branch in the multi-branch value network, and the policy network is trained until a target function of the policy network converges. The trained policy network is used to guide the robot skill learning, thereby improving the policy learning efficiency and overall performance in reinforcement learning by combining the intelligent decision-making ability of the large language model and the fine-grained feedback of the multi-branch value network, ensuring that the policy optimization process can adapt to different skill acquisition requirements and cope with different environments and complex robot tasks, and improving the accuracy, precision and efficiency of robot task execution. BRIEF DESCRIPTION OF DRAWINGS

[0034] Figure 1 is an implementation flowchart of the robot skill learning method provided by the first embodiment of the present application;

[0035] Figure 2 is a structural schematic diagram of the robot skill learning device provided by the second embodiment of the present application;

[0036] Figure 3 is a preferred structural schematic diagram of the robot skill learning device provided by the second embodiment of the present application;

[0037] Figure 4is a structural schematic diagram of the intelligent agent device provided in Embodiment Three of the present application. DETAILED DESCRIPTION

[0038] In order to make the objectives, technical solutions and advantages of the present application clearer and more comprehensible, the present application is further described in detail below in combination with the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and do not limit the present application.

[0039] The specific implementation of the present application is described in detail below in combination with specific embodiments:

[0040] Embodiment One

[0041] Figure 1 The implementation flow of the robot skill learning method provided in Embodiment One of the present application is shown. For ease of illustration, only the parts related to the embodiments of the present application are shown, and the details are described as follows:

[0042] In step S101, based on the preset learning prompt information, an auxiliary reward component function is generated by using a large language model, and the reward component function set is composed of a plurality of reward component functions and the auxiliary reward component function in the learning prompt information.

[0043] Embodiments of the present application are applicable to intelligent agent devices capable of perceiving the environment, making decisions and taking actions, such as robots, drones, etc. In embodiments of the present application, for complex tasks that may involve multiple goals, constraints and conditions, a single, global reward function may not be able to provide sufficiently accurate and detailed feedback for the actions taken by the robot in different states. Here, the original reward function is first decomposed into multiple reward component functions, each of which focuses on a specific aspect of the task (such as goal achievement, state transition, path efficiency, safety or resource utilization, etc.), so that the robot can better understand the task structure and provide clear reward or punishment signals for different behaviors taken by the robot in each specific aspect, which helps to guide the robot to take the correct action in a specific situation and avoid unnecessary exploration or wrong behavior, so that the robot learns more accurate and efficient strategies, improves the performance of the robot in complex tasks, and then task information, environment information and each reward component function are written in the form of text in the learning prompt information, and the learning prompt information is input into a large language model (Large Language Model, LLM), and through the understanding of the large language model to the training task and the optimization target, one or more additional reward component functions for assisting strategy training are generated. These reward component functions aim to enhance the reward signal in the strategy training process, which may target aspects not fully covered in the current reward mechanism, or provide additional incentives for specific behaviors. In order to distinguish the reward component functions obtained by decomposing the original reward function, the reward component functions generated by the large language model are referred to as auxiliary reward component functions. Finally, each reward component function obtained by decomposing the original reward function and the auxiliary reward component function constitute a reward component function set, and the elements in the reward component function set form the final reward function through a specific combination method (such as weighted sum). In addition, the task information and the environment information describe the task that the robot needs to complete and the environment in which the robot executes the task, respectively. The task information includes but is not limited to target tasks, desired behaviors, and the environment information includes but is not limited to state space, action space, possible interaction modes, and dynamic characteristics of the environment. The generation of auxiliary reward component functions can be represented as C a =LLM r (T t ,T e ,C r ), the reward component function set is represented as C r ,C a}={r1,r2,...,r K}, wherein T t is task information, T e is environment information, C r is each reward component function obtained by decomposing the original reward function, and C aTo assist the reward component function, r K For the Kth reward component function in the reward component function set C, K is the total number of reward component functions in the reward component function set C.

[0044] In step S102, a multi-branch value network is constructed according to the reward component function set, wherein each value branch in the multi-branch value network corresponds to each reward component function in the reward component function set.

[0045] In the embodiment of the present application, according to the reward component function set, a multi-layer perceptron (MLP) architecture in a deep neural network is used to construct a multi-branch value network, each value branch in the multi-branch value network corresponds to each reward component function in the reward component function set, and each value branch focuses on evaluating the value of the state or state feature related to the reward component function corresponding to it, which is usually extracted from the original state space to describe the specific situation or position of the robot in the environment. Specifically, in the multi-branch value network, each value branch uses the current state or state feature related to the reward component function corresponding to it as input, and processes the respective input state in parallel, calculates the corresponding value, then calculates the advantage value under the corresponding reward component function based on the value, and outputs, the advantage value output by each value branch will be used to guide the policy update, so that the policy network makes better decisions. Since the multi-branch value network considers multiple reward component functions and corresponding branch states, it can capture different situations or positions of the robot in the environment more carefully, which helps to improve the adaptability and robustness of the agent, and makes it consider more factors in the decision-making process, so as to make more accurate judgments.

[0046] In step S103, based on the weight coefficients of each value branch, the multi-branch value network is used to perform iterative policy training on the policy network until the objective function of the policy network converges.

[0047] In the embodiments of the present application, a weight coefficient is assigned to each value branch, which is used to balance the influence of different value branches on policy updating. Based on the weight coefficients of the value branches, the multi-branch value network is used to iteratively train the policy network until the objective function of the policy network converges. During the iterative training of the policy network using the multi-branch value network, in each iteration, the policy network selects an action according to the current policy, and the output of the multi-branch value network is used to evaluate the pros and cons of different actions under the current policy and guide the update of the policy network. Through iterative training, the policy network gradually learns to select actions that can maximize long-term returns. When the objective function value of the policy network no longer changes significantly in consecutive iterations, it is considered that the policy network has converged to a stable state.

[0048] In a feasible embodiment, during the policy training process, the weight coefficients of the value branches are updated according to a preset weight update policy.

[0049] In the embodiments of the present application, during the policy training process, the weight coefficients of the value branches are updated in stages according to a preset weight update policy, thereby avoiding unstable training caused by frequent adjustment of the weight coefficients, and guiding the policy network to pay more attention to certain reward components by adjusting the weight coefficients in different training stages, so as to achieve fine control of the overall behavior or performance.

[0050] In another feasible embodiment, the updating of the weight coefficients of the value branches is achieved by the following steps:

[0051] (1) During the policy training process, when a preset weight update frequency is reached, a large language model is used to select a weight adjustment rule most suitable for the current policy from a pre-constructed rule library based on the current policy performance and historical policy performance, wherein the current policy is the policy adopted by the policy network after the current training round to select the optimal action;

[0052] In this embodiment of the invention, the weight update frequency is used to periodically adjust the weight coefficients during training to optimize the performance of the policy network. It can be set according to the training strategy and requirements (e.g., based on the number of iterations of policy training (updating once every 100 rounds)). The current policy performance consists of the average expected return of each reward component function under the current policy, and the historical policy performance consists of the average expected return of each reward component function over a historical period. During policy training, the policy network is updated in each round based on the current weight coefficients and training data. After the current training round, it is determined whether the preset weight update frequency has been reached. Otherwise, the weight coefficients are not updated, and the next round of policy training is directly performed. If so, based on the current policy performance and historical policy performance, a large language model is used to select the most suitable weight adjustment rule from a pre-built rule base for the current policy. Here, the selection process of the weight adjustment rule is represented as follows: in, Based on the current strategy performance, For historical strategy performance, L represents the historical length, l represents the current training round, and B n For the rule base, B selected For the selected weight adjustment rules, Let be the average expected return of the Kth reward component function under the current policy corresponding to the current training round l.

[0053] (2) Based on the current strategy performance, update the weight coefficients of each value branch using the weight adjustment rules.

[0054] In this embodiment of the invention, after determining the current weight adjustment rule, the weight coefficients of each value branch will be updated based on the rule. Specifically, Among them, S σ =[σ 1 ,σ 2 ,...,σ K Average expected return for each value branch The variance, i.e. σ K for The corresponding variance, w K The weight coefficients are the weights of the value branch corresponding to the Kth reward component function. The next round of policy training updates the policy network based on the updated weight coefficients.

[0055] The steps (1) and (2) above realize more flexible adjustment and switching of weight adjustment rules in the training process by combining the intelligent decision-making ability of the large language model and the fine-grained feedback of the multi-branch value network, thereby realizing more reasonable configuration of dynamic weights, and further realizing more effective dynamic adjustment of policy optimization in reinforcement learning, so that the robot can effectively handle the complexity and uncertainty in different task scenarios in the process of skill learning, and improve the policy learning efficiency and overall performance in reinforcement learning.

[0056] In a feasible embodiment, in the initial stage of policy training, the update of the weight coefficient has not been triggered, and the large language model has not received the feedback information of the policy performance (i.e., the current policy performance and the historical policy performance) from the multi-value network. In this stage, the preset weight coefficient is used to train the policy network, and the preset weight coefficient can be set to 1. As an example, a task is trained for 1500 rounds, and the policy network is updated every round, but only every 100 rounds, the rule is reselected to determine the new weight coefficient. Before the next 100 rounds, the weight coefficient determined in the last 100 rounds is used to update the policy network, i.e., the weight coefficient (e.g., 1) initially set is used to update the policy network in the first 100 rounds, the weight coefficient determined after the 100th round is used to update the policy network in the 101st-200th round, the weight coefficient determined after the 200th round is used to update the policy network in the 201st-300th round, and so on, until the training is completed.

[0057] In another feasible embodiment, before the policy network is iteratively trained by the multi-branch value network based on the weight coefficients of the value branches, a rule library for updating the weights is constructed by using the large language model according to the learning prompt information and the preset rule examples. Specifically, the learning prompt information and the preset rule examples are input into the large language model, and a plurality of rules for updating the weights are generated by using the powerful reasoning and generation capabilities of the large language model. The generated rules are constructed into a rule library, wherein the rule examples are some exemplary rules or patterns for guiding the large language model to generate high-quality rules. In this embodiment, the rules proposed in the HDPG method are used as the rule examples, and are denoted as E hdpg to improve the rationality of the rules generated by the large language model. The construction of the rule library is denoted as B n = LLM g (T t , T e , C r , E hdpg), so as to dynamically generate a plurality of different effective rules through the large language model, realize that the weight coefficient is adjusted in a more fine-grained manner, and dynamically adjust the learning priority of each value branch in the subsequent training process according to more rules, so as to ensure that the skill learning demand in the strategy optimization process can be adapted to different environments and complex robot tasks.

[0058] In another feasible embodiment, the strategy training of the strategy network is realized by the following steps:

[0059] ① After each round of strategy training, the advantage value of each value branch under the corresponding reward component function is weighted and fused according to the weight coefficient corresponding to the current value branch, to obtain the total advantage value under the overall goal corresponding to the reward component function set;

[0060] In the embodiment of the application, after each round of strategy training, the advantage value output by each value branch under the corresponding reward component function is calculated, each advantage value is multiplied with the weight coefficient vector w=[w1, w2,..., w K ] composed of the weight coefficient, and the sum of the multiplied results is summed to obtain the total advantage value under the overall goal corresponding to the reward component function set, that is , wherein A k is the advantage value output by the kth value branch, w k is the weight coefficient corresponding to the kth value branch, and is the total advantage value.

[0061] ② According to the objective function, the network parameters of the strategy network are learned and updated using the total advantage value, so as to maximize the objective function.

[0062] In the embodiment of the application, according to the objective function , the network parameters of the strategy network are learned and updated using the total advantage value, so as to maximize the objective function, specifically, in order to find the network parameters θ that maximize the objective function J(π θ ) (i.e. the expected return), here, the parameters are updated in the direction of the policy log probability gradient , and the update amplitude is determined by the total advantage value , if the total advantage value of a certain action is high, then the probability of selecting this action is increased, otherwise, if the total advantage value is low, the probability of selecting this action is reduced, wherein

[0063] represents the gradient of the objective function J(π θ ) of the strategy π θ (parametrized by the network parameters θ) with respect to θ;

[0064] represents the expectation of state-action pair (s t ,a t ) under policy π θ , which is calculated according to the transition probability of the environment and the policy itself;

[0065] represents the gradient of the logarithmic probability of action a θ selected by policy π t under given state s t with respect to θ, which indicates how to adjust θ to increase the probability of selecting the current action a t if this action is considered to be good, i.e., has a high advantage value.

[0066] The above steps ① and ② directly optimize the policy parameters in a complex environment and reward structure to perform policy training on the policy network and adapt to continuous and high-dimensional action space, thereby improving the training speed and training effect.

[0067] In step S104, the policy network obtained after training is used to guide the robot skill learning.

[0068] In the embodiment of the present application, the policy network obtained after training can accurately map the environment state to the corresponding robot control action, i.e., it contains the knowledge of how the robot selects the optimal action according to the environment state. In this way, the policy network obtained after training is loaded into the robot system, and in the running process of the robot, the policy network will output an action according to the current environment state, the robot will execute this action, and the change of the environment state will be observed. After the robot executes the action, the environment will give a feedback (usually a reward or a punishment), according to which the robot will update its own state and prepare for the next decision. The robot will constantly repeat the process of executing the action output by the policy network and updating the state, and through constant interaction and learning with the environment, gradually master new skills.

[0069] In the embodiment of the present application, according to the learning prompt information, the auxiliary reward component function is generated by using the large language model, the reward component function set is constituted by the reward component functions and the auxiliary reward component function in the learning prompt information, the multi-branch value network is constructed according to the reward component function set, the policy network is iteratively trained by using the multi-branch value network based on the weight coefficients of each value branch in the multi-branch value network, until the objective function of the policy network converges, and the policy network obtained after the training is used to guide the robot skill learning, so as to improve the policy learning efficiency and overall performance in reinforcement learning by combining the intelligent decision-making ability of the large language model and the fine-grained feedback of the multi-branch value network, ensure that the policy optimization process can adapt to different skill acquisition requirements, cope with different environments and complex robot tasks, and improve the accuracy, precision and efficiency of robot task execution.

[0070] Embodiment two:

[0071] Figure 2 The structure of the robot skill learning device provided by the second embodiment of the present application is shown. For the sake of convenience, only the parts related to the embodiments of the present application are shown, which include:

[0072] The component set construction unit 21 is configured to generate an auxiliary reward component function by using a large language model based on preset learning prompt information, and the reward component function set is constituted by the reward component functions and the auxiliary reward component function in the learning prompt information.

[0073] The value network construction unit 22 is configured to construct a multi-branch value network according to the reward component function set, wherein each value branch in the multi-branch value network corresponds to each reward component function in the reward component function set.

[0074] The policy network training unit 23 is configured to iteratively train the policy network by using the multi-branch value network based on the weight coefficients of each value branch, until the objective function of the policy network converges.

[0075] The skill learning guidance unit 24 is configured to guide the robot skill learning by using the policy network obtained after the training.

[0076] As shown in Figure 3 Preferably, the policy network training unit 23 includes:

[0077] The weight coefficient updating unit 231 is configured to update the weight coefficients of each value branch according to the preset weight updating strategy during the policy training.

[0078] The weight coefficient updating unit 231 includes:

[0079] The rule selection unit 2311 is configured to, in the strategy training process, select a weight adjustment rule most suitable for the current strategy from a pre-constructed rule library based on the current strategy performance and the historical strategy performance when a preset weight update frequency is reached, wherein the current strategy is a strategy adopted by the strategy network after a current training round and used for selecting an optimal action.

[0080] The weight update sub-unit 2312 is configured to update the weight coefficients of the value branches based on the current strategy performance by using the weight adjustment rule.

[0081] Preferably, the strategy network training unit 23 further comprises:

[0082] The advantage value fusion unit is configured to, after each round of strategy training, weight and fuse the advantage values of each value branch under the corresponding reward component function according to the weight coefficients of the current value branches to obtain a total advantage value under the overall target corresponding to the set of reward component functions.

[0083] The network parameter update unit is configured to learn and update the network parameters of the strategy network by using the total advantage value according to the target function so as to maximize the target function.

[0084] Preferably, the robot skill learning device of the embodiment of the present application further comprises:

[0085] The rule library construction unit is configured to construct a rule library for updating weights by using a large language model according to the learning prompt information and a preset rule example.

[0086] In the embodiment of the present application, each unit of the robot skill learning device can be realized by a corresponding hardware or software unit, and each unit can be an independent software or hardware unit, or can be integrated into a software or hardware unit, which does not limit the present application. Specifically, the implementation manner of each unit can refer to the description of the foregoing embodiment one, which will not be described here.

[0087] Embodiment three:

[0088] Figure 4 The structure of the agent device provided by the embodiment three of the present application is shown, and only the parts related to the embodiment of the present application are shown for ease of illustration.

[0089] The agent device 4 of the embodiment of the present application comprises a processor 40, a memory 41, and a computer program 42 stored in the memory 41 and executable on the processor 40. The processor 40 implements the steps in the robot skill learning method embodiments when executing the computer program 42, for example Figure 1 The steps S101 to S104 shown. Alternatively, the processor 40 implements the functions of each unit in the foregoing device embodiments when executing the computer program 42, for exampleFigure 2 the functions of the units shown.

[0090] In the embodiment of the present application, according to the learning prompt information, the large language model is used to generate an auxiliary reward component function, a multi-branch value network is constructed according to a reward component function set composed of a plurality of reward component functions and the auxiliary reward component function in the learning prompt information, and based on the weight coefficients of each value branch in the multi-branch value network, the multi-branch value network is used to perform iterative policy training on the policy network until the objective function of the policy network converges. The policy network obtained after the training is used to guide the robot skill learning, so as to improve the policy learning efficiency and overall performance in reinforcement learning by combining the intelligent decision-making ability of the large language model and the fine-grained feedback of the multi-branch value network, ensure that the skill learning demand can be adapted to different skills in the process of policy optimization, cope with different environments and complex robot tasks, and improve the accuracy, precision and efficiency of robot task execution.

[0091] The agent device in the embodiment of the present application can be a robot. The steps implemented when the processor 40 in the agent device 4 executes the computer program 42 to implement the robot skill learning method can refer to the description of the foregoing method embodiments, which will not be described here.

[0092] Embodiment four:

[0093] In the embodiment of the present application, a computer readable storage medium is provided, which stores a computer program. When the computer program is executed by a processor, the steps in the above-mentioned robot skill learning method embodiments are implemented, for example, Figure 1 the steps S101 to S104 shown. Alternatively, when the computer program is executed by a processor, the functions of the units in the above-mentioned device embodiments are implemented, for example Figure 2 the functions of the units shown.

[0094] In the embodiment of the present application, according to the learning prompt information, the large language model is used to generate an auxiliary reward component function, a multi-branch value network is constructed according to a reward component function set composed of a plurality of reward component functions and the auxiliary reward component function in the learning prompt information, and based on the weight coefficients of each value branch in the multi-branch value network, the multi-branch value network is used to perform iterative policy training on the policy network until the objective function of the policy network converges. The policy network obtained after the training is used to guide the robot skill learning, so as to improve the policy learning efficiency and overall performance in reinforcement learning by combining the intelligent decision-making ability of the large language model and the fine-grained feedback of the multi-branch value network, ensure that the skill learning demand can be adapted to different skills in the process of policy optimization, cope with different environments and complex robot tasks, and improve the accuracy, precision and efficiency of robot task execution.

[0095] The computer readable storage medium of the embodiment of the present application can include any entity or device capable of carrying computer program codes, recording media, such as ROM / RAM, magnetic disk, optical disk, flash memory, etc.

[0096] The above merely describes the preferred embodiments of the present application and is not used to limit the present application, and any modification, equivalent replacement, and improvement made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A method of robot skill learning, characterized by, The method comprises the following steps: Based on the preset learning prompt information, an auxiliary reward component function is generated by using a large language model, and a reward component function set is constituted by a plurality of reward component functions in the learning prompt information and the auxiliary reward component function; According to the reward component function set, a multi-branch value network is constructed, wherein each value branch in the multi-branch value network corresponds to each reward component function in the reward component function set one by one; Based on the weight coefficients of each value branch, the multi-branch value network is used to perform iterative policy training on the policy network until the objective function of the policy network converges; The policy network obtained after the training is used to guide the robot skill learning.

2. The method of claim 1, wherein, The step of performing iterative policy training on the policy network by using the multi-branch value network based on the weight coefficients of each value branch comprises: During the policy training process, the weight coefficients of each value branch are updated according to a preset weight update strategy.

3. The method of claim 2, wherein, The step of updating the weight coefficients of each value branch according to the preset weight update strategy comprises: During the policy training process, when a preset weight update frequency is reached, a weight adjustment rule most suitable for the current policy is selected from a pre-constructed rule library by using the large language model based on the current policy performance and the historical policy performance, wherein the current policy is the policy adopted by the policy network after the current training round and used to select the optimal action; Based on the current policy performance, the weight adjustment rule is used to update the weight coefficients of each value branch.

4. The method of claim 3, wherein, Before the step of selecting the weight adjustment rule most suitable for the current policy from the pre-constructed rule library by using the large language model, the method further comprises: According to the learning prompt information and a preset rule example, the rule library for updating the weight is constructed by using the large language model.

5. The method of claim 1, wherein, The step of performing iterative policy training on the policy network by using the multi-branch value network based on the weight coefficients of each value branch further comprises: After each round of policy training, the advantage value of each value branch under the corresponding reward component function is weighted and fused according to the weight coefficients corresponding to the current each value branch to obtain a total advantage value under the overall target corresponding to the reward component function set; According to the objective function, the network parameters of the policy network are learned and updated by using the total advantage value to maximize the objective function.

6. A robot skill learning apparatus characterized by, The device comprises: A component set construction unit is configured to generate an auxiliary reward component function based on preset learning prompt information by using a large language model, and a reward component function set is constituted by a plurality of reward component functions in the learning prompt information and the auxiliary reward component function; A value network construction unit is configured to construct a multi-branch value network according to the reward component function set, wherein each value branch in the multi-branch value network corresponds to each reward component function in the reward component function set one by one; and The policy network training unit is configured to perform iterative policy training on the policy network by using the multi-branch value network based on the weight coefficients of the value branches until a target function of the policy network converges. The skill learning guidance unit is configured to guide robot skill learning by using the policy network obtained after the training.

7. The apparatus of claim 6, wherein, The policy network training unit includes: The weight coefficient updating unit is configured to update the weight coefficients of the value branches according to a preset weight updating policy during the policy training.

8. The apparatus of claim 7, wherein, The weight coefficient updating unit includes: The rule selection unit is configured to select a weight adjustment rule most suitable for a current policy from a pre-constructed rule library by using the large language model based on a current policy performance and a historical policy performance when a preset weight updating frequency is reached during the policy training, wherein the current policy is a policy adopted by the policy network after a current training round for selecting an optimal action. The weight updating sub-unit is configured to update the weight coefficients of the value branches by using the weight adjustment rule based on the current policy performance.

9. An agent device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, The processor executes the computer program to implement the steps of the method of any one of claims 1 to 5.

10. A computer-readable storage medium storing a computer program, the computer program comprising instructions that, when executed by a computer, cause the computer to perform the method of any one of claims 1 to 9. The computer program is executed by the processor to implement the steps of the method of any one of claims 1 to 5.

Images