Robot skill learning method, device and equipment and storage medium

Abstract

The invention is applicable to the technical field of robot motion control, and provides a robot skill learning method, which comprises the following steps of: generating an auxiliary reward component function by utilizing a large language model according to learning prompt information, constructing a multi-branch value network according to a reward component function set formed by a plurality of reward component functions and auxiliary reward component functions in the learning prompt information, and performing iterative strategy training on the strategy network by using the multi-branch value network based on the weight coefficient of each value branch until the target function of the strategy network is converged, so as to obtain the target function of the strategy network. A strategy network is used for guiding robot skill learning, so that the intelligent decision-making capability of a large language model and fine-grained feedback of a multi-branch value network are combined, the strategy learning efficiency and the overall performance in reinforcement learning are improved, it is ensured that different skill obtaining requirements can be met in the strategy optimization process, and the robot skill learning efficiency is improved. And different environments and complex robot tasks can be handled, and the accuracy, precision and efficiency of task execution are improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of robot motion control, and in particular relates to a robot skill learning method, device, equipment and storage medium. Background Art

[0002] Embodied intelligent robots collect interaction data through interaction with the environment, aiming to learn the skills expected by humans to perform specific actions or tasks. However, with the increase in the robot's degree of freedom and the increasing complexity of the dynamic environment, the realization of this task has become more difficult. This complexity requires the imposition of additional constraints as optimization objectives to ensure that the robot can complete the task efficiently and accurately. Reinforcement Learning (RL) is an effective strategy optimization method. It optimizes the strategy by maximizing the cumulative reward and effectively converts each robot constraint into a reward component. This method has been successful in various robot tasks, such as leg motion control, dexterous hand operation, and overall robot manipulation. However, current RL methods optimize the strategy by adding all reward components, which means that the robot needs to learn multiple optimization objectives in parallel at the same time, which is difficult for the robot and makes the learning inefficient.

[0003] Van et al. proposed a hybrid reward architecture (HRA), which is applied in reinforcement learning. It improves sample efficiency and learning effect by designing fine-grained reward functions. Specifically, HRA decomposes complex reward functions into multiple simple sub-reward components, each of which corresponds to different task goals or events, thereby guiding the robot to learn more effectively. This method has been proven to significantly improve learning efficiency because it allows the robot to focus more on the specific learning needs of each sub-reward component. Furthermore, Huang et al. introduced the hybrid dynamic policy gradient (HDPG) method to optimize this process. This method achieves intelligent guidance of the robot's learning process by dynamically adjusting the contribution (i.e., weight) of each reward branch in the policy optimization process. The HDPG technology enables the robot to prioritize learning components that can quickly accumulate rewards, so that the robot can focus on "simpler" components before dealing with more "challenging" components. However, it is worth noting that the effectiveness of the HDPG method largely relies on the dynamic weight calculation rules designed by human experts, which may not always be applicable or guarantee optimal performance in different robotic tasks. Therefore, the widespread application and performance optimization of the HDPG method still face challenges. Summary of the invention

[0004] The purpose of the present invention is to provide a robot skill learning method, device, equipment and storage medium, aiming to solve the problem that robot skill learning is not effective for complex tasks and environments due to the existing technology.

[0005] Poor learning performance in complex tasks and environments

[0006] In one aspect, the present invention provides a method for learning robot skills, the method comprising the following steps:

[0007] Based on preset learning prompt information, an auxiliary reward component function is generated using a large language model, and a reward component function set is formed by a plurality of reward component functions in the learning prompt information and the auxiliary reward component function;

[0008] Constructing a multi-branch value network according to the reward component function set, wherein each value branch in the multi-branch value network corresponds one-to-one to each reward component function in the reward component function set;

[0009] Based on the weight coefficients of the value branches, the multi-branch value network is used to iteratively train the policy network until the objective function of the policy network converges;

[0010] The strategy network obtained after training is used to guide the robot's skill learning.

[0011] Preferably, based on the weight coefficients of the value branches, the step of iteratively training the policy network using the multi-branch value network comprises:

[0012] During the strategy training process, the weight coefficient of each value branch is updated according to a preset weight update strategy.

[0013] Preferably, the step of updating the weight coefficient of each value branch according to a preset weight update strategy includes:

[0014] During the strategy training process, when a preset weight update frequency is reached, based on the current strategy performance and the historical strategy performance, the large language model is used to select the weight adjustment rule that best suits the current strategy from the pre-built rule library, wherein the current strategy is the strategy adopted by the strategy network after the current training round for selecting the optimal action;

[0015] Based on the current strategy performance, the weight coefficient of each value branch is updated using the weight adjustment rule.

[0016] Preferably, before the step of selecting the weight adjustment rule that best suits the current strategy from a pre-built rule base using the large language model, the method further comprises:

[0017] According to the learning prompt information and preset rule examples, the rule base for updating weights is constructed using the large language model.

[0018] Preferably, based on the weight coefficients of the value branches, the step of iteratively training the policy network using the multi-branch value network further includes:

[0019] After each round of strategy training, the advantage value of each value branch under the corresponding reward component function is weighted and fused according to the weight coefficient corresponding to each current value branch, so as to obtain the total advantage value under the overall goal corresponding to the reward component function set;

[0020] According to the objective function, the network parameters of the policy network are learned and updated using the total advantage value so as to maximize the objective function.

[0021] In another aspect, the present invention provides a robot skill learning device, the device comprising:

[0022] A component set construction unit, used to generate an auxiliary reward component function based on preset learning prompt information using a large language model, wherein a reward component function set is formed by a plurality of reward component functions in the learning prompt information and the auxiliary reward component function;

[0023] A value network construction unit, configured to construct a multi-branch value network according to the reward component function set, wherein each value branch in the multi-branch value network corresponds one-to-one to each reward component function in the reward component function set;

[0024] A policy network training unit, configured to perform iterative policy training on the policy network using the multi-branch value network based on the weight coefficients of the value branches until the objective function of the policy network converges;

[0025] The skill learning guidance unit is used to guide the robot's skill learning by using the strategy network obtained after the training.

[0026] Preferably, the policy network training unit comprises:

[0027] The weight coefficient updating unit is used to update the weight coefficient of each value branch according to a preset weight updating strategy during the strategy training process.

[0028] Preferably, the weight coefficient updating unit comprises:

[0029] A rule selection unit, configured to select, during the strategy training process, a weight adjustment rule that best suits the current strategy from a pre-built rule base using the large language model when a preset weight update frequency is reached, based on current strategy performance and historical strategy performance, wherein the current strategy is a strategy adopted by the strategy network after a current training round for selecting an optimal action;

[0030] The weight updating subunit is used to update the weight coefficient of each value branch based on the current strategy performance and using the weight adjustment rule.

[0031] On the other hand, the present invention also provides an intelligent device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps described in the above-mentioned robot skill learning method when executing the computer program.

[0032] On the other hand, the present invention also provides a computer-readable storage medium, which stores a computer program, and when the computer program is executed by a processor, it implements the steps described in the above-mentioned robot skill learning method.

[0033] The present invention generates an auxiliary reward component function using a large language model according to learning prompt information, constructs a multi-branch value network according to a reward component function set consisting of a number of reward component functions in the learning prompt information and the auxiliary reward component function, and based on the weight coefficients of each value branch in the multi-branch value network, uses the multi-branch value network to iteratively train the policy network until the objective function of the policy network converges, and uses the policy network obtained at the end of training to guide the robot skill learning, thereby improving the strategy learning efficiency and overall performance in reinforcement learning by combining the intelligent decision-making ability of the large language model and the fine-grained feedback of the multi-branch value network, ensuring that the strategy optimization process can adapt to different skill acquisition needs, can cope with different environments and complex robot tasks, and improves the accuracy, precision and efficiency of robot task execution. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 is a flowchart of the implementation of the robot skill learning method provided in the first embodiment of the present invention;

[0035] Figure 2 is a schematic diagram of the structure of a robot skill learning device provided in Embodiment 2 of the present invention;

[0036] Figure 3 This is a schematic diagram of a preferred structure of a robot skill learning device provided in Embodiment 2 of the present invention;

[0037] Figure 4It is a schematic diagram of the structure of the intelligent body device provided in the third embodiment of the present invention. DETAILED DESCRIPTION

[0038] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0039] The specific implementation of the present invention is described in detail below in conjunction with specific embodiments:

[0040] Embodiment 1:

[0041] Figure 1 The implementation process of the robot skill learning method provided in the first embodiment of the present invention is shown. For the convenience of explanation, only the part related to the embodiment of the present invention is shown, which is described in detail as follows:

[0042] In step S101, based on preset learning prompt information, an auxiliary reward component function is generated using a large language model, and a reward component function set is composed of several reward component functions in the learning prompt information and the auxiliary reward component function.

[0043] The embodiment of the present invention is applicable to intelligent devices that can perceive the environment, make decisions and take actions, such as robots, drones, etc. In the embodiment of the present invention, for complex tasks that may involve multiple goals, constraints and conditions, a single, global reward function may not be able to provide sufficiently accurate and detailed feedback for the actions taken by the robot in different states. Here, the original reward function is first decomposed into multiple reward component functions, each of which focuses on a specific aspect of the task (such as goal achievement, state transition, path efficiency, safety or resource utilization, etc.), so that the robot can better understand the task structure and provide clear reward or punishment signals for different behaviors taken by the robot in each specific aspect. This helps to guide the robot to take correct actions in specific situations and avoid unnecessary exploration or wrong behaviors, so that the robot can learn more accurate and efficient strategies and improve the performance of the robot in complex tasks. After that, the task information, environmental information and each reward component function are all written into the learning prompt information in text form, and then the learning prompt information is input into the large language model (Large Language Model). In the large language model (LLM), one or more additional reward component functions for assisting strategy training are generated through the understanding of the training task and the optimization goal by the large language model. These reward component functions are intended to enhance the reward signal in the strategy training process. They may target aspects that are not fully covered in the current reward mechanism, or provide additional incentives for specific behaviors. In order to facilitate the distinction between the reward component functions decomposed from the original reward function, the reward component function generated by the large language model is called an auxiliary reward component function. Finally, the reward component functions and the auxiliary reward component functions decomposed from the original reward function constitute a reward component function set. The elements in the reward component function set form the final reward function through a specific combination method (such as weighted sum). In addition, the task information and the environment information respectively describe the tasks that the robot needs to complete and the environment in which the robot performs the tasks. The task information includes but is not limited to the target task and the expected behavior. The environment information includes but is not limited to the state space, the action space, the possible interaction mode, and the dynamic characteristics of the environment. The generation of the auxiliary reward component function can be expressed as C a =LLM r (T t ,T e ,C r ), the reward component function set is represented by C = {C r ,C a}={r1,r2,...,r K}, where T t is the task information, T e is environmental information, C r is the reward component function decomposed from the original reward function, C ais the auxiliary reward component function, r K is the Kth reward component function in the reward component function set C, and K is the total number of reward component functions in the reward component function set C.

[0044] In step S102, a multi-branch value network is constructed according to the reward component function set, wherein each value branch in the multi-branch value network corresponds one-to-one to each reward component function in the reward component function set.

[0045] In an embodiment of the present invention, according to a set of reward component functions, a multi-layer perceptron (MLP) architecture in a deep neural network is used to construct a multi-branch value network. Each value branch in the multi-branch value network corresponds to each reward component function in the set of reward component functions. Each value branch focuses on evaluating the value of the state or state feature associated with its corresponding reward component function. These states or state features are usually extracted from the original state space and are used to describe the specific situation or position of the robot in the environment. Specifically, in the multi-branch value network, each value branch uses the current state or state feature associated with its corresponding reward component function as input, and processes the respective input states in parallel to calculate the corresponding value. After that, the advantage value under the corresponding reward component function is calculated based on the value and output. The advantage value output by each value branch will be used to guide the policy update, so that the policy network makes a better decision. Since the multi-branch value network takes into account multiple reward component functions and corresponding branch states, it can capture different situations or positions of the robot in the environment in a more detailed manner, which helps to improve the adaptability and robustness of the intelligent body, and enables it to consider more factors in the decision-making process, so as to make more accurate judgments.

[0046] In step S103, based on the weight coefficients of each value branch, the multi-branch value network is used to perform iterative policy training on the policy network until the objective function of the policy network converges.

[0047] In an embodiment of the present invention, a weight coefficient is assigned to each value branch, and these weight coefficients are used to balance the impact of different value branches on policy updates. Based on the weight coefficients of each value branch, the policy network is iteratively trained using a multi-branch value network until the objective function of the policy network converges. When the policy network is iteratively trained using a multi-branch value network, in each iteration, the policy network selects an action based on the current policy, and the output of the multi-branch value network is used to evaluate the quality of different actions under the current policy and guide the update of the policy network. Through iterative training, the policy network gradually learns to select actions that can maximize long-term returns. When the objective function value of the policy network no longer changes significantly in multiple consecutive iterations, it is considered that the policy network has converged to a stable state.

[0048] In a feasible embodiment, during the strategy training process, the weight coefficient of each value branch is updated according to a preset weight update strategy.

[0049] In an embodiment of the present invention, during the strategy training process, the weight coefficient of each value branch is updated in stages according to a preset weight update strategy, thereby avoiding training instability caused by frequent adjustment of the weight coefficient, and by adjusting the weight coefficient at different training stages, the strategy network is guided to pay more attention to certain reward components, thereby achieving fine control of the overall behavior or performance.

[0050] In another feasible embodiment, the weight coefficient of each value branch is updated by the following steps:

[0051] (1) During the policy training process, when the preset weight update frequency is reached, based on the current policy performance and the historical policy performance, the large language model is used to select the weight adjustment rule that best suits the current policy from the pre-built rule library, where the current policy is the policy adopted by the policy network after the current training round to select the optimal action;

[0052] In an embodiment of the present invention, the weight update frequency is used to regularly adjust the weight coefficient during the training process to optimize the performance of the policy network. The weight update frequency can be set according to the training strategy and requirements (for example, according to the number of iterations of the policy training (updated every 100 rounds)). The current policy performance is composed of the average expected return of each reward component function under the current policy, and the historical policy performance is composed of the average expected return of each reward component function within the historical length. During the policy training process, each round will update the policy network according to the current weight coefficient and training data, and after the current training round, it is determined whether the preset weight update frequency is reached. Otherwise, the weight coefficient is not updated, and the next round of policy training is directly carried out. If so, based on the current policy performance and the historical policy performance, the large language model is used to select the weight adjustment rule that best suits the current policy from the pre-built rule library. Here, the selection process of the weight adjustment rule is represented as in, is the current strategy performance, is the historical strategy performance, L represents the historical length, l is the current training round, and B n is the rule base, B selected is the selected weight adjustment rule, is the average expected return of the Kth reward component function under the current strategy corresponding to the current training round l.

[0053] (2) Based on the current strategy performance, the weight coefficient of each value branch is updated using the weight adjustment rules.

[0054] In the embodiment of the present invention, after determining the current weight adjustment rule, the weight coefficient of each value branch will be updated based on the rule. Specifically, Among them, S σ =[σ 1 ,σ 2 ,...,σ K ] The average expected return for each value branch The variance of K for The corresponding variance, w K is the weight coefficient of the value branch corresponding to the Kth reward component function. The next round of strategy training updates the strategy network based on the updated weight coefficient.

[0055] The above steps (1) and (2) combine the intelligent decision-making ability of the large language model and the fine-grained feedback of the multi-branch value network to achieve more flexible adjustment and switching of weight adjustment rules during the training process, thereby achieving a more reasonable configuration of dynamic weights, and then achieving more effective dynamic adjustment of strategy optimization in reinforcement learning, so that the robot can effectively handle the complexity and uncertainty in different task scenarios during the process of skill acquisition, thereby improving the strategy learning efficiency and overall performance in reinforcement learning.

[0056] In a feasible embodiment, in the initial stage of strategy training, the update of the weight coefficient has not been triggered, and the large language model has not received the feedback information of the multi-value network on the strategy performance (i.e., the current strategy performance and the historical strategy performance). At this stage, the strategy network is trained with the preset weight coefficient, and the preset weight coefficient can be set to 1. As an example, a task is trained for 1500 rounds, and the strategy network is updated in each round, but only every 100 rounds, the rules are reselected and the new weight coefficient is determined. Before the next 100 rounds, the intermediate rounds are updated with the weight coefficient determined in the last 100 rounds, that is, in rounds 1 to 100, the strategy network is updated with the initially set weight coefficient (e.g., 1), in rounds 101 to 200, the strategy network is updated with the weight coefficient determined after the 100th round, in rounds 201 to 300, the strategy network is updated with the weight coefficient determined after the 200th round, and so on, until the end of training.

[0057] In another feasible embodiment, before the multi-branch value network is used to iteratively train the policy network based on the weight coefficients of each value branch, a rule base for updating the weight is constructed using a large language model according to the learning prompt information and the preset rule examples. Specifically, the learning prompt information and the preset rule examples are input into the large language model, and a number of rules for updating the weights are generated using the powerful reasoning and generation capabilities of the large language model. A rule base is constructed from the generated rules, wherein the rule examples refer to some exemplary rules or patterns used to guide the large language model on how to generate high-quality rules. Here, the rules proposed in the HDPG method are used as rule examples and are represented as E hdpg , in order to improve the rationality of the rules generated by the large language model, the construction of the rule base is expressed as: B n =LLM g (T t ,T e ,C r ,E hdpg), thereby dynamically generating a variety of effective rules through a large language model, allowing the weight coefficients to be adjusted in a more fine-grained manner. In the subsequent training process, the learning priority of each value branch can be adjusted more flexibly and dynamically according to more diverse rules, ensuring that the strategy optimization process can adapt to different skill acquisition needs and cope with different environments and complex robot tasks.

[0058] In another feasible embodiment, the policy training of the policy network is implemented by the following steps:

[0059] ① After each round of strategy training, the advantage value of each value branch under the corresponding reward component function is weighted and fused according to the weight coefficient corresponding to the current value branch, and the total advantage value under the overall goal corresponding to the reward component function set is obtained;

[0060] In the embodiment of the present invention, after each round of strategy training, the advantage value output by each value branch under the corresponding reward component function is calculated, and the weight coefficient vector w=[w1,w2,...,w K ] are multiplied element by element, and the multiplication results are summed to obtain the total advantage value under the overall goal corresponding to the reward component function set, that is, Among them, A k is the advantage value of the k-th value branch output, w k is the weight coefficient corresponding to the kth value branch, is the total advantage value.

[0061] ② According to the objective function, the total advantage value is used to learn and update the network parameters of the strategy network to maximize the objective function.

[0062] In the embodiment of the present invention, according to the objective function The total advantage value is used to learn and update the network parameters of the policy network to maximize the objective function. Specifically, in order to find the objective function J(π θ ) (i.e., expected return) maximizes the network parameter θ, where the log probability gradient along the policy The update parameters are updated in the direction of If the total advantage value of an action is high, then the probability of selecting this action is increased. Conversely, if the total advantage value is low, then the probability of selecting this action is reduced.

[0063] Represents strategy π θ The objective function J(π) (parameterized by the network parameters θ) θ )The gradient of θ with respect to θ;

[0064] Represents a state-action pair (s t ,a t ), which is the expectation of the strategy π θ It is calculated based on the transition probability of the environment and the strategy itself;

[0065] Represents strategy π θ In a given state s t Next select action a t The gradient of the log probability of θ with respect to θ indicates how to adjust θ to increase the probability of choosing the current action a. t probability (if this action is considered good, i.e. has a high advantage value).

[0066] The above steps ① and ② realize direct optimization of policy parameters under complex environments and reward structures to train the policy network and adapt to continuous and high-dimensional action spaces, thereby improving training speed and training effect.

[0067] In step S104, the strategy network obtained after training is used to guide the robot's skill learning.

[0068] In an embodiment of the present invention, the policy network obtained after training can accurately map the environment state to the corresponding robot control action, that is, it contains the knowledge of how the robot selects the optimal action according to the environment state. Here, the policy network obtained after training is loaded into the robot system. During the operation of the robot, the policy network will output an action according to the current environment state. The robot will execute this action and observe the change of the environment state. After the robot executes the action, the environment will give a feedback (usually a reward or punishment). According to this feedback, the robot will update its own state and prepare to make the next decision. The robot will continuously repeat the above-mentioned process of executing the action output by the policy network-updating the state, and gradually master new skills through continuous interaction and learning with the environment.

[0069] In an embodiment of the present invention, an auxiliary reward component function is generated according to learning prompt information using a large language model, a reward component function set is composed of several reward component functions in the learning prompt information and auxiliary reward component functions, a multi-branch value network is constructed according to the reward component function set, and based on the weight coefficients of each value branch in the multi-branch value network, the multi-branch value network is used to iteratively train the policy network until the objective function of the policy network converges, and the policy network obtained at the end of training is used to guide the robot skill learning, thereby improving the strategy learning efficiency and overall performance in reinforcement learning by combining the intelligent decision-making ability of the large language model and the fine-grained feedback of the multi-branch value network, ensuring that the strategy optimization process can adapt to different skill acquisition needs, can cope with different environments and complex robot tasks, and improve the accuracy, precision and efficiency of robot task execution.

[0070] Embodiment 2:

[0071] Figure 2 The structure of the robot skill learning device provided by the second embodiment of the present invention is shown. For the convenience of description, only the parts related to the embodiment of the present invention are shown, including:

[0072] A component set construction unit 21 is used to generate an auxiliary reward component function based on preset learning prompt information using a large language model, wherein a reward component function set is formed by a plurality of reward component functions in the learning prompt information and the auxiliary reward component function;

[0073] A value network construction unit 22, configured to construct a multi-branch value network according to the reward component function set, wherein each value branch in the multi-branch value network corresponds one-to-one to each reward component function in the reward component function set;

[0074] A policy network training unit 23, used for iteratively training the policy network using a multi-branch value network based on the weight coefficients of each value branch until the objective function of the policy network converges;

[0075] The skill learning guidance unit 24 is used to guide the robot's skill learning using the strategy network obtained after the training.

[0076] Among them, Figure 3 As shown, preferably, the strategy network training unit 23 includes:

[0077] The weight coefficient updating unit 231 is used to update the weight coefficient of each value branch according to a preset weight updating strategy during the strategy training process.

[0078] The weight coefficient updating unit 231 includes:

[0079] A rule selection unit 2311 is used to select the weight adjustment rule that best suits the current strategy from a pre-built rule base using a large language model when a preset weight update frequency is reached during the strategy training process, based on the current strategy performance and the historical strategy performance, wherein the current strategy is the strategy adopted by the strategy network after the current training round for selecting the optimal action;

[0080] The weight updating subunit 2312 is used to update the weight coefficient of each value branch based on the current strategy performance using the weight adjustment rule.

[0081] In another embodiment, the strategy network training unit 23 further includes:

[0082] The advantage value fusion unit is used to perform weighted fusion of the advantage values ​​of each value branch under the corresponding reward component function according to the weight coefficients corresponding to the current value branches after each round of strategy training, and obtain the total advantage value under the overall goal corresponding to the reward component function set;

[0083] The network parameter updating unit is used to learn and update the network parameters of the policy network according to the objective function using the total advantage value so as to maximize the objective function.

[0084] Still preferably, the robot skill learning device of the embodiment of the present invention further includes:

[0085] The rule base construction unit is used to construct a rule base for updating weights using a large language model according to learning prompt information and preset rule examples.

[0086] In the embodiment of the present invention, each unit of the robot skill learning device can be implemented by a corresponding hardware or software unit, and each unit can be an independent software or hardware unit, or can be integrated into a software or hardware unit, which is not intended to limit the present invention. Specifically, the implementation of each unit can refer to the description of the aforementioned embodiment 1, which will not be repeated here.

[0087] Embodiment three:

[0088] Figure 4 The structure of the intelligent body device provided in the third embodiment of the present invention is shown. For the convenience of explanation, only the part related to the embodiment of the present invention is shown.

[0089] The intelligent agent device 4 of the embodiment of the present invention includes a processor 40, a memory 41, and a computer program 42 stored in the memory 41 and executable on the processor 40. When the processor 40 executes the computer program 42, the steps in the embodiment of the robot skill learning method are implemented, such as Figure 1 Alternatively, when the processor 40 executes the computer program 42, the functions of each unit in the above-mentioned device embodiments are realized, for example Figure 2 Function of the unit shown.

[0090] In an embodiment of the present invention, an auxiliary reward component function is generated by using a large language model according to learning prompt information, a multi-branch value network is constructed according to a reward component function set consisting of several reward component functions in the learning prompt information and auxiliary reward component functions, and based on the weight coefficients of each value branch in the multi-branch value network, the multi-branch value network is used to iteratively train the policy network until the objective function of the policy network converges, and the policy network obtained at the end of training is used to guide the robot skill learning, thereby improving the strategy learning efficiency and overall performance in reinforcement learning by combining the intelligent decision-making ability of the large language model and the fine-grained feedback of the multi-branch value network, ensuring that the strategy optimization process can adapt to different skill acquisition needs, can cope with different environments and complex robot tasks, and improve the accuracy, precision and efficiency of robot task execution.

[0091] The intelligent device of the embodiment of the present invention may be a robot. The steps implemented when the processor 40 in the intelligent device 4 executes the computer program 42 to implement the robot skill learning method can refer to the description of the aforementioned method embodiment, which will not be repeated here.

[0092] Embodiment 4:

[0093] In an embodiment of the present invention, a computer-readable storage medium is provided, which stores a computer program. When the computer program is executed by a processor, the steps in the above-mentioned robot skill learning method embodiment are implemented, for example, Figure 1 Alternatively, when the computer program is executed by a processor, the functions of each unit in the above-mentioned device embodiments are realized, for example Figure 2 Function of the unit shown.

[0094] In an embodiment of the present invention, an auxiliary reward component function is generated by using a large language model according to learning prompt information, a multi-branch value network is constructed according to a reward component function set consisting of several reward component functions in the learning prompt information and auxiliary reward component functions, and based on the weight coefficients of each value branch in the multi-branch value network, the multi-branch value network is used to iteratively train the policy network until the objective function of the policy network converges, and the policy network obtained at the end of training is used to guide the robot skill learning, thereby improving the strategy learning efficiency and overall performance in reinforcement learning by combining the intelligent decision-making ability of the large language model and the fine-grained feedback of the multi-branch value network, ensuring that the strategy optimization process can adapt to different skill acquisition needs, can cope with different environments and complex robot tasks, and improve the accuracy, precision and efficiency of robot task execution.

[0095] The computer-readable storage medium of the embodiment of the present invention may include any entity or device or recording medium capable of carrying computer program code, for example, ROM / RAM, magnetic disk, optical disk, flash memory and other memories.

[0096] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A robot skill learning method, characterized in that: The method comprises the following steps: Based on preset learning prompt information, an auxiliary reward component function is generated using a large language model, and a reward component function set is formed by a plurality of reward component functions in the learning prompt information and the auxiliary reward component function; Constructing a multi-branch value network according to the reward component function set, wherein each value branch in the multi-branch value network corresponds one-to-one to each reward component function in the reward component function set; Based on the weight coefficients of the value branches, the multi-branch value network is used to iteratively train the policy network until the objective function of the policy network converges; The strategy network obtained after training is used to guide the robot's skill learning.

2. The method according to claim 1, characterized in that The step of iteratively training the policy network using the multi-branch value network based on the weight coefficient of each value branch includes: During the strategy training process, the weight coefficient of each value branch is updated according to a preset weight update strategy.

3. The method according to claim 2, characterized in that The step of updating the weight coefficient of each value branch according to a preset weight update strategy includes: During the strategy training process, when a preset weight update frequency is reached, based on the current strategy performance and the historical strategy performance, the large language model is used to select the weight adjustment rule that best suits the current strategy from the pre-built rule library, wherein the current strategy is the strategy adopted by the strategy network after the current training round for selecting the optimal action; Based on the current strategy performance, the weight coefficient of each value branch is updated using the weight adjustment rule.

4. The method according to claim 3, characterized in that Before the step of selecting the weight adjustment rule that best suits the current strategy from a pre-built rule base using the large language model, the method further includes: According to the learning prompt information and preset rule examples, the rule base for updating weights is constructed using the large language model.

5. The method according to claim 1, characterized in that The step of iteratively training the strategy network using the multi-branch value network based on the weight coefficient of each value branch also includes: After each round of strategy training, the advantage value of each value branch under the corresponding reward component function is weighted and fused according to the weight coefficient corresponding to each current value branch, so as to obtain the total advantage value under the overall goal corresponding to the reward component function set; According to the objective function, the network parameters of the policy network are learned and updated using the total advantage value so as to maximize the objective function.

6. A robot skill learning device, characterized in that: The device comprises: A component set construction unit, used to generate an auxiliary reward component function based on preset learning prompt information using a large language model, wherein a reward component function set is formed by a plurality of reward component functions in the learning prompt information and the auxiliary reward component function; A value network construction unit, configured to construct a multi-branch value network according to the reward component function set, wherein each value branch in the multi-branch value network corresponds one-to-one to each reward component function in the reward component function set; A policy network training unit, configured to perform iterative policy training on the policy network using the multi-branch value network based on the weight coefficients of the value branches until the objective function of the policy network converges; The skill learning guidance unit is used to guide the robot's skill learning by using the strategy network obtained after the training.

7. The device according to claim 6, characterized in that The strategy network training unit comprises: The weight coefficient updating unit is used to update the weight coefficient of each value branch according to a preset weight updating strategy during the strategy training process.

8. The device according to claim 7, characterized in that The weight coefficient updating unit comprises: A rule selection unit, configured to select, during the strategy training process, a weight adjustment rule that best suits the current strategy from a pre-built rule base using the large language model when a preset weight update frequency is reached, based on current strategy performance and historical strategy performance, wherein the current strategy is a strategy adopted by the strategy network after a current training round for selecting an optimal action; The weight updating subunit is used to update the weight coefficient of each value branch based on the current strategy performance and using the weight adjustment rule.

9. An intelligent agent device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 5 are implemented.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 5 are implemented.

Images