Robot skill learning method, apparatus, device, and storage medium

Abstract

The application is suitable for the technical field of robot motion control, and provides a robot skill learning method, which comprises the following steps: based on task information and environment information, a preset reward-policy co-evolution strategy is used to iteratively evolve a reward function and a policy model used for guiding robot skill learning; and a target policy model obtained after evolution is used to control the robot to perform a target task in the task information, so that the skill learning effect of the robot in processing high-dimensional tasks is significantly enhanced while effectively reducing the training data requirement, and the accuracy, precision and efficiency of robot task execution are improved.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of robot motion control, and particularly relates to a robot skill learning method, device, equipment and storage medium. BACKGROUND

[0002] Robot skill learning refers to a process of enabling a robot to autonomously master and perform various complex tasks through algorithms and models, usually involving motion control, path planning and action coordination, etc. However, existing reinforcement learning (RL) methods have certain limitations in robot skill learning (such as training a robot dog to move). First, traditional RL methods usually need to train a policy model from scratch, which not only consumes time and has high computational cost, but also performs poorly when dealing with high-dimensional robot motion control tasks. In particular, when complex robot joint action coordination and diversified task objectives are involved, it is extremely challenging to design an effective reward function, which leads to low optimization efficiency of the policy model. In addition, existing methods that automatically generate reward functions using large language models (LLM) can generate reward functions through natural language task descriptions, but they need to train a policy model from scratch in each policy optimization, which greatly increases the computational cost and complexity. These methods usually rely on a large number of trial and error and large-scale search, resulting in huge training time and resource consumption in practical applications, and are unable to effectively cope with complex robot task requirements. SUMMARY

[0003] The present application aims to provide a robot skill learning method, device, equipment and storage medium, which aims to solve the problem of high learning cost, low learning efficiency and poor effect of robot skill learning caused by existing technology.

[0004] In one aspect, the present application provides a robot skill learning method, which comprises the following steps:

[0005] Based on the task information and the environment information, a preset reward-policy co-evolution strategy is used to iteratively evolve the reward function and the policy model used to guide the robot skill learning.

[0006] The target policy model obtained after evolution is used to control the robot to perform the target task in the task information.

[0007] Preferably, the step of iteratively evolving the reward function and the policy model used to guide the robot skill learning based on the task information and the environment information using the preset reward-policy co-evolution strategy comprises:

[0008] generate a plurality of reward functions based on the task information, the environment information, and prompt information;

[0009] generate a corresponding policy training model for each of the reward functions based on a preset model generation policy;

[0010] perform reinforcement learning training on the corresponding policy training model according to the reward function to obtain a score corresponding to each of the reward functions;

[0011] determine a global optimal reward function and a global optimal policy model according to the score corresponding to each of the reward functions;

[0012] determine whether an evolution end condition is reached, if yes, end the evolution and take the global optimal policy model as the target policy model, otherwise, take the global optimal reward function and the score corresponding to the global optimal reward function as the prompt information for the next round of evolution, and jump to the step of generating a plurality of reward functions based on the task information, the environment information, and prompt information through a large language model.

[0013] Preferably, the step of generating a corresponding policy training model for each of the reward functions based on a preset model generation policy comprises:

[0014] when the current evolution is the first round of evolution, randomly initialize the model parameters of the policy model to obtain a first random policy model, and take the first random policy model as the policy training model.

[0015] Preferably, the step of generating a corresponding policy training model for each of the reward functions based on a preset model generation policy further comprises:

[0016] when the current evolution is not the first round of evolution, randomly initialize the model parameters of the policy model to obtain a second random policy model;

[0017] perform parameter weighted fusion of the second random policy model and the global optimal policy model to obtain the policy training model.

[0018] Preferably, the step of performing parameter weighted fusion of the second random policy model and the global optimal policy model comprises:

[0019] search for a corresponding optimal fusion ratio for each of the reward functions based on Bayesian optimization based on a Gaussian process;

[0020] perform parameter weighted fusion of the second random policy model and the global optimal policy model based on the optimal fusion ratio to obtain the policy training model corresponding to each of the reward functions.

[0021] Preferably, the step of searching for the optimal fusion ratio corresponding to each reward function comprises:

[0022] A target function s(α)=V R (R,θ f (α),T BO ) is constructed for each reward function, wherein s(α) represents the performance score of the strategy under a given fusion ratio α, θ f (α) is a fusion strategy model constructed by taking α as the reward function R, V R () is a function for evaluating the performance or score of θ f (α) under R and training period T BO .

[0023] The Bayesian optimization of Gaussian process is performed on each target function to obtain the fusion ratio that maximizes each target function, and the fusion ratio is taken as the optimal fusion ratio of the corresponding reward function.

[0024] Preferably, the step of parameter-weighted fusion of the second random strategy model and the global optimal strategy model based on the optimal fusion ratio comprises:

[0025] According to the fusion strategy model θ f (α)=α·θ best +(1-α)·θ r and the optimal fusion ratio, the second random strategy model and the global optimal strategy model are parameter-weighted fused, and the fused θ f (α) is taken as the strategy training model corresponding to each reward function, wherein θ best is the global optimal strategy model, and θ r is the second random strategy model.

[0026] In another aspect, the present application provides a robot skill learning device, which comprises:

[0027] A strategy evolution unit is configured to perform iterative evolution on a reward function and a strategy model for guiding robot skill learning based on task information and environment information by using a preset reward-strategy co-evolution strategy.

[0028] A robot control unit is configured to control the robot to perform a target task in the task information by using the target strategy model obtained after evolution.

[0029] In another aspect, the present application also provides an agent device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the robot skill learning method as described above when executing the computer program.

[0030] In another aspect, the present application also provides a computer readable storage medium, which stores a computer program, wherein the computer program is executed by a processor to implement the steps of the robot skill learning method as described above.

[0031] The present application is based on task information and environment information, and adopts a preset reward-policy co-evolution strategy to iteratively evolve a reward function and a policy model for guiding robot skill learning, and uses the target policy model obtained after evolution to control the robot to perform a target task in the task information, thereby effectively reducing the demand for training data while significantly enhancing the skill learning effect of the robot in processing high-dimensional tasks, and improving the accuracy, precision and efficiency of robot task execution. BRIEF DESCRIPTION OF DRAWINGS

[0032] Figure 1 is an implementation flowchart of the robot skill learning method provided by the first embodiment of the present application;

[0033] Figure 2 is an implementation flowchart of the robot skill learning method provided by the second embodiment of the present application;

[0034] Figure 3 is a structural schematic diagram of the robot skill learning device provided by the third embodiment of the present application;

[0035] Figure 4 is a preferred structural schematic diagram of the robot skill learning device provided by the third embodiment of the present application;

[0036] Figure 5 is a structural schematic diagram of the agent device provided by the fourth embodiment of the present application. DETAILED DESCRIPTION

[0037] In order to make the objectives, technical solutions and advantages of the present application clearer, the present application will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application, and do not limit the present application.

[0038] It should be understood that the terms "first", "second", etc. in the present application are used to distinguish the same or similar items with basically the same function, and it should be understood that there is no logical or time sequence dependency between "first", "second", and "nth", and the number and execution order are not limited, unless otherwise specified.

[0039] It should be understood that the terms "comprising" and "having" and any variations thereof in the present application are intended to cover, but not exclusively, inclusive, for example, a product or device including a series of components does not have to be limited to those components clearly listed, but can include other components that are not clearly listed or inherent to these products or devices.

[0040] It should be understood that the model structures of the strategy model, the strategy training model, the target strategy model, the first random strategy model and the second random strategy model of the present application are all the same.

[0041] The specific implementation of the present application is described in detail below in combination with specific embodiments:

[0042] Example One:

[0043] Figure 1 The implementation process of the robot skill learning method provided by the embodiment of the present application is shown, only the parts related to the embodiments of the present application are shown for the convenience of description, and the details are as follows:

[0044] In step S101, based on the task information and the environment information, the pre-set reward-policy co-evolution strategy is used to iteratively evolve the reward function and the policy model for guiding the robot skill learning.

[0045] The embodiment of the present application is applicable to an intelligent agent device capable of perceiving an environment, making decisions and taking actions, such as a robot, a drone and the like. In the embodiment of the present application, task information and environment information respectively describe a task that needs to be completed by a robot and an environment in which the robot executes the task, the task information includes but is not limited to a target task, an expected behavior, and the environment information includes but is not limited to a state space, an action space, a possible interaction mode and dynamic characteristics of the environment, and these information is crucial for generating a reward function matched with the task and the environment, a policy model refers to a model used by the robot to decide what action (or action, behavior) to take in a given state, and the reward function is used to evaluate the quality of the policy adopted by the robot in executing the task, for example, after the robot executes an action in a given environment, the environment will give a corresponding reward value according to the reward function, and this reward value reflects the quality of the current action of the robot, and a preset reward-policy co-evolution strategy is a mechanism for alternately optimizing the reward function and the policy model, and here, based on the task information and the environment information, the reward function and the policy model used to guide skill learning of the robot are iteratively evolved by using the preset reward-policy co-evolution strategy, that is, in each evolution round, the reward function and the policy model are alternately optimized to form a co-evolution cycle, and specifically, in each evolution round, first, the reward function is adjusted and optimized according to the performance and feedback of the current policy model, to ensure that the reward function can accurately reflect the task that needs to be completed by the robot and guide the robot to learn a policy that meets the expectation of a human, and after the reward function is optimized, the policy model is trained and optimized based on the new reward function, in each evolution round, the reward function and the policy model depend on and influence each other and co-evolve, and this co-evolution process is continuously performed until a preset evolution end condition is met.

[0046] In a feasible embodiment, the specific implementation of iteratively evolving the reward function and the policy model used to guide skill learning of the robot based on the task information and the environment information by using the preset reward-policy co-evolution strategy is described in Embodiment Two below, and will not be described here again.

[0047] In step S102, the robot is controlled to execute the target task in the task information by using the target policy model obtained after the evolution is completed.

[0048] In the embodiment of the present application, the performance-optimal one is selected as the target policy model from the plurality of policy models whose evolution is completed, and the target policy model can accurately map the environment state to the corresponding robot control action, when the robot is controlled to execute the target task in the task information by using the target policy model obtained after the evolution is completed, the robot collects state information of the current environment, then inputs the state information into the target policy model to obtain the corresponding control action, and the robot executes the control action to achieve the target task.

[0049] In the embodiment of the present application, based on the task information and the environment information, the preset reward-policy co-evolution strategy is used to iteratively evolve the reward function and the policy model for guiding the robot skill learning, and the target policy model obtained after evolution is used to control the robot to perform the target task in the task information, so as to effectively reduce the training data requirement, significantly enhance the skill learning effect of the robot in processing high-dimensional tasks, and improve the accuracy, precision and efficiency of the robot task execution.

[0050] Example Two:

[0051] Figure 2 The implementation process of the robot skill learning method provided in the second embodiment of the present application is shown, only the parts related to the embodiments of the present application are shown for the convenience of description, and the details are as follows:

[0052] In step S201, based on the task information, the environment information and the prompt information, a large language model is used to generate a plurality of reward functions.

[0053] In the embodiment of the present application, the task information, the environment information and the prompt information are provided in the form of natural language, so that the large language model can understand and generate the reward functions related to the task and the environment, wherein the prompt information is used to guide the large language model to generate more optimal reward functions, improve the evolution speed and quality of the reward functions, and ensure that the reward functions can effectively guide the policy improvement and effectively improve the performance of the policy model in complex tasks. In this way, the task information, the environment information and the prompt information are input into the large language model, and the large language model generates a plurality of reward functions related to the task and the environment, each reward function provides a reward standard suitable for the task based on the task target and the environment constraint, and the plurality refers to 2 or more.

[0054] In a feasible embodiment, the prompt information includes the reward function with the best performance in the last round of evolution (i.e. the global optimal reward function) and the score obtained by evaluating the reward function under the corresponding policy, and the generation of the reward function is represented as wherein n is the round of the current evolution, I d is the task information, I e is the environment information, is the reward function with the best performance in the last round of evolution, and V(θ) is the score obtained by evaluating according to the policy θ, The kth reward function generated by the large language model in the nth round of evolution, so as to realize the automatic generation of the reward function optimized for the task (especially the complex task) by the large language model by combining the reinforcement learning task target, reduce the complexity of manually designing the reward function, and reduce the high labor cost caused by relying on human experts to design the reward function.

[0055] In another feasible embodiment, at the beginning of the first round of evolution, there is no prompt information for the large language model to refer to, that is, there is no global optimal reward function, therefore, the large language model needs to generate a plurality of reward functions according to the input task information and environment information, which cover different possibilities and strategies, and since these reward functions are generated based on the internal knowledge of the large language model and the understanding of the task environment, the quality of the reward functions may be poor, and from the second round, the global optimal reward function already exists, therefore, for reward evolution, the large language model already has a reference example, and the current global optimal reward function can be used as prompt information for the evolution of the new round of reward functions, and in each round of evolution, the large language model generates a new reward function, thereby introducing a dynamic reward generation mechanism based on the LLM, improving the adaptability and generalization ability of the robot in the complex task, and being capable of adaptively generating the optimal reward function according to different tasks and environments, ensuring that the reward function can always effectively guide the policy improvement, and significantly improving the precision and efficiency of task execution.

[0056] In step S202, based on the pre-constructed policy model, a preset model generation strategy is used to generate a corresponding policy training model for each reward function.

[0057] In the embodiment of the application, the policy model can be a neural network constructed using a deep learning framework, or a decision tree or other type of model, and the model structure of the policy model is not specifically limited here.

[0058] In a feasible embodiment, when the current evolution is the first round of evolution, the model parameters of the policy model are randomly initialized to obtain a first random policy model, and the first random policy model is used as the policy training model.

[0059] In the embodiment of the application, in the first round of evolution, since there is no global optimal policy model, the model parameters of the policy model are randomly initialized, and each reward function corresponds to a randomly initialized policy model, in order to facilitate description and differentiation, the randomly initialized policy model is referred to as a first random policy model, and the first random policy model corresponding to each reward function is used as the policy training model of the first round of evolution.

[0060] In another feasible embodiment, when the current evolution is not the first round of evolution, the model parameters of the strategy model are randomly initialized to obtain a second random strategy model, and the second random strategy model is parameter-weighted and fused with the global optimal strategy model to obtain a strategy training model.

[0061] In the embodiment of the application, from the second round of evolution, there is already a global optimal strategy model (i.e., the strategy model with the best performance in the history evolution), at this time, for a new batch of reward functions generated by the large language model, the global optimal strategy model is parameter-weighted and fused with a randomly initialized strategy model, and the fused model is used as the strategy training model for each reward function, that is, for each reward function, a strategy model is first randomly initialized, which is referred to as a second random strategy model for the convenience of description and differentiation, and then the second random strategy model is parameter-weighted and fused with the global optimal strategy model to obtain a strategy training model.

[0062] In another feasible embodiment, the second random strategy model is parameter-weighted and fused with the global optimal strategy model by the following implementation:

[0063] (1) Bayesian optimization based on Gaussian process is adopted to search for the best fusion ratio for each reward function;

[0064] In the embodiment of the application, since different reward functions need to search for different fusion ratios to optimize their performance, it is impossible to set a general fixed value to make each reward function adopt the same fusion ratio. Here, Bayesian optimization based on Gaussian process is adopted to gradually find which fusion ratio can bring higher scores by continuously iterating and evaluating the actual performance of the model in the task under different fusion ratios, and to gradually adjust the fusion ratio. Specifically, in each iteration, a fusion ratio is selected for evaluation, and the score corresponding to the fusion ratio is recorded. Then, these known data points (fusion ratio and score pairs) are used to guide the selection of the next iteration to find the fusion ratio that can bring the highest score. Through continuous iteration and evaluation, the optimal relationship between the real fusion ratio and the score is gradually approached, and the best fusion ratio is finally found, so that the best fusion ratio for each reward function is found with the least search computing power. This not only improves the search efficiency, but also effectively reduces the training cost of searching for the best fusion ratio.

[0065] Preferably, the search for the best fusion ratio is implemented by the following steps:

[0066] ① For each reward function, a corresponding objective function s(α) = V R (R,θ f (α),T BO), where s(a) represents the policy performance score at a given fusion ratio a, R is the reward function that is currently being searched for the fusion ratio, and a is the fusion ratio f (α) is a fusion policy model constructed using a as the reward function R, V R () is a function for evaluating the performance or score of f (α) under R and training period T BO .

[0067] ②Carry out Bayesian optimization of each objective function by using a Gaussian process to obtain a fusion ratio that maximizes each objective function, and take the fusion ratio as the optimal fusion ratio corresponding to the reward function.

[0068] In the embodiments of the present application, the fusion policy model f (α) is a new policy model obtained by fusing the global optimal policy model and the second random policy model using a, and a controls the fusion degree of the new policy model to the global optimal policy model and the second random policy model. Specifically, f (α) = a · best +(1-a)· r , where best is the global optimal policy model, and r is the second random policy model. By steps ① and ②, the historical optimal policy is partially inherited and fused with the random policy, and the policy fusion ratio is quickly adjusted by using Bayesian optimization. Compared with the traditional policy optimization method which usually relies on training the policy from scratch, the policy optimization scheme of the present application effectively reduces the consumption of invalid training samples and improves the policy optimization efficiency, especially in high-dimensional tasks.

[0069] (2) Based on the optimal fusion ratio, the second random policy model and the global optimal policy model are parameter-weighted fused to obtain a policy training model corresponding to each reward function.

[0070] In the embodiments of the present application, after the optimal fusion ratio a best is searched, the second random policy model and the global optimal policy model are parameter-weighted fused by using a f , that is, best +(1-a)· r , where best is the global optimal policy model, and f (α best ) = a best · best +(1-a best )· r , the fused f (α best) as the corresponding strategy training model of each reward function, so that the new strategy model retains the advantages of the historical optimal strategy, while having the ability to explore the new strategy space, retaining a certain plasticity while inheriting the past learning experience, effectively improving the diversity of strategy evolution.

[0071] Through the above steps (1) and (2), the second random strategy model is fused with the global optimal strategy model by parameter weighting, thereby accelerating the strategy parameter search through Bayesian optimization, reducing the training sample consumption of the search fusion ratio α, and effectively improving the training efficiency.

[0072] In step S203, the corresponding strategy training model is trained by reinforcement learning according to the reward function, and the scores corresponding to each reward function are obtained.

[0073] In the embodiment of the present application, each reward function is used to train the corresponding strategy training model. In the reinforcement learning training process, each strategy training model attempts to learn how to make optimal decisions under a given corresponding reward function to find a behavior strategy that maximizes the reward function. The performance of each strategy training model in the environment (such as cumulative reward) is evaluated to calculate the score for each corresponding reward function.

[0074] In a feasible embodiment, a training period is set, and during the training period, each strategy training model will start from the initial parameters and perform multiple iteration training through the reinforcement learning algorithm. In each iteration, the strategy training model will continuously learn and update its parameters according to the reward function, so that the robot continuously tries different strategies to maximize the cumulative reward, and the score of the reward function obtained in each iteration training is accumulated, which reflects the performance of the strategy training model under the current strategy. The process of obtaining the score corresponding to the reward function can be represented as where T max is the length of the training period, i.e. the number of iterations that each strategy training model is trained during evaluation, θ is the strategy training model corresponding to the kth reward function in the (n-1)th evolution, and θ0 is the initial parameter of the strategy training model.

[0075] In step S204, the global optimal reward function and the global optimal strategy model are determined according to the scores corresponding to each reward function.

[0076] In the embodiment of the present application, among all the trained strategy training models, the scores of all reward functions are compared, and the reward function with the highest score is selected as the optimal reward function of the current evolution round, i.e. wherein, For the optimal reward function of the current evolution round (i.e., the (n-1)th round), the corresponding strategy training model trained by the reward function is taken as the optimal strategy model of the current evolution round, and the optimal reward function and the optimal strategy model of the current evolution round are taken as the global optimal reward function and the global optimal strategy model for the next round of evolution.

[0077] In a feasible embodiment, the scores of both the optimal reward function of the current round of evolution and the global optimal reward function obtained in the historical evolution are compared, and when the score of the optimal reward function of the current round of evolution does not exceed the score of the global optimal reward function obtained in the historical evolution, the global optimal reward function obtained in the historical evolution is maintained unchanged, so that the global optimal reward function is not covered due to the exploration of a round of failed reward function, a vicious cycle is avoided, and it is ensured that the evolution of the reward function always develops in a good direction to obtain a better reward function, and the high-quality reward function can effectively improve the performance of the strategy model in complex tasks.

[0078] In step S205, when the evolution end condition is reached, the evolution is ended, and the global optimal strategy model is taken as the target strategy model.

[0079] In step S206, when the evolution end condition is not reached, the global optimal reward function and the score corresponding to the global optimal reward function are taken as the prompt information for the next round of evolution, and the process jumps to step S201.

[0080] In the embodiment of the application, it is judged whether the evolution end condition is reached, if yes, the evolution is ended, and the global optimal strategy model is taken as the target strategy model for controlling the robot action, otherwise, the global optimal reward function and the score corresponding to the global optimal reward function are taken as the prompt information for the next round of evolution to optimize the reward function, and the global optimal strategy model is used for the strategy optimization of the next round of evolution, and the process jumps to step S201 for continuous execution. The evolution end condition can be reaching a predetermined number of evolution times, the performance improvement of the strategy model being no longer significant, or meeting a specific performance index, etc., which is not specifically limited here.

[0081] In the embodiment of the application, first, a large language model is used to generate diversified reward functions, and then the optimal reward function is dynamically screened according to the performance of the robot task, so as to improve the task learning efficiency. In the strategy optimization, the random strategy and the historical optimal strategy are fused, and the Bayesian optimization method is used to adjust the strategy fusion parameters, so as to quickly find the optimal strategy combination. The collaborative evolution mechanism of the reward function and the strategy of the application can gradually optimize the skill learning process of the robot in a complex environment, and avoid the high cost of training the strategy model from scratch, greatly reduce the demand for training data, and significantly improve the learning speed and performance accuracy, realizing efficient and low-cost robot task learning and execution.

[0082] Example Three:

[0083] Figure 3 The structure of the robot skill learning device provided by the third embodiment of the present application is shown, only the parts related to the embodiments of the present application are shown for the convenience of description, which include:

[0084] The strategy evolution unit 31 is configured to perform iterative evolution on the reward function and the policy model for guiding the robot skill learning based on the task information and the environment information, using a preset reward-policy co-evolution strategy.

[0085] The robot control unit 32 is configured to control the robot to perform the target task in the task information using the target policy model obtained after the evolution is completed.

[0086] Preferably, as shown in the figure, the strategy evolution unit 31 includes: Figure 4 The reward function generation unit 311 is configured to generate a plurality of reward functions based on the task information, the environment information and the prompt information, using a large language model.

[0087] The training model generation unit 312 is configured to generate a corresponding policy training model for each reward function based on the pre-constructed policy model, using a preset model generation strategy.

[0088] The model reinforcement training unit 313 is configured to perform reinforcement learning training on the corresponding policy training model according to the reward function, to obtain a score corresponding to each reward function.

[0089] The global optimal determination unit 314 is configured to determine a global optimal reward function and a global optimal policy model according to the scores corresponding to each reward function.

[0090] The co-evolution end unit 315 is configured to end the evolution when the evolution end condition is reached, and take the global optimal policy model as the target policy model.

[0091] The prompt information setting unit 316 is configured to take the global optimal reward function and the score corresponding to the global optimal reward function as the prompt information for the next round of evolution when the evolution end condition is not reached, and trigger the reward function generation unit 311 to generate a plurality of reward functions based on the task information, the environment information and the prompt information, using a large language model.

[0092] Preferably, the training model generation unit 312 includes:

[0093]

[0094] ​The first random initialization unit is configured to randomly initialize model parameters of the policy model to obtain a first random policy model when the current evolution is the first round of evolution, and the first random policy model is used as the policy training model.

[0095] Preferably, the training model generation unit 312 further comprises:

[0096] The second random initialization unit is configured to randomly initialize model parameters of the policy model to obtain a second random policy model when the current evolution is not the first round of evolution.

[0097] The model parameter fusion unit is configured to perform parameter weighted fusion on the second random policy model and the global optimal policy model to obtain the policy training model.

[0098] Preferably, the model parameter fusion unit comprises:

[0099] The fusion ratio searching unit is configured to search for a best fusion ratio corresponding to each reward function based on Bayesian optimization of a Gaussian process.

[0100] The parameter fusion sub-unit is configured to perform parameter weighted fusion on the second random policy model and the global optimal policy model based on the best fusion ratio to obtain a policy training model corresponding to each reward function.

[0101] Preferably, the fusion ratio searching unit comprises:

[0102] The objective function construction unit is configured to construct a corresponding objective function s(α) = V R (R, θ f (α), T BO ) for each reward function for Bayesian optimization, where s(α) represents a policy performance score under a given fusion ratio α, θ f (α) is a fusion policy model constructed by using α as the reward function R, V R () is a function for evaluating the performance or score of θ f (α) under R and a training period T BO ;

[0103] The ratio searching sub-unit is configured to perform Bayesian optimization of a Gaussian process on each objective function to obtain a fusion ratio that maximizes each objective function, and the fusion ratio is used as the best fusion ratio of the corresponding reward function.

[0104] Preferably, the parameter fusion sub-unit comprises:

[0105] The training model obtaining unit is configured to obtain the fusion policy model θ f (α) = α·θ best + (1-α)·θ rand the optimal fusion ratio, the second random strategy model is parameter-weighted fused with the global optimal strategy model, and the fused θ f (α) is a strategy training model corresponding to each reward function, wherein θ best is a global optimal strategy model, and θ r is a second random strategy model.

[0106] In the embodiments of the present application, each unit of the robot skill learning device can be implemented by a corresponding hardware or software unit, and each unit can be an independent software or hardware unit, or can be integrated into a software or hardware unit, which does not limit the present application. Specifically, the implementation of each unit can refer to the description of the foregoing embodiments I and II, which will not be repeated here.

[0107] Example Four:

[0108] Figure 5 The structure of the agent device provided in the fourth embodiment of the present application is shown, and only the parts related to the embodiments of the present application are shown for ease of illustration.

[0109] The agent device 5 of the embodiment of the present application includes a processor 50, a memory 51, and a computer program 52 stored in the memory 51 and executable on the processor 50. The processor 50 implements the steps in the robot skill learning method embodiments described above when executing the computer program 52, such as Figure 1 the steps S101 to S102 shown. Alternatively, the processor 50 implements the functions of each unit in the device embodiments described above when executing the computer program 52, such as Figure 3 the functions of the units shown.

[0110] In the embodiments of the present application, based on the task information and the environment information, the reward function and the strategy model used to guide the robot skill learning are iteratively evolved using a preset reward-strategy co-evolution strategy, and the target strategy model obtained after evolution is used to control the robot to perform the target task in the task information, thereby effectively reducing the training data requirement while significantly enhancing the skill learning effect of the robot in processing high-dimensional tasks, and improving the accuracy, precision and efficiency of the robot task execution.

[0111] The agent device of the embodiment of the present application can be a robot. The steps implemented by the processor 50 in the agent device 5 when implementing the robot skill learning method while executing the computer program 52 can refer to the description of the foregoing method embodiments, which will not be repeated here.

[0112] Example Five:

[0113] In the embodiments of the present application, a computer readable storage medium is provided, which stores a computer program. When the computer program is executed by a processor, the steps in the above-mentioned robot skill learning method embodiments are implemented, for example, Figure 1 the steps S101 to S102 shown. Alternatively, when the computer program is executed by the processor, the functions of the units in the above-mentioned various device embodiments are implemented, for example Figure 3 the functions of the units shown.

[0114] In the embodiments of the present application, based on the task information and the environment information, the preset reward-policy co-evolution strategy is used to iteratively evolve the reward function and the policy model for guiding the robot skill learning, and the target policy model obtained after evolution is used to control the robot to perform the target task in the task information, so that the skill learning effect of the robot in processing high-dimensional tasks is significantly enhanced while effectively reducing the training data requirement, and the accuracy, precision and efficiency of the robot task execution are improved.

[0115] The computer readable storage medium of the embodiments of the present application can include any entity or device capable of carrying computer program code, recording medium, such as ROM / RAM, magnetic disk, optical disk, flash memory, etc.

[0116] The above only describes the preferred embodiments of the present application and is not intended to limit the present application. Any modification, equivalent replacement and improvement made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A method of robot skill learning, characterized by, The method comprises the following steps: Based on the task information and the environment information, the preset reward-policy co-evolution strategy is used to iteratively evolve the reward function and the policy model for guiding the skill learning of the robot; The robot is controlled to perform the target task in the task information by using the target policy model obtained after evolution; Based on the task information, the environment information and the prompt information, a plurality of reward functions are generated by a large language model; Based on the pre-constructed policy model, a preset model generation strategy is used to generate a corresponding policy training model for each reward function; According to the reward function, the corresponding policy training model is trained by reinforcement learning to obtain a score corresponding to each reward function; According to the score corresponding to each reward function, a global optimal reward function and a global optimal policy model are determined; It is judged whether the evolution end condition is reached, if yes, the evolution is ended, and the global optimal policy model is taken as the target policy model, otherwise, the global optimal reward function and the score corresponding to the global optimal reward function are taken as the prompt information for the next round of evolution, and the step of generating a plurality of reward functions based on the task information, the environment information and the prompt information by a large language model is jumped to; The step of generating a corresponding policy training model for each reward function by using a preset model generation strategy comprises: When the current evolution is the first round of evolution, the model parameters of the policy model are randomly initialized to obtain a first random policy model, and the first random policy model is taken as the policy training model; When the current evolution is not the first round of evolution, the model parameters of the policy model are randomly initialized to obtain a second random policy model; the second random policy model and the global optimal policy model are parameter weighted and fused to obtain the policy training model. The step of parameter weighted fusion of the second random policy model and the global optimal policy model comprises:

2. The method of claim 1, wherein, Using Gaussian process-based Bayesian optimization to search for the best fusion ratio corresponding to each reward function; Based on the best fusion ratio, the second random policy model and the global optimal policy model are parameter weighted and fused to obtain the policy training model corresponding to each reward function. The step of using Gaussian process-based Bayesian optimization to search for the best fusion ratio corresponding to each reward function comprises:

3. The method of claim 2, wherein, The Gaussian process-based Bayesian optimization is performed on each target function to obtain a fusion ratio that maximizes each target function, and the fusion ratio is taken as the best fusion ratio corresponding to the reward function. a corresponding objective function for Bayesian optimization is constructed for each of the reward functions wherein, denotes a policy performance score under a given fusion ratio is a fusion policy model constructed for the reward function is a function for evaluating the performance or score of under the conditions of and training period ​​​​ Based on the best fusion ratio, the second random policy model and the global optimal policy model are parameter weighted and fused to obtain the policy training model corresponding to each reward function.

4. The method of claim 3, wherein, The device comprises: According to the fusion strategy model And the optimal fusion ratio, the second random strategy model is parameter weighted and fused with the global optimal strategy model, and the fused As the strategy training model corresponding to each reward function, wherein The global optimal strategy model is The second random strategy model is 5. A robot skill learning apparatus characterized by, ​ A policy evolution unit is configured to evolve, based on task information and environment information, a reward function and a policy model for guiding skill learning of a robot by using a preset reward-policy co-evolution strategy. A robot control unit is configured to control the robot to perform a target task in the task information by using a target policy model obtained after evolution is completed. The policy evolution unit includes: A reward function generation unit is configured to generate a plurality of reward functions by using a large language model based on the task information, the environment information, and prompt information. A training model generation unit is configured to generate, for each of the reward functions, a corresponding policy training model by using a preset model generation strategy based on a pre-constructed policy model. A model reinforcement training unit is configured to perform reinforcement learning training on the corresponding policy training model according to the reward function, to obtain a score corresponding to each of the reward functions. A global optimum determination unit is configured to determine a global optimum reward function and a global optimum policy model according to the scores corresponding to each of the reward functions. A co-evolution end unit is configured to end evolution when an evolution end condition is reached, and to use the global optimum policy model as the target policy model. A prompt information setting unit is configured to, when the evolution end condition is not reached, use the global optimum reward function and the score corresponding to the global optimum reward function as the prompt information for the next round of evolution, and to trigger the reward function generation unit to generate a plurality of reward functions by using a large language model based on the task information, the environment information, and the prompt information. The training model generation unit includes: A first random initialization unit is configured to, when the current evolution is the first round of evolution, randomly initialize model parameters of the policy model to obtain a first random policy model, and to use the first random policy model as the policy training model. A second random initialization unit is configured to, when the current evolution is not the first round of evolution, randomly initialize model parameters of the policy model to obtain a second random policy model. A model parameter fusion unit is configured to perform parameter weighted fusion on the second random policy model and the global optimum policy model to obtain the policy training model. 6.An agent device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein, The processor executes the computer program to implement the steps of the method of any one of claims 1 to 4.

7. A computer-readable storage medium storing a computer program, wherein the computer program comprises the following steps of: receiving a request for a resource from a client; determining whether the client is authorized to access the resource; and if the client is authorized to access the resource, providing the resource to the client. The computer program is executed by the processor to implement the steps of the method of any one of claims 1 to 4.

Images