Robot skill learning method, device and equipment and storage medium

Abstract

The invention is suitable for the technical field of robot motion control, and provides a robot skill learning method, which comprises the following steps: based on task information and environment information, carrying out iterative evolution on a reward function and a strategy model for guiding robot skill learning by adopting a preset reward-strategy coevolution strategy, and the target strategy model obtained after evolution is completed is used for controlling the robot to execute the target task in the task information, so that the skill learning effect of the robot during high-dimensional task processing is remarkably enhanced while training data requirements are effectively reduced, and the task execution accuracy, precision and efficiency of the robot are improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of robot motion control, and in particular relates to a robot skill learning method, device, equipment and storage medium. Background Art

[0002] Robot skill learning refers to the process of enabling robots to autonomously master and perform various complex tasks through algorithms and models, usually involving motion control, path planning, and action coordination, etc., and the existing reinforcement learning (RL) methods have shown certain limitations in robot skill learning (such as training robot dogs to move). First, traditional RL methods usually need to train policy models from scratch, which is not only time-consuming and computationally expensive, but also performs poorly when dealing with high-dimensional robot motion control tasks, especially when it comes to complex robot joint motion coordination and diverse task goals. It is extremely challenging to design an effective reward function, which leads to low optimization efficiency of the policy model. In addition, the existing methods of automatically generating reward functions using large language models (LLMs) can generate reward functions through natural language task descriptions, but they need to train policy models from scratch in each policy optimization, which greatly increases the computational cost and complexity. These methods usually rely on a lot of trial and error and large-scale search, resulting in huge training time and resource consumption in practical applications, and cannot effectively cope with complex robot task requirements. Summary of the invention

[0003] The purpose of the present invention is to provide a robot skill learning method, device, equipment and storage medium, aiming to solve the problems of high learning cost, low learning efficiency and poor effect of robot skill learning caused by the existing technology.

[0004] In one aspect, the present invention provides a method for learning robot skills, the method comprising the following steps:

[0005] Based on task information and environment information, a preset reward-strategy co-evolution strategy is used to iteratively evolve the reward function and strategy model used to guide the robot's skill acquisition;

[0006] The target strategy model obtained after the evolution is completed is used to control the robot to execute the target task in the task information.

[0007] Preferably, based on the task information and the environment information, the step of iteratively evolving the reward function and the strategy model for guiding the robot skill acquisition using a preset reward-strategy co-evolution strategy includes:

[0008] Based on the task information, the environment information and the prompt information, generating a plurality of reward functions through a large language model;

[0009] Based on the pre-built policy model, a preset model generation strategy is used to generate a corresponding policy training model for each of the reward functions;

[0010] Performing reinforcement learning training on the corresponding strategy training model according to the reward function to obtain scores corresponding to each reward function;

[0011] Determining a global optimal reward function and a global optimal strategy model according to the scores corresponding to each of the reward functions;

[0012] Determine whether the evolution end condition is met, if yes, end the evolution and use the global optimal policy model as the target policy model, otherwise, use the global optimal reward function and the score corresponding to the global optimal reward function as the prompt information for the next round of evolution, and jump to the step of generating several reward functions through a large language model based on the task information, the environmental information and the prompt information.

[0013] Preferably, the step of using a preset model generation strategy to generate a corresponding strategy training model for each reward function includes:

[0014] When the current evolution is the first round of evolution, the model parameters of the strategy model are randomly initialized to obtain a first random strategy model, and the first random strategy model is used as the strategy training model.

[0015] Preferably, the step of using a preset model generation strategy to generate a corresponding strategy training model for each reward function further includes:

[0016] When the current evolution is not the first round of evolution, randomly initializing the model parameters of the strategy model to obtain a second random strategy model;

[0017] The second random strategy model is weightedly fused with the global optimal strategy model to obtain the strategy training model.

[0018] Preferably, the step of performing parameter weighted fusion of the second random strategy model and the global optimal strategy model comprises:

[0019] Using Bayesian optimization based on Gaussian process to search for the corresponding optimal fusion ratio for each reward function;

[0020] Based on the optimal fusion ratio, the second random strategy model and the global optimal strategy model are parameter-weighted fused to obtain the strategy training model corresponding to each reward function.

[0021] Preferably, the step of searching for the corresponding optimal fusion ratio for each reward function by using Bayesian optimization based on Gaussian process includes:

[0022] Construct a corresponding objective function s(α)=V for Bayesian optimization for each reward function R (R,θ f (α),T BO ), where s(α) represents the strategy performance score under a given fusion ratio α, θ f (α) is the fusion strategy model constructed using α as the reward function R, V R () is used to evaluate θ f (α) in R and training period T BO a function of performance or score under conditions;

[0023] A Bayesian optimization of the Gaussian process is performed on each of the objective functions to obtain a fusion ratio that maximizes each of the objective functions, and the fusion ratio is used as the optimal fusion ratio of the corresponding reward function.

[0024] Preferably, based on the optimal fusion ratio, the step of performing parameter weighted fusion of the second random strategy model and the global optimal strategy model comprises:

[0025] According to the fusion strategy model θ f (α) = α·θ best +(1-α)·θ r and the optimal fusion ratio, performing parameter weighted fusion of the second random strategy model and the global optimal strategy model, and taking the fused θ f (α) as the strategy training model corresponding to each of the reward functions, where θ best is the global optimal strategy model, θ r is the second random strategy model.

[0026] In another aspect, the present invention provides a robot skill learning device, the device comprising:

[0027] A strategy evolution unit, which is used to iteratively evolve the reward function and strategy model used to guide the robot's skill acquisition based on task information and environment information using a preset reward-strategy co-evolution strategy;

[0028] The robot control unit is used to control the robot to execute the target task in the task information by using the target strategy model obtained after the evolution is completed.

[0029] On the other hand, the present invention also provides an intelligent device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps described in the above-mentioned robot skill learning method when executing the computer program.

[0030] On the other hand, the present invention also provides a computer-readable storage medium, which stores a computer program, and when the computer program is executed by a processor, it implements the steps described in the above-mentioned robot skill learning method.

[0031] Based on task information and environmental information, the present invention adopts a preset reward-strategy co-evolution strategy to iteratively evolve the reward function and strategy model used to guide the robot's skill acquisition, and uses the target strategy model obtained after the evolution to control the robot to execute the target task in the task information, thereby effectively reducing the demand for training data while significantly enhancing the robot's skill learning effect when processing high-dimensional tasks, thereby improving the accuracy, precision and efficiency of the robot's task execution. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 is a flowchart of the implementation of the robot skill learning method provided in the first embodiment of the present invention;

[0033] Figure 2 is a flowchart of the implementation of the robot skill learning method provided in the second embodiment of the present invention;

[0034] Figure 3 is a schematic diagram of the structure of a robot skill learning device provided in Embodiment 3 of the present invention;

[0035] Figure 4 This is a schematic diagram of a preferred structure of a robot skill learning device provided in Embodiment 3 of the present invention;

[0036] Figure 5 It is a structural diagram of the intelligent body device provided in Example 4 of the present invention. DETAILED DESCRIPTION

[0037] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0038] It should be understood that the terms "first", "second", etc. in the present invention are used to distinguish between identical or similar items with substantially the same effects and functions. It should be understood that there is no logical or temporal dependency between "first", "second", and "nth", nor is there any limitation on quantity and execution order, unless otherwise specified.

[0039] It should be understood that the terms "including" and "having" and any variations thereof in the present invention are intended to cover but not exclude inclusion. For example, a product or device comprising a series of components is not necessarily limited to those components explicitly listed, but may include other components not explicitly listed or inherent to these products or devices.

[0040] It should be understood that the model structures of the strategy model, strategy training model, target strategy model, first random strategy model and second random strategy model of the present invention are all the same.

[0041] The specific implementation of the present invention is described in detail below in conjunction with specific embodiments:

[0042] Embodiment 1:

[0043] Figure 1 The implementation process of the robot skill learning method provided in the first embodiment of the present invention is shown. For the convenience of explanation, only the part related to the embodiment of the present invention is shown, which is described in detail as follows:

[0044] In step S101, based on task information and environment information, a preset reward-strategy co-evolution strategy is used to iteratively evolve the reward function and strategy model used to guide robot skill acquisition.

[0045] The embodiments of the present invention are applicable to intelligent devices that can perceive the environment, make decisions and take actions, such as robots, drones, etc. In the embodiments of the present invention, the task information and the environment information respectively describe the task that the robot needs to complete and the environment in which the robot performs the task. The task information includes but is not limited to the target task and the expected behavior. The environment information includes but is not limited to the state space, the action space, the possible interaction mode, and the dynamic characteristics of the environment. This information is crucial for generating a reward function that matches the task and the environment. The strategy model refers to the model used by the robot to decide what action (or action, behavior) to take in a given state. The reward function is used to evaluate the quality of the strategy adopted by the robot in performing the task. For example, after the robot performs an action in a given environment, the environment will give a corresponding reward value according to the reward function. This reward value reflects the quality of the robot's current action. The preset reward-strategy co-evolution strategy is a reward function and a strategy model. The mechanism of alternating optimization, here, based on task information and environmental information, a preset reward-strategy co-evolution strategy is used to iteratively evolve the reward function and strategy model used to guide the robot's skill acquisition, that is, in each round of evolution, the reward function and the strategy model are optimized alternately to form a co-evolution cycle. Specifically, in each round of evolution, first, according to the performance and feedback of the current strategy model, the reward function is adjusted and optimized to ensure that the reward function can accurately reflect the task that the robot needs to complete, and guide the robot to learn a strategy that meets human expectations. After the reward function is optimized, the strategy model is trained and optimized based on the new reward function. In each round of evolution, the reward function and the strategy model depend on each other, influence each other, and evolve together. This co-evolution process continues until the preset evolution end conditions are met.

[0046] In a feasible embodiment, based on task information and environmental information, a preset reward-strategy co-evolution strategy is used to iteratively evolve the reward function and strategy model used to guide robot skill acquisition. The specific implementation method is detailed in the description of the following embodiment 2 and will not be repeated here.

[0047] In step S102, the target strategy model obtained after the evolution is completed is used to control the robot to execute the target task in the task information.

[0048] In an embodiment of the present invention, the model with the best performance is selected from multiple evolved strategy models as the target strategy model. The target strategy model can accurately map the environment state to the corresponding robot control action. When the target strategy model obtained after the evolution is used to control the robot to perform the target task in the task information, the robot collects the state information of the current environment, and then inputs the state information into the target strategy model to obtain the corresponding control action. The robot executes these control actions to achieve the target task.

[0049] In an embodiment of the present invention, based on task information and environmental information, a preset reward-strategy co-evolution strategy is adopted to iteratively evolve the reward function and strategy model used to guide the robot's skill acquisition, and the target strategy model obtained after the evolution is completed is used to control the robot to perform the target task in the task information, thereby effectively reducing the demand for training data while significantly enhancing the robot's skill learning effect when processing high-dimensional tasks, thereby improving the accuracy, precision and efficiency of the robot's task execution.

[0050] Embodiment 2:

[0051] Figure 2 The implementation process of the robot skill learning method provided by the second embodiment of the present invention is shown. For the convenience of explanation, only the part related to the embodiment of the present invention is shown, which is described in detail as follows:

[0052] In step S201, several reward functions are generated through a large language model based on task information, environment information and prompt information.

[0053] In an embodiment of the present invention, task information, environmental information, and prompt information are all provided in the form of natural language so that the large language model can understand and generate a reward function related to the task and environment, wherein the prompt information is used to guide the large language model to generate a better reward function while improving the evolution speed and quality of the reward function to ensure that the reward function can always effectively guide the strategy improvement and effectively improve the performance of the strategy model in complex tasks. Here, the task information, environmental information, and prompt information are input into the large language model, and the large language model generates several reward functions related to the task and environment, each of which is based on the task goal and environmental constraints, and provides a reward standard suitable for the task, wherein several refers to 2 or more.

[0054] In a feasible embodiment, the prompt information includes the best performing reward function in the last round of evolution (i.e., the global optimal reward function) and the score obtained by evaluating the reward function under the corresponding strategy. Here, the generation of the reward function is represented as Among them, n is the current evolution round, I d is the task information, I e For environmental information, is the reward function with the best performance in the last round of evolution, and V(θ) is the reward function evaluated according to strategy θ. The score, The kth reward function generated by the large language model in the nth round of evolution is realized by combining the reinforcement learning task objectives, so that the large language model can automatically generate reward functions optimized for tasks (especially complex tasks), reducing the complexity of manually designing reward functions and the high labor cost of relying on human experts to design reward functions.

[0055] In another feasible embodiment, at the beginning of the first round of evolution, there is no prompt information for the large language model to refer to, that is, there is no global optimal reward function. Therefore, it is necessary to let the large language model generate zero samples based on the input task information and environment information to generate several reward functions. These reward functions cover different possibilities and strategies. Since these reward functions are generated entirely based on the internal knowledge of the large language model and the understanding of the task environment, the quality of the reward function may be poor. Starting from the second round, the global optimal reward function already exists. Therefore, for reward evolution, the large language model already has a reference sample at this time, and the current global optimal reward function can be used as prompt information for a new round of reward function evolution. In each round of evolution, the large language model will generate a new reward function, thereby introducing a dynamic reward generation mechanism based on LLM, which improves the adaptability and generalization ability of the robot in complex tasks, and can adaptively generate the optimal reward function according to different tasks and environments, ensuring that the reward function can always effectively guide strategy improvement, significantly improving the accuracy and efficiency of task execution.

[0056] In step S202, based on the pre-built policy model, a preset model generation strategy is adopted to generate a corresponding policy training model for each reward function.

[0057] In an embodiment of the present invention, the policy model may be a neural network constructed using a deep learning framework, or a decision tree or other types of models. The model structure of the policy model is not specifically limited herein.

[0058] In a feasible embodiment, when the current evolution is the first round of evolution, the model parameters of the policy model are randomly initialized to obtain a first random policy model, and the first random policy model is used as the policy training model.

[0059] In an embodiment of the present invention, in the first round of evolution, since there is no global optimal policy model, the model parameters of the policy model are randomly initialized, and each reward function corresponds to a randomly initialized policy model. For the convenience of description and distinction, the randomly initialized policy model is called the first random policy model, and the first random policy model corresponding to each reward function is used as the policy training model for the first round of evolution.

[0060] In another feasible embodiment, when the current evolution is not the first round of evolution, the model parameters of the strategy model are randomly initialized to obtain a second random strategy model, and the second random strategy model is weightedly fused with the global optimal strategy model to obtain a strategy training model.

[0061] In an embodiment of the present invention, starting from the second round of evolution, a global optimal policy model (i.e., the policy model with the best performance in the historical evolution) already exists. At this time, for a new batch of reward functions generated by the large language model, the global optimal policy model is weightedly fused with the randomly initialized policy model, and the fused model is used as the policy training model for each reward function. That is, for each reward function, a policy model is first randomly initialized. For the sake of ease of description and distinction, this model is called the second random policy model. The second random policy model is then weightedly fused with the global optimal policy model to obtain a policy training model.

[0062] In another feasible embodiment, the second random strategy model is weightedly fused with the global optimal strategy model by the following implementation:

[0063] (1) Using Bayesian optimization based on Gaussian process to search for the optimal fusion ratio for each reward function;

[0064] In an embodiment of the present invention, since different reward functions need to search for different fusion ratios to optimize their performance, it is impossible to set a universal fixed value so that each reward function uses the same fusion ratio. Here, Bayesian optimization based on Gaussian process is adopted. By continuously iterating and evaluating the actual performance of the model in the task under different fusion ratios, it is gradually discovered which fusion ratios can bring higher scores, and the fusion ratio is gradually adjusted. Specifically, in each iteration, a fusion ratio is selected for evaluation, and the score corresponding to the fusion ratio is recorded. Then, these known data points (fusion ratio and score pairs) are used to guide the selection of the next iteration to find the fusion ratio that can bring the highest score. Through continuous iteration and evaluation, the optimal relationship between the actual fusion ratio and the score is gradually approached, and finally the optimal fusion ratio is found, so as to achieve the use of the least search computing power to find an optimal fusion ratio for each reward function, which not only improves the search efficiency, but also effectively reduces the training cost of searching for the optimal fusion ratio.

[0065] Preferably, the search for the optimal fusion ratio is achieved through the following steps:

[0066] ① Construct a corresponding objective function s(α)=V for Bayesian optimization for each reward function R (R,θ f (α),T BO), where s(α) represents the strategy performance score under a given fusion ratio α, R is the reward function currently searching for the fusion ratio, and θ f (α) is the fusion strategy model constructed using α as the reward function R, V R () is used to evaluate θ f (α) in R and training period T BO a function of performance or score under conditions;

[0067] ② Perform Bayesian optimization of Gaussian process on each objective function to obtain the fusion ratio that maximizes each objective function, and use this fusion ratio as the optimal fusion ratio of the corresponding reward function.

[0068] In the embodiment of the present invention, the fusion strategy model θ f (α) is the new strategy model obtained by fusing the global optimal strategy model and the second random strategy model using α. α controls the degree of fusion of the new strategy model with the global optimal strategy model and the second random strategy model. Specifically, θ f (α) = α·θ best +(1-α)·θ r , where θ best is the global optimal strategy model, θ r For the second random strategy model, steps ① and ② are used to partially inherit the historical optimal strategy, fuse the historical optimal strategy with the random strategy, and use Bayesian optimization to quickly adjust the strategy fusion ratio. Compared with the traditional strategy optimization method that usually relies on training strategies from scratch, the strategy optimization scheme of this application effectively reduces the consumption of invalid training samples and improves the efficiency of strategy optimization, especially in high-dimensional tasks.

[0069] (2) Based on the optimal fusion ratio, the second random strategy model is weightedly fused with the global optimal strategy model to obtain the strategy training model corresponding to each reward function.

[0070] In the embodiment of the present invention, after searching for the best fusion ratio α best After that, according to the fusion strategy model θ f (α) = α·θ best +(1-α)·θ r , using α best The second random strategy model is weightedly fused with the global optimal strategy model, that is, θ f (α best )=α best ·θ best +(1-α best )·θ r , the fused θ f (α best) is used as the strategy training model corresponding to each reward function, so that the new strategy model retains the advantages of the historical optimal strategy and has the ability to explore new strategy space. While inheriting the past experience, the model retains a certain plasticity, effectively improving the diversity of strategy evolution.

[0071] Through the above steps (1) and (2), the second random strategy model and the global optimal strategy model are weightedly fused, thereby accelerating the strategy parameter search through Bayesian optimization, reducing the training sample consumption of the search fusion ratio α, and effectively improving the training efficiency.

[0072] In step S203, reinforcement learning training is performed on the corresponding strategy training model according to the reward function to obtain the score corresponding to each reward function.

[0073] In an embodiment of the present invention, each reward function is used to train the corresponding strategy training model. During the reinforcement learning training process, each strategy training model attempts to learn how to make the best decision under a given corresponding reward function to find the behavior strategy that maximizes the reward function. By evaluating the performance of each strategy training model in the environment (such as accumulated rewards), a score is calculated for each corresponding reward function.

[0074] In a feasible embodiment, a training cycle is set. During the training cycle, each policy training model will start from the initial parameters. Through the reinforcement learning algorithm, each policy training model will undergo multiple iterations of training. In each iteration, the policy training model will continue to learn and update its parameters according to the reward function, so that the robot will continue to try different strategies to maximize the cumulative reward and accumulate the score of the reward function obtained from each iteration of training. This score reflects the performance of the policy training model under the current strategy. The process of obtaining the score corresponding to the reward function can be expressed as Among them, T max is the length of the training cycle, that is, the number of iterations that each policy training model receives during the evaluation period, θ is the policy training model corresponding to the kth reward function in the (n-1)th round of evolution, and θ 0 Initial parameters for training the model for this strategy.

[0075] In step S204, the global optimal reward function and the global optimal strategy model are determined according to the scores corresponding to the various reward functions.

[0076] In this embodiment of the present invention, among all the trained strategy training models, the scores of all reward functions are compared, and the reward function with the highest score is selected as the optimal reward function for the current evolution round, that is, in, is the optimal reward function of the current evolution round (i.e. the (n-1)th round). Correspondingly, the strategy training model trained by the reward function is used as the optimal strategy model of the current evolution round, and the optimal reward function and optimal strategy model of the current evolution round are used as the global optimal reward function and the global optimal strategy model for the next round of evolution.

[0077] In a feasible embodiment, the scores of the optimal reward function of the current round of evolution and the global optimal reward function obtained in the historical evolution are compared. When the score of the optimal reward function of the current round of evolution does not exceed the score of the global optimal reward function obtained in the historical evolution, the global optimal reward function obtained in the historical evolution is maintained unchanged, so that the global optimal reward function will not be overwritten by a round of failed reward function exploration, avoiding a vicious circle and ensuring that the evolution of the reward function always develops in a good direction to obtain a better reward function. A high-quality reward function can effectively improve the performance of the policy model in complex tasks.

[0078] In step S205, when the evolution end condition is reached, the evolution ends and the global optimal policy model is used as the target policy model.

[0079] In step S206, when the evolution end condition is not met, the global optimal reward function and the score corresponding to the global optimal reward function are used as prompt information for the next round of evolution, and jump to step S201.

[0080] In an embodiment of the present invention, it is determined whether the evolution termination condition is met. If so, the evolution is terminated, and the global optimal policy model is used as the target policy model to control the robot's movements. Otherwise, the global optimal reward function and the score corresponding to the global optimal reward function are used as prompt information for the next round of evolution to optimize the reward function. The global optimal policy model is used for policy optimization in the next round of evolution, and the process jumps to step S201 to continue execution. The evolution termination condition may be reaching a predetermined number of evolutions, the performance improvement of the policy model is no longer significant, or specific performance indicators are met, etc., which are not specifically limited here.

[0081] In an embodiment of the present invention, a large language model is first used to generate a variety of reward functions, and then the optimal reward function is dynamically screened according to the robot's task performance, thereby improving the task learning efficiency. In strategy optimization, by fusing random strategies with historical optimal strategies and using the Bayesian optimization method to adjust the strategy fusion parameters, the optimal strategy combination is quickly found. The co-evolution mechanism of the reward function and strategy of the present application can gradually optimize the robot's skill learning process in a complex environment, avoid the high cost of training the strategy model from scratch, greatly reduce the demand for training data, and significantly improve the learning speed and performance accuracy, thereby achieving efficient and low-cost robot task learning and execution.

[0082] Embodiment three:

[0083] Figure 3 The structure of the robot skill learning device provided by the third embodiment of the present invention is shown. For the convenience of description, only the parts related to the embodiment of the present invention are shown, including:

[0084] A strategy evolution unit 31 is used to iteratively evolve a reward function and a strategy model for guiding the robot's skill acquisition based on task information and environment information using a preset reward-strategy co-evolution strategy;

[0085] The robot control unit 32 is used to control the robot to execute the target task in the task information using the target strategy model obtained after the evolution is completed.

[0086] Among them, Figure 4 As shown, preferably, the strategy evolution unit 31 includes:

[0087] A reward function generating unit 311, used to generate a plurality of reward functions through a large language model based on task information, environment information and prompt information;

[0088] A training model generation unit 312 is used to generate a corresponding strategy training model for each reward function based on a pre-built strategy model and using a preset model generation strategy;

[0089] The model reinforcement training unit 313 is used to perform reinforcement learning training on the corresponding strategy training model according to the reward function to obtain the score corresponding to each reward function;

[0090] A global optimal determination unit 314, used to determine a global optimal reward function and a global optimal strategy model according to the scores corresponding to each reward function;

[0091] The collaborative evolution ending unit 315 is used to end the evolution when the evolution ending condition is reached, and use the global optimal strategy model as the target strategy model.

[0092] The prompt information setting unit 316 is used to use the global optimal reward function and the score corresponding to the global optimal reward function as prompt information for the next round of evolution when the evolution end condition is not met, and trigger the reward function generation unit 311 to generate several reward functions through the large language model based on the task information, environmental information and prompt information.

[0093] Preferably, the training model generating unit 312 includes:

[0094] The first random initialization unit is used to randomly initialize the model parameters of the strategy model when the current evolution is the first round of evolution, to obtain a first random strategy model, and to use the first random strategy model as a strategy training model.

[0095] Preferably, the training model generating unit 312 further includes:

[0096] A second random initialization unit, used for randomly initializing the model parameters of the strategy model when the current evolution is not the first round of evolution, to obtain a second random strategy model;

[0097] The model parameter fusion unit is used to perform parameter weighted fusion of the second random strategy model and the global optimal strategy model to obtain a strategy training model.

[0098] Preferably, the model parameter fusion unit includes:

[0099] A fusion ratio search unit is used to search for the optimal fusion ratio corresponding to each reward function using Bayesian optimization based on Gaussian process;

[0100] The parameter fusion subunit is used to perform parameter weighted fusion of the second random strategy model and the global optimal strategy model based on the optimal fusion ratio to obtain the strategy training model corresponding to each reward function.

[0101] Preferably, the fusion ratio search unit comprises:

[0102] The objective function construction unit is used to construct a corresponding objective function s(α)=V for Bayesian optimization for each reward function. R (R,θ f (α),T BO ), where s(α) represents the strategy performance score under a given fusion ratio α, θ f (α) is the fusion strategy model constructed using α as the reward function R, V R () is used to evaluate θ f (α) in R and training period T BO a function of performance or score under conditions;

[0103] The ratio search subunit is used to perform Bayesian optimization of Gaussian process on each objective function to obtain the fusion ratio that maximizes each objective function, and use the fusion ratio as the optimal fusion ratio of the corresponding reward function.

[0104] Preferably, the parameter fusion subunit includes:

[0105] The training model obtains the unit, which is used to model the fusion strategy θ f (α) = α·θ best +(1-α)·θ rand the best fusion ratio, the second random strategy model is weightedly fused with the global optimal strategy model, and the fused θ f (α) is the strategy training model corresponding to each reward function, where θ best is the global optimal strategy model, θ r is the second random strategy model.

[0106] In the embodiment of the present invention, each unit of the robot skill learning device can be implemented by a corresponding hardware or software unit, and each unit can be an independent software or hardware unit, or can be integrated into a software or hardware unit, which is not intended to limit the present invention. Specifically, the implementation of each unit can refer to the description of the aforementioned embodiments 1 and 2, and will not be repeated here.

[0107] Embodiment 4:

[0108] Figure 5 The structure of the intelligent body device provided by the fourth embodiment of the present invention is shown. For the convenience of explanation, only the part related to the embodiment of the present invention is shown.

[0109] The intelligent agent device 5 of the embodiment of the present invention includes a processor 50, a memory 51, and a computer program 52 stored in the memory 51 and executable on the processor 50. When the processor 50 executes the computer program 52, the steps in the embodiment of the robot skill learning method are implemented, such as Figure 1 Alternatively, when the processor 50 executes the computer program 52, the functions of each unit in the above-mentioned device embodiments are realized, for example Figure 3 Function of the unit shown.

[0110] In an embodiment of the present invention, based on task information and environmental information, a preset reward-strategy co-evolution strategy is adopted to iteratively evolve the reward function and strategy model used to guide the robot's skill acquisition, and the target strategy model obtained after the evolution is completed is used to control the robot to perform the target task in the task information, thereby effectively reducing the demand for training data while significantly enhancing the robot's skill learning effect when processing high-dimensional tasks, thereby improving the accuracy, precision and efficiency of the robot's task execution.

[0111] The intelligent device of the embodiment of the present invention may be a robot. The steps implemented when the processor 50 in the intelligent device 5 executes the computer program 52 to implement the robot skill learning method can refer to the description of the aforementioned method embodiment, which will not be repeated here.

[0112] Embodiment five:

[0113] In an embodiment of the present invention, a computer-readable storage medium is provided, which stores a computer program. When the computer program is executed by a processor, the steps in the above-mentioned robot skill learning method embodiment are implemented, for example, Figure 1 Alternatively, when the computer program is executed by a processor, the functions of each unit in the above-mentioned device embodiments are realized, for example Figure 3 Function of the unit shown.

[0114] In an embodiment of the present invention, based on task information and environmental information, a preset reward-strategy co-evolution strategy is adopted to iteratively evolve the reward function and strategy model used to guide the robot's skill acquisition, and the target strategy model obtained after the evolution is completed is used to control the robot to perform the target task in the task information, thereby effectively reducing the demand for training data while significantly enhancing the robot's skill learning effect when processing high-dimensional tasks, thereby improving the accuracy, precision and efficiency of the robot's task execution.

[0115] The computer-readable storage medium of the embodiment of the present invention may include any entity or device or recording medium capable of carrying computer program code, for example, ROM / RAM, magnetic disk, optical disk, flash memory and other memories.

[0116] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A robot skill learning method, characterized in that: The method comprises the following steps: Based on task information and environment information, a preset reward-strategy co-evolution strategy is used to iteratively evolve the reward function and strategy model used to guide the robot's skill acquisition; The target strategy model obtained after the evolution is completed is used to control the robot to execute the target task in the task information.

2. The method according to claim 1, characterized in that Based on the task information and the environment information, the steps of iteratively evolving the reward function and the strategy model used to guide the robot's skill acquisition using the preset reward-strategy co-evolution strategy include: Based on the task information, the environment information and the prompt information, generating a plurality of reward functions through a large language model; Based on the pre-built policy model, a preset model generation strategy is used to generate a corresponding policy training model for each of the reward functions; Performing reinforcement learning training on the corresponding strategy training model according to the reward function to obtain scores corresponding to each reward function; Determining a global optimal reward function and a global optimal strategy model according to the scores corresponding to each of the reward functions; Determine whether the evolution end condition is met, if yes, end the evolution and use the global optimal policy model as the target policy model, otherwise, use the global optimal reward function and the score corresponding to the global optimal reward function as the prompt information for the next round of evolution, and jump to the step of generating several reward functions through a large language model based on the task information, the environmental information and the prompt information.

3. The method according to claim 2, characterized in that The step of using a preset model generation strategy to generate a corresponding strategy training model for each of the reward functions includes: When the current evolution is the first round of evolution, the model parameters of the strategy model are randomly initialized to obtain a first random strategy model, and the first random strategy model is used as the strategy training model.

4. The method according to claim 2, characterized in that The step of using a preset model generation strategy to generate a corresponding strategy training model for each of the reward functions also includes: When the current evolution is not the first round of evolution, randomly initializing the model parameters of the strategy model to obtain a second random strategy model; The second random strategy model is weightedly fused with the global optimal strategy model to obtain the strategy training model.

5. The method according to claim 4, characterized in that The step of performing parameter weighted fusion of the second random strategy model and the global optimal strategy model comprises: Using Bayesian optimization based on Gaussian process to search for the corresponding optimal fusion ratio for each reward function; Based on the optimal fusion ratio, the second random strategy model and the global optimal strategy model are parameter-weighted fused to obtain the strategy training model corresponding to each reward function.

6. The method according to claim 5, characterized in that The step of searching for the corresponding optimal fusion ratio for each reward function by using Bayesian optimization based on Gaussian process includes: Construct a corresponding objective function s(α)=V for Bayesian optimization for each reward function R (R,θ f (α),T BO ), where s(α) represents the strategy performance score under a given fusion ratio α, θ f (α) is the fusion strategy model constructed using α as the reward function R, V R () is used to evaluate θ f (α) in R and training period T BO a function of performance or score under conditions; A Bayesian optimization of Gaussian process is performed on each objective function to obtain a fusion ratio that maximizes each objective function, and the fusion ratio is used as the optimal fusion ratio of the corresponding reward function.

7. The method according to claim 6, characterized in that Based on the optimal fusion ratio, the step of performing parameter weighted fusion of the second random strategy model and the global optimal strategy model comprises: According to the fusion strategy model θ f (α) = α·θ best +(1-α)·θ r and the optimal fusion ratio, performing parameter weighted fusion of the second random strategy model and the global optimal strategy model, and taking the fused θ f (α) as the strategy training model corresponding to each of the reward functions, where θ best is the global optimal strategy model, θ r is the second random strategy model.

8. A robot skill learning device, characterized in that: The device comprises: A strategy evolution unit, which is used to iteratively evolve the reward function and strategy model used to guide the robot's skill acquisition based on task information and environment information using a preset reward-strategy co-evolution strategy; The robot control unit is used to control the robot to execute the target task in the task information by using the target strategy model obtained after the evolution is completed.

9. An intelligent agent device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images