Model-based robot operation skill parameter learning method

Abstract

This invention discloses a model-based method for learning robot operation skill parameters, including a single-skill policy learning module and a task-skill parameter learning module. The single-skill policy learning module includes an object-oriented representation model and a policy model conditioned on semantic goals. The task-skill parameter learning module includes an implicit state transition model learning module and an online skill parameter planning module. The single semantic skill policy learning module of this invention combines representation learning and reinforcement learning, integrating the reasoning transformation from visual input to the logical representation of the current system state with task-oriented robot policy learning to generate action parameters that satisfy the target logical state. Using reinforcement learning avoids extensive model design and processing, as well as the need for expert teaching data. Furthermore, by predefining the action sequences of skills based on an operation knowledge base, the search space for actions is reduced, lowering the exploration burden on the agent.

Description

Technical Field

[0001] This invention relates to representation learning technology, reinforcement learning technology, and robot skill planning technology. More specifically, it relates to a model-based method for learning robot operation skill parameters. Background Technology

[0002] With the development of various technologies, robots are widely used in fields such as home service and industrial production, learning operational skills such as serving tea and water and assembling mechanical parts to help people complete a variety of complex tasks. However, traditional methods such as manual programming and teach-in programming have problems such as low efficiency, high cost, and poor autonomy, making it difficult to meet the diverse needs of complex scenarios and greatly limiting the widespread application of robots. Therefore, higher requirements are placed on the autonomy and generalization of robot operating skills.

[0003] Multi-step planning techniques for long-sequence tasks do not require manual programming, detailed model specifications, or task-specific manual processing. Instead, they only require determining the target form and the set of executable primitive skills based on the task content. The robot can then select appropriate skill sequences and determine effective motion parameters through planning methods, and generate corresponding motion trajectories via the underlying controller. However, this method often requires specific knowledge of environmental states and dynamics, including determining the prerequisites and effects of different primitive skills, which are typically difficult to obtain in real-world environments. Furthermore, the completion status of the task often needs to be manually determined by the user using predefined mapping relationships or manual annotation, resulting in extremely low efficiency.

[0004] Model-based reinforcement learning is an important development direction that can meet these requirements. The study of model-based reinforcement learning focuses on how to use collected environmental interaction data to learn dynamic models of environmental states, so as to conveniently and effectively represent the influence of different robot primitive skills on environmental states, and better select appropriate skill sequences and accurate action parameters to complete the target task.

[0005] Meanwhile, current representation methods can perform detailed symbolic representation of environmental image information, generating corresponding symbolic states that effectively reflect the execution progress of the current task, thereby automatically determining the completion status of the target operation task. Furthermore, this method can be transferred to other similar operation scenarios, significantly reducing the user's burden.

[0006] Combining model-based reinforcement learning methods with skill planning techniques, along with effective environmental representation methods, can not only reduce cumbersome manual processing and avoid over-reliance on environment-specific knowledge, but also greatly improve the robot's performance and generalization of long-sequence tasks. Summary of the Invention

[0007] To overcome the shortcomings of existing technologies, the present invention aims to provide a model-based method for learning robot operation skill parameters, enabling the generation of motion parameters for robots in long-sequence operation tasks. This invention is achieved through the following technical solution:

[0008] This invention discloses a model-based method for learning robot operation skill parameters, including a strategy learning module for a single skill and a parameter learning module for a task skill;

[0009] The single-skill strategy learning module includes an object-oriented representation model and a policy model conditioned on semantic objectives.

[0010] The parameter learning module for task skills includes an implicit state transition model learning module and an online skill parameter planning module;

[0011] The single-skill policy learning module can train multiple policy models for single semantic skills, which serve as a pre-trained prior module to support the parameter learning module for subsequent task skills.

[0012] As a further improvement, the single-skill policy learning module of the present invention uses representation learning to represent the spatial relationships of objects in the environment, trains an object-oriented representation model, and uses the encoder in the representation model to encode environmental observations. The agent is trained using reinforcement learning methods so that it can generate action parameters for the corresponding task based on a single semantic skill objective given by the user, and obtains multiple policy models.

[0013] The parameter learning module for task skills employs supervised learning to train a state transition model in the implicit space. This implicit space is obtained based on the encoder of the representation model, representing the impact of different skill action parameters on the environmental state. The implicit state transition model is used to predict the environmental state after the action parameters are executed in multiple steps. The decoder in the representation model is used to decode the predicted implicit state. The decoded symbolic state is used as the planning target, and a planning method is employed to plan the appropriate action parameters for each step in the given skill sequence, thereby achieving the semantic goals specified by the user in sequence.

[0014] As a further improvement, the present invention includes:

[0015] 1) Object-oriented representation model, which performs semantic information representation of environmental observations and infers the spatial relationships of objects corresponding to the current environmental image;

[0016] 2) A policy model conditioned on semantic goals takes the implicit features of the environmental image obtained through the representation model and the semantic goals specified by the user as input to generate action parameters for the corresponding semantic skills until the semantic information corresponding to the current environmental observation matches the semantic goals.

[0017] 3) Implicit state transition equation learning module, used to learn the state transition model of environmental observation in the implicit space of the representation model from the training dataset, and predict the impact of different primitive skills on the environmental state;

[0018] 4) The online planning module for skill parameters samples and generates multiple action parameters based on the policy model with semantic goals as conditions for the corresponding semantic skills, constructs a parameter candidate set, predicts future states based on the implicit state transition model, and selects action parameters that can achieve semantic goals in sequence using planning methods.

[0019] As a further improvement, the object-oriented representation model training method of the present invention includes the following steps:

[0020] 1) For target operation scenarios, randomize primitive skill types and action parameters to enable the robot to interact with the environment and collect image data for each frame and corresponding label information of object spatial relationships;

[0021] 2) Constructing an object-oriented representation network model ,in For the network parameters, an encoder-decoder architecture is adopted. The encoder uses a Transformer network architecture, taking as input an RGB image of the scene observation and images of each manipulated object from a fixed camera viewpoint, mapping image information to an implicit space. The decoder uses a two-layer fully connected network, each layer containing 512 neurons. Each neuron uses the GELU function as the activation function to extract the spatial relationships and predicate information of objects in the image. This representation model is used for learning to obtain the corresponding network model parameters. The loss function for the object-oriented representation network is defined as shown in Equation 1):

[0022]

[0023] in It is the network's predicted output. It's a real label.

[0024] As a further improvement, the policy model conditioned on semantic objectives described in this invention includes the following steps:

[0025] 1) Design a basic action library based on the target operation scenario, and select appropriate action types from it to form corresponding action sequences for each semantic skill, including basic actions such as end-effector movement, opening the gripper, and closing the gripper;

[0026] 2) Randomly initialize the joint angles of the robotic arm and the poses of each object in the scene in the simulation environment;

[0027] 3) Use a fully connected layer as the policy network, whose input is defined as... The policy network includes latent image representations and semantic objectives. It has two output branches, one of which acts as the executor for reinforcement learning, outputting the next action the simulation environment needs to perform. The action content includes the Cartesian space of the robotic arm's end effector. Another branch acts as the evaluator for reinforcement learning, outputting a value function. ,in The fully connected layer representing the policy network. This represents the desired position and orientation of the robotic arm's end effector.

[0028] 4) In the policy network simulation, RGB images of the desktop scene are acquired through a camera, and the latent representation of the scene image is obtained using the encoder in the object-oriented representation model. Then the potential representation and semantic goals As input to the policy network, the output includes the appropriate action type and parameters in the action sequence. Execute actions in the simulation environment and return the reward function. The reward function is defined as follows:

[0029]

[0030] in This indicates that the semantic information corresponding to the current environmental observation matches the semantic target specified by the user.

[0031] 5) Record the execution trajectory of the robotic arm. ,Record The trajectory is used to complete the training data collection. The collected data is used to train the network parameters of the policy. Based on backpropagation, the gradient of the network parameters is calculated, and then the gradient descent is used to update the parameter gradient.

[0032] 6) Repeat steps 2)-5) until the network converges, i.e., the simulation environment has the highest success rate in completing a single semantic skill.

[0033] As a further improvement, the implicit state transition equation learning module of the present invention includes the following steps:

[0034] 1) Randomize the initial state of the task scenario, randomize the primitive skill type and action parameters, collect data during the robot's interaction with the environment, and collect the result image and corresponding skill type and action parameters after each skill execution, using a single primitive skill as the segmentation scale.

[0035] 2) Using fully connected layers as the implicit state transition model network Its input is defined as This includes the latent representation of the image, the skill type of the semantic skill, and the action parameters, outputting the latent representation of the scene image after the robot performs the corresponding action. ;

[0036] 3) Using the image data collected in step 1), the latent representation of each frame of the image is obtained through the encoder of the object representation network. and skill type Action parameters Both serve as inputs to the network to predict the latent representation of the scene image after skill execution. The loss function of the prediction model is shown in Equation 3):

[0037]

[0038] Among them For the network's predicted output, It is the potential representation of a real image after being processed by an encoder.

[0039] As a further improvement, the online skill parameter planning module of the present invention includes the following steps:

[0040] 1) Parse the target semantic information of multi-step tasks to obtain the skill sequence and semantic target sequence of multi-step operations;

[0041] 2) Based on the operation sequence and semantic target sequence obtained in step 1), input the RGB image of the task scene, call the policy network of the corresponding single semantic skill, sample and generate multiple feasible action parameters, and construct the corresponding parameter candidate set;

[0042] 3) Based on the candidate set of action parameters obtained in step 2), the implicit state transition model is used to predict the potential representation of the corresponding primitive skill after execution with each action parameter. The decoder in the object-oriented representation network is used to generate the semantic information corresponding to each potential representation. The matching of semantic information and corresponding semantic target is used as the evaluation index for action parameter selection.

[0043] 4) Based on the skill sequence in step 1), repeat steps 2) and 3) in a loop, using the CEM planning method and MPC method to select action parameters that can complete the corresponding semantic goal from the action parameter set of the corresponding semantic skill, and achieve the final semantic goal.

[0044] Compared with existing technologies, the advantages of this invention are as follows:

[0045] 1) The single semantic skill policy learning module of this invention combines representation learning and reinforcement learning, integrating the inference transformation of visual input into the logical representation of the current system state with task-oriented robot policy learning to generate action parameters that satisfy the target logical state. This module allows users to instruct the robot to perform different operational semantic skills and specify the target object and target state, thereby generating specific action parameters to enable the system to reach the target logical state. Using representation learning, it is possible to extract relatively fine continuous spatial information representations, such as information on distance and direction between objects, and can generalize to task scenarios with different colors, backgrounds, and perspectives to infer corresponding logical representations. Using reinforcement learning, it avoids a large amount of model design and processing and the need for expert teaching data, and predefines the action sequence of skills based on the operational knowledge base, reducing the search space of actions and lowering the exploration burden of the agent.

[0046] 2) In the task skill parameter learning module described in this invention, based on multiple pre-trained single semantic skill strategy modules, preliminary inference of the operation task is achieved, and multiple action parameter samples obtained from them are placed into a candidate set. Combining the robot's initial environmental observation and semantic skill sequence, the system dynamic state is modeled and generalized through an implicit state transition model, inferring the influence of different primitive skills and different action parameters on the system state, predicting the future implicit state, judging the explicit semantic information of the implicit state through a decoder, and filtering from the candidate set in combination with the task semantic requirements. It can intelligently find the robot execution parameters that can be executed by the robot and meet all task semantic requirements from the action parameters generated by the single semantic skill strategy module, solving the problem of dependency between the execution of previous and subsequent skills, and greatly improving the robot's execution effect on long sequence operation tasks. Attached Figure Description

[0047] Figure 1 The flowchart shows the model-based robot skill parameter learning method of this invention.

[0048] Figure 2 This is a schematic diagram of the online planning module for skill parameters in the model-based robot skill parameter learning method of the present invention. Detailed Implementation

[0049] The technical solution of the present invention will be further described below with reference to the accompanying drawings and specific embodiments:

[0050] This invention discloses a model-based method for learning robot operation skill parameters, including a strategy learning module for a single skill and a parameter learning module for a task skill. Each part contains software algorithms built to achieve specific functions.

[0051] This invention discloses a model-based method for learning robot operation skill parameters, comprising the following steps:

[0052] 1. Build a simulation environment in PyBullet software, including a robotic arm, a camera, a desktop, and various manipulatory objects (blocks and tools of different shapes); the robotic arm is fixed at the origin of the world coordinate system, and the camera is fixed above the side of the desktop, facing the desktop at a 45-degree angle;

[0053] 2. A randomized simulation environment is used to collect image data and corresponding label data, and to train a unified representation model for objects.

[0054] 3. Conduct trial-and-error interaction with the simulation environment to collect training data to train policy networks for multiple single semantic skills;

[0055] 4. Randomly interact with the simulation environment to collect training data and train the implicit state transition model based on the pre-trained object-oriented representation model.

[0056] 5. Based on the above pre-training module, the action parameters for long sequence operation tasks are generated through the online skill parameter planning module;

[0057] 6. Construct an environment similar to that in the simulation in the physical world, collect physical data to fine-tune the above pre-trained model, and perform virtual-to-real transfer.

[0058] Based on the above technical solution, each step can be further implemented in the following specific ways.

[0059] The specific implementation steps of step 2 are as follows:

[0060] 2-1. For target operation scenarios, randomize primitive skill types and action parameters to enable the robot to interact with the environment and collect image data for each frame and corresponding label information of object spatial relationships;

[0061] 2-2. Constructing an object-oriented representation network model ,in For the network parameters, an encoder-decoder architecture is adopted. The encoder uses a Transformer network architecture, taking as input an RGB image of the scene observation and images of each manipulated object from a fixed camera viewpoint, mapping image information to an implicit space. The decoder uses a two-layer fully connected network, each layer containing 512 neurons. Each neuron uses the GELU function as the activation function to extract the spatial relationships and predicate information of objects in the image. This representation model is used for learning to obtain the corresponding network model parameters. The loss function for the object-oriented representation network is defined as shown in Equation 1):

[0062]

[0063] in It is the network's predicted output. These are real labels;

[0064] Furthermore, the specific implementation steps of step 3 are as follows:

[0065] 3-1. Design a basic action library based on the target operation scenario, and select appropriate action types from it to form corresponding action sequences for each semantic skill, including basic actions such as end-effector movement, opening the gripper, and closing the gripper;

[0066] 3-2. Randomly initialize the poses of the robotic arm end effector and each object in the scene in the simulation environment;

[0067] 3-3. In a simulation environment, acquire RGB images of the desktop scene using a camera, and obtain the latent representation of the scene image using the encoder in a pre-trained object-oriented representation model. Then the potential representation and semantic goals As input to the policy network, the output includes the appropriate action type and parameters in the action sequence. Execute actions in the simulation environment and return the reward function. The reward function is defined as follows:

[0068]

[0069] in This indicates that the semantic information corresponding to the current environmental observation matches the semantic target specified by the user.

[0070] 3-4. Record the execution trajectory of the robotic arm. ,Record The trajectory is used to complete the training data collection. The collected data is used to train the network parameters of the policy. Based on backpropagation, the gradient of the network parameters is calculated, and then the gradient descent is used to update the parameter gradient.

[0071] 3-5. Repeat steps 3-1 to 3-4 until the network converges, meaning the simulation environment achieves the highest success rate in completing the target semantic skill.

[0072] Furthermore, step 4 is specifically implemented as follows:

[0073] 4-1. Randomize the initial state of the task scenario, randomize the primitive skill type and action parameters, collect data during the robot's interaction with the environment, and collect the result image and corresponding skill type and action parameters after each skill execution, using a single primitive skill as the segmentation scale.

[0074] 4-2. Using fully connected layers as implicit state transition model networks Its input is defined as This includes the latent representation of the image, the skill type of the semantic skill, and the action parameters, outputting the latent representation of the scene image after the robot performs the corresponding action. ;

[0075] 4-3. Using the image data collected in step 1), obtain the latent representation of each frame of the image through the encoder of the object representation network. and skill type Action parameters Both serve as inputs to the network to predict the latent representation of the scene image after skill execution. The loss function of the prediction model is shown in Equation 3):

[0076]

[0077] Among them For the network's predicted output, It is the potential representation of a real image after being processed by an encoder.

[0078] The specific implementation steps of step 5 are as follows:

[0079] 5-1. Analyze the target semantic information of multi-step tasks to obtain the skill sequence and semantic target sequence of multi-step operations;

[0080] 5-2. Based on the operation sequence and semantic target sequence obtained in step 1), input the RGB image of the task scene, call the corresponding single semantic skill policy network, sample and generate multiple feasible action parameters, and construct the corresponding parameter candidate set;

[0081] 5-3. Based on the candidate set of action parameters obtained in step 2), the implicit state transition model is used to predict the potential representation of the corresponding primitive skill after execution with each action parameter. The decoder in the object-oriented representation network is used to generate the semantic information corresponding to each potential representation. The matching of semantic information and corresponding semantic target is used as the evaluation index for action parameter selection.

[0082] 5-4. Based on the skill sequence in step 1), repeat steps 2) and 3) in a loop, using the CEM planning method and MPC method to select action parameters that can complete the corresponding semantic goal from the action parameter set of the corresponding semantic skill, and achieve the final semantic goal.

[0083] Obviously, the above is not a limitation on the implementation method. For those skilled in the art, different variations can be made based on the above description, and these variations are still within the protection scope of this invention.

Claims

1. A model-based method for learning robot operation skill parameters, characterized in that, It includes a strategy learning module for single skills and a parameter learning module for task skills; The single-skill strategy learning module includes an object-oriented representation model and a policy model conditioned on semantic objectives. The parameter learning module for the task skill includes an implicit state transition model learning module and an online skill parameter planning module; The single-skill policy learning module can train multiple policy models for single semantic skills, which serve as a pre-trained prior module to support the parameter learning module for subsequent task skills. The single-skill policy learning module uses representation learning to represent the spatial relationships of objects in the environment, trains an object-oriented representation model, and uses the encoder in the representation model to encode environmental observations. It uses reinforcement learning to train the agent so that it can generate action parameters for the corresponding task based on the single semantic skill objective given by the user, and obtains multiple policy models. The parameter learning module for the task skill uses supervised learning to train a state transition model in the implicit space, where the implicit space is obtained based on the encoder of the representation model, representing the influence of different skill action parameters on the environmental state. The implicit state transition model is used to predict the environmental state after the execution of action parameters in multiple steps. The decoder in the representation model is used to decode the predicted implicit state. The decoded symbolic state is used as the planning target. The planning method is used to plan the appropriate action parameters for each step in the given skill sequence, which can realize the semantic goals specified by the user in sequence.

2. The model-based robot operation skill parameter learning method according to claim 1, characterized in that, 1) The object-oriented representation model described above performs semantic information representation of environmental observations and infers the spatial relationships of objects corresponding to the current environmental image. 2) The policy model based on semantic target takes the implicit features of the environmental image obtained by the representation model and the semantic target specified by the user as input to generate action parameters of the corresponding semantic skill until the semantic information corresponding to the current environmental observation matches the semantic target. 3) The implicit state transition equation learning module is used to learn the state transition model of environmental observation in the implicit space of the representation model from the training dataset, and to predict the impact of different primitive skills on the environmental state. 4) The skill parameter online planning module generates multiple action parameters based on the strategy model of the corresponding semantic skill with semantic goal as the condition, constructs a parameter candidate set, predicts future states based on implicit state transition model, and selects action parameters that can achieve semantic goals in sequence by using planning method.

3. The model-based robot operation skill parameter learning method according to claim 2, characterized in that, The object-oriented representation model training method includes the following steps: 1) For target operation scenarios, randomize primitive skill types and action parameters to enable the robot to interact with the environment and collect image data for each frame and corresponding label information of object spatial relationships; 2) Constructing an object-oriented representation network model ,in For network parameters, an encoder-decoder architecture is adopted, where the encoder uses the Transformer network architecture. The input includes an RGB image of the scene observation and images of each manipulated object from a fixed camera viewpoint, mapping the image information to the implicit space. The decoder employs a two-layer fully connected network, each containing 512 neurons. Each neuron uses the GELU function as its activation function to extract the spatial relationships and predicate information of objects in the image. This representation model is used for learning to obtain the corresponding network model parameters. The loss function for the object-oriented representation network is defined as shown in Equation 1): ； in It is the network's predicted output. It's a real label.

4. The model-based robot operation skill parameter learning method according to claim 3, characterized in that, The policy model conditioned on semantic objectives includes the following steps: 1) Design a basic action library based on the target operation scenario, and select appropriate action types from it to form corresponding action sequences for each semantic skill, including basic actions such as end-effector movement, opening the gripper, and closing the gripper; 2) Randomly initialize the joint angles of the robotic arm and the poses of each object in the scene in the simulation environment; 3) Use a fully connected layer as the policy network, whose input is defined as... The policy network includes latent image representations and semantic objectives. It has two output branches, one of which acts as the enforcer for reinforcement learning, outputting the next action the simulation environment needs to perform. The action content includes the Cartesian space of the robotic arm's end effector. Another branch acts as the evaluator for reinforcement learning, outputting a value function. ,in The fully connected layer representing the policy network. This represents the desired position and orientation of the robotic arm's end effector. 4) In the policy network simulation, RGB images of the desktop scene are acquired through a camera, and the latent representation of the scene image is obtained using the encoder in the object-oriented representation model. Then the potential representation and semantic goals As input to the policy network, the output includes the appropriate action type and parameters in the action sequence. Execute actions in the simulation environment and return the reward function. The reward function is defined as follows: ； in This indicates that the semantic information corresponding to the current environmental observation matches the semantic target specified by the user. 5) Record the execution trajectory of the robotic arm. ,Record The trajectory is used to complete the training data collection. The collected data is used to train the network parameters of the policy. Based on backpropagation, the gradient of the network parameters is calculated, and then the gradient descent is used to update the parameter gradient. 6) Repeat steps 2)-5) until the network converges, i.e., the simulation environment has the highest success rate in completing a single semantic skill.

5. The model-based robot operation skill parameter learning method according to claim 2, characterized in that, The implicit state transition equation learning module includes the following steps: 1) Randomize the initial state of the task scenario, randomize the primitive skill type and action parameters, collect data during the robot's interaction with the environment, and collect the result image and corresponding skill type and action parameters after each skill execution, using a single primitive skill as the segmentation scale. 2) Using fully connected layers as the implicit state transition model network Its input is defined as This includes the latent representation of the image, the skill type of the semantic skill, and the action parameters, outputting the latent representation of the scene image after the robot performs the corresponding action. ; 3) Using the image data collected in step 1), the latent representation of each frame of the image is obtained through the encoder of the object representation network. and skill type Action parameters Both serve as inputs to the network to predict the latent representation of the scene image after skill execution. The loss function of the prediction model is shown in Equation 3): ； Among them For the network's predicted output, This is the potential representation of a real image after being processed by an encoder.

6. The model-based robot operation skill parameter learning method according to claim 3, 4, or 5, characterized in that, The online skill parameter planning module includes the following steps: 1) Parse the target semantic information of multi-step tasks to obtain the skill sequence and semantic target sequence of multi-step operations; 2) Based on the operation sequence and semantic target sequence obtained in step 1), input the RGB image of the task scene, call the policy network of the corresponding single semantic skill, sample and generate multiple feasible action parameters, and construct the corresponding parameter candidate set; 3) Based on the candidate set of action parameters obtained in step 2), the implicit state transition model is used to predict the potential representation of the corresponding primitive skill after execution with each action parameter. The decoder in the object-oriented representation network is used to generate the semantic information corresponding to each potential representation. The matching of semantic information and corresponding semantic target is used as the evaluation index for action parameter selection. 4) Based on the skill sequence in step 1), repeat steps 2) and 3) in a loop, using the CEM planning method and MPC method to select action parameters that can complete the corresponding semantic goal from the action parameter set of the corresponding semantic skill, and achieve the final semantic goal.

Images