Dexterous hand operation method and device, electronic equipment and computer readable medium

Abstract

The embodiment of the invention provides a dexterous hand operation method and device, electronic equipment and a computer readable medium. The method comprises the following steps: establishing a simulation environment for simulating interaction between the dexterous hand and an object, constructing a strategy neural network, training the strategy neural network by using interaction data generated in the simulation environment, and then controlling the entity dexterous hand by using the trained strategy neural network so as to complete an operation task on a target object. The action of the dexterous hand is controlled by adopting the strategy neural network, and an operation strategy is autonomously learned in a simulation environment through reinforcement learning, so that the dependence on precise modeling of complex power equipment can be bypassed, and the trained strategy network can directly adapt to diversified field working conditions of power operation and maintenance scenes; therefore, the problem of insufficient generalization ability caused by model mismatch in a traditional method is solved, and the applicability of the dexterous hand in an electric power operation and maintenance scene is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of power equipment control, and particularly relates to a dexterous hand operation method and device, electronic equipment and a computer readable medium. BACKGROUND

[0002] In the field of power production, transmission and distribution, the normal operation and maintenance of a large number of key equipment depend on precise physical operations. Operating these devices often requires being located in a complex, high-risk or hard-to-reach environment, and an automated solution is urgently needed. As a machine hand with high degrees of freedom and anthropomorphic operation capabilities, a multi-fingered dexterous hand can mimic human hand movements such as grasping, rotating and pressing, and provides an ideal technical carrier for the automation of power equipment operation and maintenance.

[0003] The control operation method of traditional dexterous hands relies heavily on accurate mathematical modeling of the operation target and the operation process, and is often only effective in extremely limited scenarios. However, the devices and control methods in actual power operation scenarios are complex and diverse, and the dexterous hand operation method based on traditional modeling methods is difficult to meet the diverse operation tasks of complex power equipment, which greatly limits the practical application of dexterous hands in power operation scenarios. SUMMARY

[0004] The embodiments of the present application provide a dexterous hand operation method, device, electronic equipment and computer readable medium to solve the problem of poor dexterous hand operation effect.

[0005] In a first aspect, the embodiments of the present application provide a dexterous hand operation method, comprising:

[0006] establishing a simulation environment for simulating the interaction between the dexterous hand and the object;

[0007] constructing a policy neural network based on reinforcement learning according to the physical information of the dexterous hand and the target object;

[0008] training the policy neural network using the interaction data generated by the simulation environment;

[0009] controlling the physical dexterous hand using the trained policy neural network to complete the operation task on the target object.

[0010] In some possible implementation manners, training the policy neural network using the interaction data generated by the simulation environment comprises:

[0011] generating the dexterous hand and the target object in the simulation environment according to the operation task;

[0012] randomizing the shape and size of the target object and the initial grasping pose of the target object in the dexterous hand;

[0013] The randomization result is taken as an input of the policy neural network, and the dexterous hand interacts with the target object in the simulation environment according to an output of the policy neural network;

[0014] Based on the interaction data of the dexterous hand and the target object, the parameters of the policy neural network are optimized through a preset reward function.

[0015] In some possible implementation manners, the operation task is to control the power equipment, and the reward function includes an anti-collision penalty for punishing the contact between the dexterous hand and other parts of the power equipment except the target operation component.

[0016] In some possible implementation manners, based on the interaction data of the dexterous hand and the target object, the parameters of the policy neural network are optimized through a preset reward function, including:

[0017] The policy neural network is set to an evaluation mode, the network parameter update is frozen, and the dexterous hand interacts with the target object in the simulation environment for a preset number of time steps, and observation data, action data and reward data in the interaction process are recorded;

[0018] The policy neural network is switched to a training mode, and the parameters of the policy neural network are iteratively updated based on the data recorded in the evaluation mode.

[0019] In some possible implementation manners, the parameters of the policy neural network are iteratively updated based on the data recorded in the evaluation mode, including:

[0020] The data recorded in the evaluation mode is input into the policy neural network;

[0021] Based on a proximal policy optimization algorithm, a training loss is calculated according to an output result of the policy neural network;

[0022] The parameters of the policy neural network are updated according to the training loss.

[0023] In some possible implementation manners, the policy neural network includes:

[0024] A privileged information processing network is configured to map privileged information of the target object in the simulation environment into a low-dimensional embedding vector;

[0025] A feature extraction network is configured to extract deep features from the observation data, and the observation data includes current joint positions of the dexterous hand, historical joint positions, historical actions and the low-dimensional embedding vector;

[0026] An action distribution branch is connected to an output end of the feature extraction network and is configured to output a mean value of the target action;

[0027] A value function branch is connected to the output end of the feature extraction network and is configured to output a value estimation of the target action and the current state.

[0028] In some possible implementation manners, a mean value of the target action output by the action distribution branch, and a learnable parameter independent of the policy neural network constitute an action distribution, and the learnable parameter is an action variance of the target action.

[0029] During training of the policy neural network, the execution action is obtained by sampling from the action distribution; and during actual deployment of the policy neural network for performing the operation task, the mean value of the target action is directly used as the execution action.

[0030] In a second aspect, an embodiment of the present application provides a dexterous hand operation device, comprising:

[0031] A simulation module is configured to establish a simulation environment for simulating interaction between the dexterous hand and an object;

[0032] A construction module is configured to construct a policy neural network based on reinforcement learning according to physical information of the dexterous hand and the target object;

[0033] A training module is configured to train the policy neural network by using interaction data generated by the simulation environment;

[0034] An operation module is configured to control the physical dexterous hand by using the trained policy neural network, so as to complete the operation task on the target object.

[0035] In a third aspect, an embodiment of the present application provides an electronic device, comprising a memory and a processor.

[0036] The memory stores computer execution instructions.

[0037] The processor executes the computer execution instructions stored in the memory, so that the processor executes the first aspect and / or various possible implementation manners of the first aspect.

[0038] In a fourth aspect, an embodiment of the present application provides a computer readable storage medium, which stores computer execution instructions, and the computer execution instructions are executed by the processor to implement the first aspect and / or various possible implementation manners of the first aspect.

[0039] In a fifth aspect, an embodiment of the present application provides a computer program product, which comprises a computer program, and the computer program is executed by the processor to implement the first aspect and / or various possible implementation manners of the first aspect.

[0040] The method, device, electronic device and computer readable medium provided by the embodiment of the present application can train the strategy neural network by establishing a simulation environment simulating the interaction between the dexterous hand and the object and constructing the strategy neural network, and then control the physical dexterous hand by using the trained strategy neural network to complete the operation task on the target object. The strategy neural network is used to control the dexterous hand action, and the operation strategy is autonomously learned in the simulation environment through reinforcement learning, so that the dependence on accurate modeling of complex power equipment can be bypassed, the trained strategy network can be directly adapted to the diversified field working conditions of the power operation and maintenance scene, thereby solving the problem of insufficient generalization ability caused by model mismatch in the traditional method, and improving the applicability of the dexterous hand in the power operation and maintenance scene. BRIEF DESCRIPTION OF DRAWINGS

[0041] The accompanying drawings, which are incorporated herein and form part of the specification, illustrate embodiments consistent with the present application and, together with the description, further serve to explain the principles of the application.

[0042] Figure 1 A flowchart of a dexterous hand operation method provided by an embodiment of the present application;

[0043] Figure 2 A general logic diagram of a dexterous hand operation method provided by an embodiment of the present application;

[0044] Figure 3 A hierarchical architecture diagram of a simulation environment provided by an embodiment of the present application;

[0045] Figure 4 A logic diagram of a strategy neural network provided by an embodiment of the present application;

[0046] Figure 5 An architecture diagram of a privileged information processing network provided by an embodiment of the present application;

[0047] Figure 6 A structure diagram of a dexterous hand operation device provided by an embodiment of the present application;

[0048] Figure 7 A structure diagram of an electronic device provided by the present application.

[0049] Through the above drawings, the specific embodiments of the present application have been shown, and will be described in more detail hereinafter. These drawings and textual descriptions are not intended to limit the scope of the concept of the present application by any means, but to illustrate the concept of the present application to those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION

[0050] The exemplary embodiments will be described in detail herein with reference to the accompanying drawings. Wherever possible, the same reference numbers will be used throughout the drawings to refer to the same or like elements. The following description of exemplary embodiments is not representative of all embodiments consistent with the present application. Rather, it is merely an example of apparatus and methods consistent with some aspects of the present application as detailed in the appended claims.

[0051] In the field of power generation, transmission and distribution, the normal operation and maintenance of a large number of key equipment rely on precise physical operations, such as opening and closing circuit breakers, rotating valve knobs, pulling and inserting fuse blocks, and replacing fuses, etc. These operations are usually located in complex, high-risk or hard-to-reach environments, and urgently need automation solutions. As a multi-fingered dexterous hand with high degrees of freedom and anthropomorphic operation capabilities, it can simulate human hand grasping, rotating, pressing and other dexterous actions, providing an ideal technical carrier for the realization of automated operation and maintenance and remote precise control of power equipment, and has important significance for improving the intelligent level and operation safety of power grids.

[0052] In the actual scenario of power operation and maintenance, there are many types of equipment involved, with different specifications, from the thick operating rod of large disconnectors to the small knob of precision instruments, and their operation methods are also different, which poses a severe challenge to dexterous hands based on traditional modeling control. This method relies on accurate dynamic modeling of specific objects and operation tasks, and is often only effective in extremely limited scenarios, making it difficult to meet the requirements of many operation tasks in the power operation and maintenance scenario.

[0053] For example, some operation tasks require continuous rotation of key parts of equipment, such as valves or knobs, which requires the dexterous hand to stably and continuously rotate the object in the palm under complex contact mechanics. It is extremely difficult for traditional control systems that rely on accurate models to adapt to the changing contact points and friction forces in the rotation process in real time, which can easily lead to object slipping or operation stagnation, and cannot reliably complete the intended task.

[0054] For power operation and maintenance scenarios involving multiple types of complex equipment, dexterous hands based on traditional modeling are difficult to adapt to diversified tasks in unstructured environments, and their robustness and generalization ability are severely insufficient, which greatly limits the practical application of dexterous hands in power operation and maintenance scenarios.

[0055] Based on this, the present application proposes a technical concept of constructing a strategy neural network according to the dexterous hand and the type of power equipment. This network can be used to output control targets according to the joint positions of the dexterous hand. The action data of the dexterous hand is generated in a simulation environment according to the network, and the strategy neural network is trained according to these data. The trained strategy neural network can be used to control the entity dexterous hand to complete diversified operation tasks in the power operation and maintenance scenario.

[0056] The technical solutions of the present application and how the technical solutions solve the above technical problems will be described in detail below with specific examples. The following specific examples can be combined with each other, and the same or similar concepts or processes may not be described again in some examples. The embodiments of the present application will be described below with reference to the accompanying drawings.

[0057] Figure 1 A flowchart of a dexterous hand operation method provided in an embodiment of the present application is shown in FIG. 1, which includes the following steps. Figure 1

[0058] In step S101, a simulation environment for simulating the interaction between the dexterous hand and the object is established.

[0059] In an embodiment of the present application, a simulation environment framework can be constructed based on a simulation platform (such as Isaac Gym or Isaac Lab, etc.), including defining an environment base class and a task environment class, then a hierarchical architecture can be used to construct a parallel simulation environment, multiple independent environments can be generated in a single simulation instance, and the physical properties, initial poses and collision rules of the dexterous hand and the object in each environment can be configured. The simulation environment supports environment stepping, which can output pre-processing, multi-step simulation control and post-processing according to the strategy, and can also reset the global or partial simulation environment.

[0060] In step S102, a policy neural network based on reinforcement learning is constructed according to the physical information of the dexterous hand and the target object.

[0061] The target object can include various power equipment in the electric power operation scene, and the physical information can include the size, mass and friction coefficient of the equipment.

[0062] In an embodiment of the present application, the dexterous hand joint information, equipment size, operation mode and other data can be obtained according to the type of dexterous hand and the type of power equipment, and then a policy neural network is constructed according to these data, which maps the dexterous hand body perception state and the equipment physical information to the dexterous hand target action value.

[0063] ​Exemplarily, the policy neural network can be constructed based on an Actor-Critic architecture. First, neural network related parameters and reinforcement learning related parameters can be obtained, and after obtaining the parameters, the neural network can be instantiated based on an MLP (Multilayer Perceptron) and an Actor-Critic architecture. The neural network related parameters can include the number of hidden layer units of the Actor network, the number of hidden layer units of the privileged information network, the dimension of the action space, the dimension of the observation space, and the like. Such parameters can be organized into a dictionary for subsequent network construction. The reinforcement learning related parameters can include data dimensions, training periods, sampling steps, and the like. Such parameters are hyperparameters directly related to the training effect of the neural network.

[0064] In step S103, the policy neural network is trained using the interaction data generated by the simulation environment.

[0065] In the embodiments of the present application, the size and pose data of the dexterous hand and the object in the simulation environment can be input into the policy neural network to obtain the operation strategy output by the network. The operation strategy output by the policy neural network is used to control the interaction between the dexterous hand and the object in the simulation environment. The state, action and reward data generated by the interaction between the dexterous hand and the object in the simulation environment are collected. A PPO (Proximal Policy Optimization) algorithm is used to construct a comprehensive optimization objective based on these interaction data by calculating the policy loss (based on the action probability ratio and the advantage function), the value loss and the policy entropy reward. The network parameters are dynamically adjusted through multiple rounds of small batch gradient updates, so that the policy neural network gradually learns the control strategy that can efficiently complete the rotation operation.

[0066] In step S104, the trained policy neural network is used to control the entity dexterous hand to complete the operation task on the target object.

[0067] After the policy neural network is trained, the trained network can be deployed to the entity dexterous hand, and the entity dexterous hand can be used in the power operation scene to complete the operation and maintenance tasks such as operating power equipment.

[0068] In the above embodiments, by establishing a simulation environment simulating the interaction between the dexterous hand and the object and constructing a policy neural network, the policy neural network can be trained using the interaction data generated in the simulation environment, and the trained policy neural network can be used to control the entity dexterous hand to complete the operation task on the target object. Using the policy neural network to control the dexterous hand action and using reinforcement learning to learn the operation strategy in the simulation environment can bypass the dependence on accurate modeling of complex power equipment, so that the trained policy network can directly adapt to the diversified field conditions in the power operation scene, thereby solving the problem of insufficient generalization ability caused by model mismatch in traditional methods and improving the applicability of the dexterous hand in the power operation scene.

[0069] Figure 2 The overall logic diagram of a dexterous hand operation method provided by an embodiment of the present application is shown in FIG. 1. As shown in the figure, the method includes the parts of building a simulation environment, constructing a neural network, training the neural network, and deploying an application, etc. Figure 2

[0070] For building a simulation environment, Isaac Gym can be used as a platform, an environment base class defining basic settings and functions of the simulation environment is created, corresponding task environment classes are created according to different dexterous hand models and specific operation tasks, and finally a simulation environment for simulating the interaction between the dexterous hand and the object is built based on the environment base class and the task environment class.

[0071] For constructing and training the neural network, an initial policy neural network can be built based on MLP and Actor-Critic architecture, then the operation strategy generated by the policy neural network is used to control the interaction between the dexterous hand and the object in the simulation environment through the simulation platform, the interaction data is collected and the PPO (Proximal Policy Optimization) algorithm is used to optimize the parameters of the policy neural network, the optimized parameters are used to update the policy neural network, and the iteration is repeated multiple times. Finally, the trained policy neural network is deployed to the entity dexterous hand to realize specific operation tasks such as rotating objects in hand.

[0072] First, how to build a simulation environment will be described in conjunction with specific embodiments.

[0073] Figure 3 The hierarchical architecture of a simulation environment provided by an embodiment of the present application is shown in FIG. 2.

[0074] As shown in the figure, the program implementation of the simulation environment can be completed by creating an environment class, and the environment class framework can be divided into two layers, the upper layer is an environment base class, and the lower layer is a task environment class. The environment base class can include an abstract base class Env and an abstract base class VecTask that inherits the Env class and is designed for some general functions of the Isaac Gym simulation environment. The task environment class can correspond to each different hardware (such as power equipment) and operation and maintenance task. Figure 3

[0075] Among them, the environment base class design is the basis for the implementation of the task environment, and unified environment configuration and efficient simulation management can be realized through abstract base class (Abstract Base Class, ABC) and multi-environment management class based on Python code.

[0076] The functions that can be implemented by the environment base class include:

[0077] ​​(I) Set up the simulation device. According to the configuration file, specify the GPU or CPU as the computing processor, and specify the default name of the GPU device as "cuda:0".

[0078] (II) Define the observation space and action space. Use the space.Box statement to create the initial observation space obs_space and action space act_space, with shapes [n obs ], [n act ], and [n obs ] respectively. [n act ] represents a one-dimensional array with a length of n obs = 2 * 3 * n act , the coefficient 2 comes from the observation result of one time step including the current state and target position of all joints, and the coefficient 3 represents the observation result of the last 3 time steps as complete observation data.

[0079] (III) Create a controller. The controller uses a PD (Proportional control) controller, which can include torque control mode and position control mode. The difference between the two control modes lies in the logic of updating the low-level control instructions in each subsequent simulation control step (simulation step).

[0080] (IV) Parse simulation parameters. Read the configuration from the configuration file, including simulation step size, sub-step number, etc. Use the Gym simulator method to set the upper axis and gravity value.

[0081] (V) Create a tensor buffer. Create a tensor buffer for observation, immediate reward, and environment ID that needs to be reset, with shapes [n env ], where n env represents the number of all parallel environments.

[0082] (VI) Set the viewer. Setting the viewer can achieve visualization of the training process and the effect of the neural network, which is an important function to assist code improvement.

[0083] The task environment class is the second level of environment class creation. Based on the environment base class, it can be designed separately for each different dexterous hand model and task (such as operating objects and generating initial grasp pose dataset). During the training process, different operation names are specified through the configuration file, which will call the corresponding task environment class to execute training. Through this method, the code can be designed in a targeted and standardized manner.

[0084] As a subclass of the environment base class, the task environment class can implement the specific functions declared in the environment base class. The goal of these functions is to create and maintain the environment during the entire training process.

[0085] The functions that can be implemented by the task environment class include:

[0086] (I) Set the environment configuration. Based on the configuration file, the following four configurations can be set in turn: domain randomization, privileged information, object information, and reward function configuration.

[0087] Domain randomization refers to randomizing the physical properties of objects and dexterous hands during training, so that the policy neural network is trained in a sufficiently diverse environment, thereby ensuring the robust performance of the policy neural network in different situations.

[0088] Privileged information is physical information that can only be obtained in the simulation environment, including object size, position, and friction parameters. Inputting privileged information during training can improve the training speed and final performance of the policy neural network.

[0089] Object information refers to the type and allocation method of the object primitives operated in this training process. For example, cubes, cylinders, and spheres of different radii.

[0090] The reward function is an important indicator for evaluating the performance of the policy neural network during training. The reward function can include multiple reward items and penalty items, and the weights of each reward can be set when setting the configuration.

[0091] (II) Create a simulation environment. This function can initialize multiple environments in parallel based on the multi-environment parallel simulation function of the Isaac Gym framework, generate multiple independent environments in a single simulation instance, and configure the physical properties of the robot and objects in each environment to build the overall training scene for dexterous hand operation of electrical equipment and other objects.

[0092] First, create the ground using the gym (simulation platform) method, set the simulator ground normal to the z-axis as the unified physical support for the environment. Set the layout upper and lower limits of the environment instance and the spacing between adjacent environments to form a spatial matrix with a grid distribution.

[0093] Then load the resources of the hand and objects and configure the joints of the dexterous hand. Read the urdf model according to the preset path, and the dexterous hand control mode is divided into torque control mode and position control mode. This will be reflected in the subsequent update of the low-level control. Read the joint setting tensor from gym, iterate through each joint, and set its angle upper and lower limits, stiffness, damping, friction, and inertia. After loading the model, set the initial pose of the dexterous hand and objects, which has 7 dimensions, consisting of the first three dimensions x, y, z and the last four dimensions of the quaternion value. urdf (Unified Robot Description Format) is a format based on XML specifications for describing robot structures.

[0094] Further, the loop is opened according to the number of environments, and an instance is created for each environment. The instance of the robot hand and the object is created by calling the gym.create_actor method according to the pose and joint attribute parameters set in the previous step. When creating the instance, the collision filter and the collision group can be set, and the dexterous hand is set to (-1, 0), that is, the urdf predefined collision filtering rule and the default collision group are used, and the object is set to (0, 0), that is, the default group is collided. Through this setting, while ensuring that the internal collision of the robot hand is processed according to the design, the correct physical interaction between the robot hand and the object is allowed.

[0095] Finally, the randomization of the physical properties of the object can be started, including the randomization of the object size scale, the center of mass offset, the mass, and the friction coefficient, and the specific values of each environment are written into the privilege information buffer.

[0096] (Three) Step in the simulation environment. In the embodiment of the application, the simulation environment established supports stepping, that is, one decision step or time step of reinforcement learning. According to the joint action value output by the policy network, all parallel simulation environments can be further pushed, and the reward and observation value of the action and other information can be calculated therein.

[0097] When the target action value (operation policy) output by the policy neural network is obtained, the action value can be clipped to a specified range, and a preprocessing step is performed. In the preprocessing step, the target position is first updated according to the action, and the update process can be represented as formula (1).

[0098] (1)

[0099] Wherein, q t and q target represent the current position and the target position of the dexterous hand joint respectively, a t is the output action value, is the action scaling coefficient (which can be set to 1 / 24).

[0100] Exemplarily, in the preprocessing, a random force disturbance can also be applied to the object (such as a device model of a power equipment in the simulation environment), and then a scaled random new force is applied to the object according to the decay rate after updating the current force considering the size and mass of the object.

[0101] After the preprocessing step is performed, control can be applied in each decision step at a control frequency, and the control frequency and the policy output frequency satisfy the following relationship:

[0102] (2)

[0103] In the above formula, n represents the number of simulation steps performed each time the step is executed, The frequency of the policy output, dt is the time interval set for each simulation step.

[0104] In each simulation step, the low-level control instruction can be updated based on the control mode: when it is the position control mode, the output target position of the preprocessing step is directly taken as the instruction; when it is the torque control mode, the target position is calculated as the torque based on the PD controller formula, and the calculation process can be represented by formula (3).

[0105] (3)

[0106] In the above formula, u(t) is the output torque, K p and K d are the proportional and differential gains respectively, and e(t) is the error between the current position and the target position. After updating the instruction, the simulation interface can be refreshed.

[0107] After all simulation steps are completed, the post-processing step can be executed. In the post-processing process, first, the reward for the current decision step is calculated, and after calculating the reward, it is checked whether the current needs to be reset according to the set termination condition (maximum step number, object height threshold). If it needs to be reset, the reset environment function is executed. Finally, the new observation value for the current environment can be calculated, and the historical observation data is stored in a sliding window with a shape of [n env , 80, n obs / 3], which represents the storage of all environment observation data of the last 80 frames (time steps), and is composed of the observation data of the last 3 frames, so n obs needs to be divided by 3. Therefore, the calculation of the new observation value is to add the observation data of the current decision step to the sliding window, and then obtain and flatten the last 3 frames of data in the sliding window as the final observation data.

[0108] (Four) Reset the environment. This function can reset the pose state of the object and the dexterous hand in the environment, and reset all buffers related to the environment. The environment id that needs to be reset is marked by a binary one-dimensional tensor env_ids with a shape of [n env ], and the corresponding environment position value is 1, which means it needs to be reset. The assignment operation of this list is executed in the post-processing step of advancing the simulation step, and it is initialized to be assigned as all 1.

[0109] Exemplarily, the reset_idx method can be called to perform operations on each environment that needs to be reset. First, the PD gain of the dexterous hand in the environment is re-randomized, and then the contact force of the specified environment is set to zero. Then the poses of the object and the dexterous hand are reset, and if random scaling of the object is involved in the training process, all scaling ratios are traversed when setting the pose, the environment id of the environment to be reset specified for the scaling ratio is obtained, and an initial pose is read from the corresponding initial pose data set to set the pose. After setting the pose, the new state is updated to the gym simulator, and all buffers related to the environment (such as the number of steps, observation data, privilege information, etc.) are reset.

[0110] After resetting the specified environment, all-zero actions can be generated for all environments, and the step function can be called to advance all environments by one step to ensure that all environments have similar and stable training starting points and buffers.

[0111] It should be noted that the above process is a process for resetting all environments, and the timing is usually when the environment is initialized, before training starts, before evaluation starts, and when manually requesting to reset all environments. Resetting the environment that meets the termination condition after advancing the simulation by one step is only a partial reset, which only requires calling the reset_idx method without creating all-zero actions and stepping once.

[0112] In one embodiment, the policy neural network can include four parts: a privilege information processing network, a feature extraction network, an action distribution branch, and a value function branch.

[0113] In the embodiments of the present application, the policy neural network can be implemented as a PPO class in the code. When creating a policy neural network through the PPO class, first, the training parameters are loaded from the configuration file. These parameters can be divided into two categories:

[0114] (1) Parameters related to the neural network, including the number of hidden layer units of the Actor network, the number of hidden layer units of the privilege information network, the dimension of the action space, the dimension of the observation space, etc. This type of parameter will be organized as a dictionary for subsequent network construction.

[0115] (2) Parameters related to the implementation of the PPO method, such as n mbsize , the number of sampling steps n step for a single epoch, the GAE (Advantage Estimation) time step, the KL divergence threshold, etc. This type of parameter is a hyperparameter directly related to the training effect of the policy neural network. After reading the configuration, the network can be instantiated based on the MLP and Actor-Critic architecture.

[0116] Figure 4 A logic diagram of a policy neural network provided in an embodiment of the present application. As Figure 4As shown, the network includes a privileged information processing network, a feature extraction network, an action distribution branch, and a value function branch, where the numbers in the brackets represent the dimension of data, for example, privileged information (9) means that the dimension of the privileged information data is 9. The following will be described respectively.

[0117] (I) Privileged information processing network, used to map the privileged information of the target object in the simulation environment into a low-dimensional embedding vector.

[0118] The network is created based on MLP, the input is the privileged information that can be accurately measured in the simulation environment, which is represented by a 9-dimensional vector at time step t , specifically including: three-dimensional coordinates of object position, object scale, object mass, object friction, and three-dimensional coordinates of object centroid. The output is a processed 8-dimensional embedding value . Using the embedding value instead of directly using the privileged information can assist the training of the proprioceptive adaptation part: it is more difficult and unnecessary to use a network to directly estimate the privileged information of the object, on the contrary, it is much easier to train the network to estimate the low-dimensional embedding value.

[0119] Figure 5 An architecture schematic diagram of a privileged information processing network provided by an embodiment of the present application is shown in the figure. As shown in the figure, the network has four layers in total, the input layer and the output layer at the beginning and the end, respectively, and two hidden layers in the middle, each layer contains 9, 256, 128, and 8 neurons, respectively, and the sigmoid function is used as the activation function of each layer. Figure 5

[0120] (II) Feature extraction network, used to extract deep features from observation data, which includes the current joint position of the dexterous hand, the historical joint position, the historical action, and the low-dimensional embedding vector.

[0121] The network is created based on MLP, which is used as the backbone network of the policy neural network, and the input includes the observation value of the defined observation space, specifically the joint position of the dexterous hand , the predicted action at the previous time step , and the output value of the privileged information processing network . In order to make the observation value implicit velocity and acceleration information, the observation value can additionally add the joint position and predicted action of the dexterous hand at the next two time steps, therefore, the final dimension of the observation space is: n obs =(12+12)*3+8=80.

[0122] ​Exemplarily, the critic network and the actor network can share the first several layers in actual use, and the feature extraction network here is the shared network, not the complete actor network, and the output is not the target action value for each joint, but high-level information (i.e., deep features) abstracted from the observation value.

[0123] The network has four layers in common, with the input layer and the output layer at the beginning and the end, respectively, and two hidden layers in the middle, each layer containing 80, 512, 256, and 128 neurons, respectively, and using a sigmoid function as the activation function of each layer.

[0124] (Three) The action distribution branch is connected to the output end of the feature extraction network and is used to output the mean value of the target action.

[0125] The branch is one of the end branches of the network, and is not a neural network, but a fully connected layer, and the output is the output of the actor network: the mean value of the target action of each joint.

[0126] Exemplarily, the mean value of the target action output by the action distribution branch can constitute an action distribution with a learnable parameter independent of the policy neural network, and the learnable parameter is the action variance of the target action; during the training process of the policy neural network, the execution action is obtained by sampling from the action distribution; when the policy neural network is actually deployed for execution tasks, the mean value of the target action is directly used as the execution action.

[0127] The action variance is not generated by the neural network, but is directly defined as an independent learnable parameter, and this design not only reduces the training amount of the neural network, but also enables the action variance to be a global shared learnable parameter independent of the state, and controls the policy exploration behavior through the dynamic change of the variance.

[0128] During the training process, using the action distribution (action mean value and action variance) instead of directly using the action value has the following beneficial effects:

[0129] 1) Support for exploration and utilization balance. Reinforcement learning algorithms need to balance between "exploration" (trying new actions) and "exploitation" (selecting known good actions). The action variance can reflect the degree of policy exploration, and in the early training stage, the variance is large, encouraging the policy to explore, while in the later training stage, the variance is reduced to ensure the stability of the output. The target action output by the policy neural network during the training process is obtained by sampling from the normal distribution restored according to the mean value and the variance to ensure the exploration behavior, while in actual deployment, the mean value is directly used as the target action.

[0130] 2) To realize the calculation requirement of policy gradient algorithm. The theoretical basis of policy gradient algorithm such as PPO requires that the policy is a differentiable probability distribution, and the network update algorithm needs to calculate the importance sampling ratio, loss, policy entropy and other parameters according to the action distribution. Directly outputting the action value cannot provide these necessary information.

[0131] Finally, the Adam (Adaptive Moment Estimation, an optimization algorithm for deep learning) optimizer is created to realize the optimization of network parameters. The overall learning rate and weight decay coefficient are used to update each network parameter. The overall learning rate can be adjusted adaptively according to the KL divergence and the following formula (4):

[0132] (4)

[0133] In the above formula, represents the learning rate, and its maximum and minimum values can be set artificially. T is a constant value set in advance.

[0134] (Four) The value function branch is connected to the output end of the feature extraction network and is used to output the value estimation of the target action and the current state.

[0135] This branch is one of the end branches of the network and is not a neural network, but a fully connected layer that only performs linear transformation, i.e. without passing through a nonlinear activation function. In the Actor-Critic architecture, the last output layer usually does not use an activation function, because its output usually needs to be an arbitrary real number value, and adding an activation function may limit the output range and cannot reflect the true value. For example, this branch is a value network branch, the input is the high-level information abstracted by the feature extraction network, and the output is the value estimation of the state-action pair.

[0136] In one embodiment, the policy neural network is trained using interaction data generated by a simulation environment, including:

[0137] According to the operation task, a dexterous hand and a target object are generated in the simulation environment; the shape, size and initial grasping pose of the target object in the dexterous hand are randomized; the randomized results are used as the input of the policy neural network, and the output of the policy neural network is used to control the dexterous hand and the target object in the simulation environment; based on the interaction data of the dexterous hand and the target object, the parameters of the policy neural network are optimized through a preset reward function.

[0138] In the embodiments of the present application, the dexterous hand and the target object can be dynamically generated in simulation, and the uncertainty of the equipment specifications and operation postures in power operation and maintenance can be simulated by randomizing the shape, size and initial grasping pose thereof. Then, the policy neural network takes these randomized states as input and outputs a control policy to drive the dexterous hand to interact with the object. In this process, the system evaluates the interaction effect in real time through a preset reward function (such as encouraging rotation, punishing excessive torque and posture deviation, etc.), and repeatedly optimizes the network parameters based on the PPO algorithm using the collected interaction data (state, action, reward), so that the policy gradually learns to complete the operation task stably and efficiently under diversified conditions.

[0139] Illustratively, the operation task is to manipulate the power equipment, and the reward function includes a collision avoidance penalty for penalizing contact between the dexterous hand and other parts of the power equipment except the target operating component.

[0140] In the above embodiments, the domain randomization function is provided in the simulation environment construction stage. By randomizing the physical properties (such as mass, friction, size, etc.) of the object during the training process, the adaptability of the policy neural network to changes in physical parameters can be enhanced, the robustness of the dexterous hand operation can be improved, and the dexterous hand can be more suitable for complex power operation and maintenance scenarios.

[0141] In one embodiment, based on the interaction data between the dexterous hand and the target object, the parameters of the policy neural network are optimized through a preset reward function, including:

[0142] The policy neural network is set to evaluation mode, the network parameter update is frozen, and the dexterous hand interacts with the target object in the simulation environment for a preset number of time steps, and the observation data, action data and reward data in the interaction process are recorded. The policy neural network is switched to training mode, and the parameters of the policy neural network are iteratively updated based on the data recorded in the evaluation mode.

[0143] In some possible implementations, based on the data recorded in the evaluation mode, the parameters of the policy neural network are iteratively updated, including:

[0144] The data recorded in the evaluation mode is input into the policy neural network; based on the proximal policy optimization algorithm, the training loss is calculated according to the output result of the policy neural network; and the parameters of the policy neural network are updated according to the training loss.

[0145] In the embodiments of the present application, after successfully creating and initializing all simulation environments, the training will be performed in a main loop, which will continue to run until the total number of steps of the dexterous hand interacting with the environment reaches a preset maximum value or is actively interrupted by the trainer.

[0146] Each execution of the loop corresponds to a complete training cycle (epoch), which contains two main stages: data collection and network update. Both stages can be implemented by the train_epoch function. In addition to these two stages, the design of the reward function will be introduced separately, as it has a significant impact on the final action. The following sections will provide example-based explanations of data collection, network update, and reward function design.

[0147] (I) Data collection.

[0148] In this stage, the train_epoch function will first set the policy neural network to evaluation mode, freeze the network parameter update, and then call the environment step function to interact with the environment for a specified number of steps n step in all parallel environments. The observations, actions, rewards, and other data collected during the interaction process are recorded. Therefore, when the number of parallel environments is n env , the total amount of interaction data sampled in each epoch n bsize can be represented as:

[0149] (5)

[0150] (II) Network update. After data collection is complete, the train_epoch function switches the policy neural network back to training mode and begins the network update stage. It iterates through the data in the experience replay buffer multiple times. In each iteration, the data is divided into multiple minibatches.

[0151] For each minibatch, the following operations can be performed:

[0152] (1) After obtaining the data (old action probability, value prediction, advantage function estimate, return, action, observation, etc.) from the buffer, normalize the observation data.

[0153] (2) Input the processed observations and previous actions into the Actor-Critic network to obtain the current action log probability, value estimate, policy entropy, and mean and standard deviation of the action distribution.

[0154] (3) Calculate the core loss of the PPO algorithm: including policy loss (Actor loss), value loss (Critic loss), and optional action boundary loss, and calculate the total training loss combined with policy entropy.

[0155] (4) Perform backpropagation to calculate the gradient and perform gradient clipping to prevent gradient explosion.

[0156] (5) Update the network parameters using the Adam optimizer.

[0157] (6) Calculate the KL divergence between the new and old policy, which is used for the subsequent learning rate adjustment.

[0158] In each epoch, the sampled data will be traversed n mbepoch times. In each traversal, all data n bsize will be divided into n mbsize minibatches, so n mb should be set to be divisible by n mbsize . The network parameters will be updated in each minibatch, so the total number of updates n bsize for a complete epoch is: totalupdate

[0159] (6)

[0160] (Three) Reward function design. When optimizing the parameters of the policy neural network, the following reward function r can be used.

[0161] (7)

[0162] In the above formula, (including each subscript ) are customizable weights, and the reward and penalty terms include: rotation reward , which encourages the object to rotate along the desired rotation axis; r linvel is the object linear velocity penalty, which is used to penalize the object's excessive translational speed; r pose is the hand pose deviation penalty, which penalizes the difference between the dexterous hand and the initial grasp pose; r torque is the torque penalty to penalize excessive driving torque; energy consumption penalty , which encourages the dexterous hand to adopt more energy-efficient actions; height penalty , which penalizes the object from deviating from the preset initial height; target position change penalty , which penalizes the large change in target joint angle in consecutive control steps, encouraging smoother motion.

[0163] After completing the main training process of an epoch, the remaining part of the main loop can record the average reward and length statistics of the current training round, and record the policy neural network parameters that exceed the historical recent reward as a checkpoint file. The policy neural network can be trained or actually deployed by reading the checkpoint file, and thus a main loop ends.

[0164] ​In the above embodiments, by explicitly punishing the hand pose deviation, driving torque and energy consumption in the reward function, the policy neural network is guided to autonomously explore a smooth, energy-efficient and physically coordinated continuous action sequence during the learning process, thereby improving the control stability of the dexterous hand. In addition, a complete method for training and using the policy neural network is provided, including two core stages of data collection and network updating. By collecting interaction data in batches in multiple parallel environments and updating in multiple rounds in small batches, the stability and convergence of the training process in the high-dimensional continuous control space are ensured.

[0165] The dexterous hand operation method of the present application provides a complete dexterous hand operation strategy generation scheme from simulation environment building, network construction to policy training, forming a repeatable technical process. This process provides a clear and specific technical implementation path for solving the operation problem of the multi-fingered dexterous hand in the power operation scene.

[0166] Figure 6 The structure diagram of the dexterous hand operation device provided by the present application is shown in Figure 6 The dexterous hand operation device 600 provided by the present embodiment includes:

[0167] The simulation module 601 is configured to establish a simulation environment for simulating the interaction between the dexterous hand and the object;

[0168] The construction module 602 is configured to construct a policy neural network based on reinforcement learning according to the physical information of the dexterous hand and the target object;

[0169] The training module 603 is configured to train the policy neural network using the interaction data generated by the simulation environment;

[0170] The operation module 604 is configured to control the entity dexterous hand using the trained policy neural network to complete the operation task on the target object.

[0171] In some possible implementation manners, the training module 603 can also be configured to: generate the dexterous hand and the target object in the simulation environment according to the operation task; randomize the shape, size of the target object and the initial grasping pose of the target object in the dexterous hand; take the randomized results as the input of the policy neural network, control the dexterous hand and the target object in the simulation environment according to the output of the policy neural network; and optimize the parameters of the policy neural network based on the interaction data of the dexterous hand and the target object through the preset reward function.

[0172] In some possible implementation manners, the operation task is to manipulate the power equipment, and the reward function includes an anti-collision penalty for punishing the contact between the dexterous hand and other parts of the power equipment except the target operation component.

[0173] In some possible implementation manners, the training module 603 can also be configured to: set the policy neural network to an evaluation mode, freeze network parameter updating, and control the dexterous hand to interact with the target object for a preset number of time steps in the simulation environment, and record observation data, action data, and reward data in the interaction process; switch the policy neural network to a training mode, and iteratively update parameters of the policy neural network based on the data recorded in the evaluation mode.

[0174] In some possible implementation manners, the training module 603 can also be configured to: input the data recorded in the evaluation mode into the policy neural network; calculate a training loss according to an output result of the policy neural network based on a proximal policy optimization algorithm; and update the parameters of the policy neural network according to the training loss.

[0175] Figure 7 A structural schematic diagram of an electronic device is provided in the present application. As shown in the figure, the electronic device 70 provided in the embodiment includes at least one processor 701 and a memory 702. Optionally, the device 70 further includes a communication component 703. The processor 701, the memory 702, and the communication component 703 are connected through a bus 704. Figure 7

[0176] In the specific implementation process, the at least one processor 701 executes the computer-executed instructions stored in the memory 702, so that the at least one processor 701 executes the method described above.

[0177] The specific implementation process of the processor 701 can refer to the method embodiments described above, which has similar implementation principles and technical effects, and will not be described here in detail.

[0178] In the above embodiments, it should be understood that the processor can be a central processing unit (CPU), and can also be other general-purpose processors, digital signal processors (DSPs), application specific integrated circuits (ASICs), etc. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor and the like. The steps of the method disclosed in the application can be directly embodied as execution completed by a hardware processor, or executed by a combination of hardware and software modules in the processor.

[0179] The memory can include a random access memory (RAM) and can also include a non-volatile memory (NVM), for example, at least one disk memory.​

[0180] The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, the bus in the drawings of the present application does not limit to only one bus or one type of bus.

[0181] The present application also provides a computer program product, comprising a computer program, which, when executed by a processor, implements the method described above.

[0182] The present application also provides a computer readable storage medium, which stores computer execution instructions, and when a processor executes the computer execution instructions, the method described above is implemented.

[0183] The readable storage medium described above can be implemented by any type of volatile or non-volatile storage device or their combination, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk or optical disk. The readable storage medium can be any available medium that can be accessed by a general or special purpose computer.

[0184] An exemplary readable storage medium is coupled to the processor, so that the processor can read information from the readable storage medium, and can write information to the readable storage medium. Of course, the readable storage medium can also be an integral part of the processor. The processor and the readable storage medium can be located in an application specific integrated circuit (ASIC). Of course, the processor and the readable storage medium can also exist as discrete components in the device.

[0185] The division of units is only a logical functional division, and in actual implementation, there can be another division manner, for example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the units shown or discussed can be indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0186] The units described as separate components may or may not be physically separate, and the components displayed as units may or may not be physical units, i.e., may be located in one place, or may be distributed to multiple network units. Part or all of the units can be selected according to actual needs to achieve the purpose of the embodiment scheme.

[0187] In addition, each functional unit in various embodiments of the application can be integrated into one processing unit, or each unit can exist physically, or two or more units can be integrated into one unit.

[0188] If the function is realized in the form of a software function unit and sold or used as an independent product, it can be stored in a computer readable storage medium. Based on this understanding, the technical solutions of the application essentially or the part that contributes to the prior art or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes a number of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the various embodiment methods of the application. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk or an optical disk, and various media that can store program codes.

[0189] Those skilled in the art can understand that all or part of the steps of the above-mentioned method embodiments can be completed by program instruction related hardware. The aforementioned program can be stored in a computer readable storage medium. The program executes to perform the steps of the above-mentioned method embodiments; and the aforementioned storage medium includes: ROM, RAM, magnetic disk or optical disk, and various media that can store program codes.

[0190] Finally, it should be noted that those skilled in the art, after considering the specification and practicing the application disclosed herein, will easily think of other embodiments of the application. The application is intended to cover any variations, uses, or adaptations of the application that follow the general principles of the application and include common knowledge or conventional technical means in the art that are not disclosed by the application, and is not limited to the precise structure described above and shown in the drawings, and various modifications and changes can be made without departing from the scope thereof. The scope of the application is only limited by the appended claims.

Claims

1. A dexterous hand operation method characterized by, The method comprises the steps of: establishing a simulation environment for simulating the interaction between a dexterous hand and an object; constructing a policy neural network based on reinforcement learning according to the physical information of the dexterous hand and the target object; training the policy neural network using the interaction data generated by the simulation environment; controlling the physical dexterous hand using the trained policy neural network to complete the operation task on the target object.

2. The method of claim 1, wherein, The step of training the policy neural network using the interaction data generated by the simulation environment comprises the steps of: generating the dexterous hand and the target object in the simulation environment according to the operation task; randomizing the shape, size and initial grasping pose of the target object in the dexterous hand; controlling the dexterous hand to interact with the target object in the simulation environment according to the output of the policy neural network, taking the randomized results as the input of the policy neural network; optimizing the parameters of the policy neural network based on the interaction data of the dexterous hand and the target object through a preset reward function.

3. The method of claim 2, wherein, The operation task is to manipulate a power device, and the reward function includes a collision avoidance penalty for punishing the contact between the dexterous hand and other parts of the power device except the target operation component.

4. The method according to claim 2 or 3, characterized in that, The step of optimizing the parameters of the policy neural network based on the interaction data of the dexterous hand and the target object through a preset reward function comprises the steps of: setting the policy neural network to an evaluation mode, freezing the network parameter update, and controlling the dexterous hand to interact with the target object in the simulation environment for a preset number of time steps, and recording the observation data, action data and reward data in the interaction process; switching the policy neural network to a training mode, and iteratively updating the parameters of the policy neural network based on the data recorded in the evaluation mode.

5. The method of claim 4, wherein, The step of iteratively updating the parameters of the policy neural network based on the data recorded in the evaluation mode comprises the steps of: inputting the data recorded in the evaluation mode into the policy neural network; calculating a training loss according to the output result of the policy neural network based on a proximal policy optimization algorithm; updating the parameters of the policy neural network according to the training loss.

6. The method according to any one of claims 1 to 3, characterized in that, The policy neural network comprises: a privileged information processing network for mapping the privileged information of the target object in the simulation environment into a low-dimensional embedding vector; a feature extraction network for extracting deep features from observation data, the observation data including the current joint position, the historical joint position, the historical action of the dexterous hand and the low-dimensional embedding vector; an action distribution branch connected to the output end of the feature extraction network for outputting the mean value of the target action; a value function branch connected to the output end of the feature extraction network for outputting the value estimate of the target action and the current state.

7. The method of claim 6, wherein, The mean value of the target action output by the action distribution branch and a learnable parameter independent of the policy neural network constitute an action distribution, and the learnable parameter is the action variance of the target action. In the training process of the policy neural network, an execution action is obtained by sampling from the action distribution; in actual deployment of the policy neural network for performing an operation task, a mean value of the target action is directly used as an execution action.

8. A dexterous hand operating device characterized by comprising: The method comprises the steps of: An emulation module is configured to establish an emulation environment for simulating interaction between a dexterous hand and an object; A construction module is configured to construct a policy neural network based on reinforcement learning according to physical information of the dexterous hand and the target object; A training module is configured to train the policy neural network by using interaction data generated by the emulation environment; An operation module is configured to control an actual dexterous hand by using the trained policy neural network to complete an operation task on a target object.

9. An electronic device, comprising: The method comprises the steps of: A processor and a memory connected to the processor in communication; The memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory to implement the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, and the computer-executable instructions are executed by the processor to implement the method according to any one of claims 1 to 7.

Images