Autonomous operation training system and method for humanoid robot dexterous hand

Abstract

The invention discloses an autonomous operation training system and method for a humanoid robot dexterous hand, and the system comprises an image collection unit, a pose estimation network unit, a humanoid dexterous hand structure, a micro-control unit, a host, a feature fusion and extraction network unit, an autonomous strategy network unit, and a PD controller. Wherein the host receives a voltage digital signal, and performs binarization processing on the voltage digital signal according to a preset threshold value to obtain pressure sensing information; the feature fusion and extraction network unit receives the attitude information, the pressure sensing information and the joint angle information, and performs feature fusion on the attitude information, the pressure sensing information and the joint angle information to obtain global sensing spatial-temporal features; and the autonomous strategy network unit receives the global perception spatio-temporal characteristics, and obtains an optimal action sequence according to the global perception spatio-temporal characteristics. According to the invention, the problem of dexterous hand robust intelligent control under the condition of lack of key modal information of joint angles and object poses is solved.

Description

Technical Field

[0001] The present invention belongs to the technical field of robot intelligent control, and in particular relates to an autonomous operation training system and method for the dexterous hands of a humanoid robot. Background Art

[0002] Robot autonomous control is to organically combine the perception, decision-making, coordination and action capabilities of autonomous control systems without human intervention, make self-decision and continuously execute a series of control functions according to certain control strategies in unstructured environments, so as to achieve the predetermined goals. Robot autonomous control technology is an important development direction of intelligent control, and its research and application are of great significance to improving the intelligence level of robot systems. Dexterous hands have a bionic structure similar to that of human hands, and have high degrees of freedom and rich sensing functions. In theory, they can perform complex and delicate operations that human hands can perform. They are the most ideal end effectors for humanoid robot systems, and intelligent operation control is the key technology that determines their development and application. For example, through deep reinforcement learning algorithms, complex dexterous hand operation behaviors can be learned in simulation, and various autonomous operation controls such as grasping of dexterous hands and redirection of objects in the palm can be completed.

[0003] Although dexterous hands can improve operational dexterity in complex and unstructured environments, the fine control method is the main reason that restricts the actual improvement of flexibility. At present, there are two main methods for controlling the operation of dexterous hands of humanoid robots: one is a model-based method, involving patent [A method and system for controlling fingers of a robot dexterous hand, 202011020923.5, authorized], which proposes a control method based on motor current detection that drives finger movement. This method requires obtaining the shape and material of the operated object in advance, and establishing a control model for each finger of the dexterous hand to grasp each type of object. It is impossible to complete the precise operation of objects with unmodeled shapes and materials. Therefore, this method is difficult to adapt to the needs of humanoid robots to operate various objects in service scenarios; the other is a learning-based method, involving patent [Control method for five-finger dexterous robotic hand based on multi-agent deep reinforcement learning, 202310373017.0, authorized], which provides a dexterous robotic hand control method based on multi-agent deep reinforcement learning, treating each finger, wrist and arm as a separate agent and training them separately, and using reinforcement learning to make each agent cooperate with each other to complete precise dexterous operation. However, this method only considers the control task of the dexterous hand in the virtual simulation scenario, and has not completed the model migration and method verification for the real dexterous hand structure. Summary of the invention

[0004] The technical problem solved by the present invention is: to overcome the deficiencies of the prior art, to provide an autonomous operation training system and method for the dexterous hands of a humanoid robot, and to solve the problem of robust intelligent control of dexterous hands in the absence of key modal information such as joint angles and object postures.

[0005] The object of the present invention is achieved through the following technical solutions: an autonomous operation training system for humanoid robot dexterous hands, comprising: an image acquisition unit, a posture estimation network unit, a humanoid dexterous hand structure, a microcontroller unit, a host, a feature fusion and extraction network unit, an autonomous strategy network unit and a PD controller; wherein the image acquisition unit acquires image set information of the operated object and inputs the image set information into the posture estimation network unit; the posture estimation network unit obtains the posture information of the operated object according to the image set information and transmits the posture information to the feature fusion and extraction network unit; the FSR thin film pressure strain sensor of the humanoid dexterous hand structure is connected to the microcontroller unit; the FSR thin film pressure strain sensor of the humanoid dexterous hand structure transmits the acquired analog voltage signal to the microcontroller unit; the microcontroller unit receives the analog voltage signal, performs analog-to-digital conversion on the analog voltage signal to obtain a voltage digital signal, and transmit the voltage digital signal to the host; the host receives the voltage digital signal, binarizes the voltage digital signal with a preset threshold to obtain pressure sensing information, and transmits the pressure sensing information to the feature fusion and extraction network unit; the joint angle information of the anthropomorphic dexterous hand structure is transmitted to the feature fusion and extraction network unit; the feature fusion and extraction network unit receives posture information, pressure sensing information and joint angle information, performs feature fusion on the posture information, pressure sensing information and joint angle information to obtain global perception spatiotemporal features, and transmits the global perception spatiotemporal features to the autonomous strategy network unit; the autonomous strategy network unit receives the global perception spatiotemporal features, obtains the optimal action sequence according to the global perception spatiotemporal features, and transmits the optimal action sequence to the PD controller; the PD controller receives the optimal action sequence, generates control instructions according to the optimal action sequence, and controls the actions of the anthropomorphic dexterous hand structure through the control instructions.

[0006] In the above-mentioned autonomous operation training system for humanoid robot dexterous hands, the image acquisition unit acquires image set information of the operated object, including: placing three cameras above the humanoid dexterous hand structure; each camera acquires a set of image information, and the three sets of image information are combined into image set information.

[0007] In the above-mentioned autonomous operation training system for humanoid robot dexterous hands, the posture estimation network unit obtains the posture information of the operated object based on the image set information, including: the ResNet50 network of the posture estimation network unit generates a feature map for each image information; the posture estimation network unit expands and splices the three feature maps together, and generates the posture information of the operated object after passing through the fully connected layer Linear.

[0008] In the above autonomous operation training system for humanoid robot dexterous hands, the posture information of the operated object is obtained by the following formula:

[0009] pose = (x, y, z, α, β, γ);

[0010]

[0011] Among them, pose is the posture information of the operated object, flatten(·) is the matrix expansion operation, concat(·) is the matrix concatenation operation, x is the x-axis coordinate value of the object position in the three-dimensional space coordinate system, y is the y-axis coordinate value of the object position in the three-dimensional space coordinate system, z is the z-axis coordinate value of the object position in the three-dimensional space coordinate system, α is the angle of the object posture relative to the x-axis, β is the angle of the object posture relative to the y-axis, γ is the angle of the object posture relative to the z-axis, Linear() is a series of fully connected layers, is the first feature map, is the second feature map, is the third feature map.

[0012] In the above-mentioned autonomous operation training system for humanoid robot dexterous hands, the posture information, pressure sensor information and joint angle information are fused to obtain global perception spatiotemporal features, including: fusing the pressure sensor information and joint angle information to obtain the dexterous hand proprioception features; fusing the dexterous hand proprioception features with the posture information to obtain the global perception spatiotemporal features.

[0013] In the above autonomous operation training system for humanoid robot dexterous hands, the proprioceptive characteristics of the dexterous hands are obtained by the following formula:

[0014]

[0015] Among them, Z is the proprioceptive feature of the dexterous hand, angle is the joint angle information, and F is the pressure sensing information. is multiplication, G(angle,F) is a nested attention module, and Att() is an adaptive channel attention module.

[0016] The invention discloses an autonomous operation training method for a dexterous hand of a humanoid robot, comprising: collecting image set information of an operated object, inputting the image set information into a posture estimation network; the posture estimation network obtains posture information of the operated object according to the image set information, and transmits the posture information to a feature fusion and extraction network; collecting analog voltage signals; converting the analog voltage signals into voltage digital signals after analog-to-digital conversion; binarizing the voltage digital signals with a preset threshold value to obtain pressure sensing information, and transmitting the pressure sensing information to a feature fusion and extraction network; transmitting joint angle information of a humanoid dexterous hand structure to a feature fusion and extraction network; the feature fusion and extraction network receives posture information, pressure sensing information and joint angle information, performs feature fusion on the posture information, pressure sensing information and joint angle information to obtain global perception spatiotemporal features, and transmits the global perception spatiotemporal features to the autonomous strategy network; the autonomous strategy network receives the global perception spatiotemporal features, and obtains an optimal action sequence according to the global perception spatiotemporal features; generates control instructions according to the optimal action sequence, and controls the action of the humanoid dexterous hand structure through the control instructions.

[0017] In the above-mentioned autonomous operation training method for humanoid robot dexterous hands, collecting image set information of the operated object includes: placing three cameras above the humanoid dexterous hand structure; each camera collects a set of image information, and combining the three sets of image information into image set information.

[0018] In the above-mentioned autonomous operation training method for humanoid robot dexterous hands, the pose estimation network obtains the pose information of the operated object based on the image set information, including: the ResNet50 network layer of the pose estimation network generates a feature map for each image information; the pose estimation network expands and splices the three feature maps together, and generates the pose information of the operated object after passing through the fully connected layer Linear.

[0019] An electronic device, characterized in that it includes: a memory for storing computer-readable instructions; and a processor for running the computer-readable instructions to execute an autonomous operation training method for a humanoid robot's dexterous hand.

[0020] Compared with the prior art, the present invention has the following beneficial effects:

[0021] The present invention takes into account the absolute spatial posture of the operated object and the characterization of the multi-finger contact mechanical state of the dexterous hand, and proposes a hierarchical attention recursive fusion module for large-scale complex heterogeneous data. By gradually fusing the posture estimation information with the distributed pressure sensing spatiotemporal feature information, a globally consistent feature representation is obtained, and complementary enhanced active perception is achieved, which solves the problem of robust intelligent control of dexterous hands in the absence of key modal information such as joint angles and object postures. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Various other advantages and benefits will become apparent to those of ordinary skill in the art by reading the detailed description of the preferred embodiments below. The accompanying drawings are only for the purpose of illustrating the preferred embodiments and are not to be considered as limiting the present invention. Moreover, the same reference symbols are used throughout the accompanying drawings to represent the same components. In the accompanying drawings:

[0023] Figure 1 It is a composition diagram of the overall training system for autonomous operation of the dexterous hand of a humanoid robot provided by an embodiment of the present invention;

[0024] Figure 2 is a flow chart of a method for strengthening learning and training autonomous operation of dexterous hands provided by an embodiment of the present invention;

[0025] Figure 3 is a schematic diagram of the logical structure of the attention recursive fusion module provided by an embodiment of the present invention;

[0026] Figure 4 Schematic diagram of the structure of the humanoid dexterous hand provided by an embodiment of the present invention;

[0027] Figure 5 It is a structural block diagram of an autonomous operation training system for a humanoid robot's dexterous hand provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0028] Exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although exemplary embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided in order to enable a more thorough understanding of the present disclosure and to be able to fully convey the scope of the present disclosure to those skilled in the art. It should be noted that, in the absence of conflict, the embodiments of the present invention and the features described in the embodiments can be combined with each other. The present invention will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0029] Figure 5 1 is a structural block diagram of an autonomous operation training system for a humanoid robot dexterous hand provided by an embodiment of the present invention. Figure 5 As shown, the autonomous operation training system for humanoid robot dexterous hands includes: an image acquisition unit, a posture estimation network unit, a humanoid dexterous hand structure, a micro control unit, a host, a feature fusion and extraction network unit, an autonomous strategy network unit and a PD controller.

[0030] The image acquisition unit acquires image set information of the operated object, and inputs the image set information into the posture estimation network unit; the posture estimation network unit obtains posture information of the operated object according to the image set information, and transmits the posture information to the feature fusion and extraction network unit; the FSR thin film pressure strain sensor of the humanoid dexterous hand structure is connected to the microcontroller unit; the FSR thin film pressure strain sensor of the humanoid dexterous hand structure transmits the acquired analog voltage signal to the microcontroller unit; the microcontroller unit receives the analog voltage signal, performs analog-to-digital conversion on the analog voltage signal to obtain a voltage digital signal, and transmits the voltage digital signal to the host; the host receives the voltage digital signal, performs binarization processing on the voltage digital signal with a preset threshold value, and obtains a pressure signal. The sensing information is transmitted to the pressure sensing information and the feature fusion and extraction network unit; the joint angle information of the humanoid dexterous hand structure is transmitted to the feature fusion and extraction network unit; the feature fusion and extraction network unit receives the posture information, pressure sensing information and joint angle information, performs feature fusion on the posture information, pressure sensing information and joint angle information to obtain the global perception spatiotemporal features, and transmits the global perception spatiotemporal features to the autonomous strategy network unit; the autonomous strategy network unit receives the global perception spatiotemporal features, obtains the optimal action sequence according to the global perception spatiotemporal features, and transmits the optimal action sequence to the PD controller; the PD controller receives the optimal action sequence, generates control instructions according to the optimal action sequence, and controls the actions of the humanoid dexterous hand structure through the control instructions.

[0031] The image acquisition unit acquires the image set information of the operated object, including: placing three cameras above the humanoid dexterous hand structure; each camera acquires a pair of image information, and the three pairs of image information are combined into the image set information.

[0032] The pose estimation network unit obtains the pose information of the operated object according to the image set information, including: the ResNet50 network of the pose estimation network unit generates a feature map for each image information; the pose estimation network unit expands and splices the three feature maps together, and generates the pose information of the operated object after passing through the fully connected layer Linear.

[0033] The posture information of the operated object is obtained by the following formula:

[0034] pose = (x, y, z, α, β, γ);

[0035]

[0036] Among them, pose is the posture information of the operated object, flatten(·) is the matrix expansion operation, concat(·) is the matrix concatenation operation, x is the x-axis coordinate value of the object position in the three-dimensional space coordinate system, y is the y-axis coordinate value of the object position in the three-dimensional space coordinate system, z is the z-axis coordinate value of the object position in the three-dimensional space coordinate system, α is the angle of the object posture relative to the x-axis, β is the angle of the object posture relative to the y-axis, γ is the angle of the object posture relative to the z-axis, and Linear() is a series of fully connected layers. is the first feature map, is the second feature map, is the third feature map.

[0037] The global perception spatiotemporal features are obtained by fusing the posture information, pressure sensor information and joint angle information, including: fusing the pressure sensor information and joint angle information to obtain the dexterous hand proprioception features; fusing the dexterous hand proprioception features with the posture information to obtain the global perception spatiotemporal features.

[0038] The proprioceptive characteristics of the dexterous hand are obtained by the following formula:

[0039]

[0040] Among them, Z is the proprioceptive feature of the dexterous hand, angle is the joint angle information, and F is the pressure sensing information. is multiplication, G(angle,F) is a nested attention module, and Att() is an adaptive channel attention module.

[0041] like Figure 1 As shown in the figure, an autonomous operation training system for humanoid robot dexterous hands mainly includes a dexterous hand structure modeling unit, a parallel simulation training platform, a reinforcement learning training unit, an automatic domain randomization unit, a posture estimation unit, and a real dexterous hand experimental platform. Among them, the dexterous hand structure modeling unit is responsible for designing a virtual proportional dexterous hand simulation model according to the actual dexterous hand mechanical structure; the parallel simulation training platform provides a high-performance simulation environment for reinforcement learning research and supports parallel training of multiple models on the GPU; the reinforcement learning training unit is responsible for learning the spatiotemporal feature extraction model M from a large number of interactive data samples collected by the simulation training platform e , feature fusion model M f 、Strategy Model M p , and finally train to obtain an accurate autonomous operation model M; the automatic domain randomization unit is responsible for realizing the migration of the autonomous operation model from simulation to reality through physical randomization and non-physical randomization; the posture estimation unit is responsible for estimating the posture model M lThe accurate posture of the operated object is extracted from multi-angle images as the model input when the autonomous operation model is applied to the actual operation platform; the real dexterous hand experimental platform consists of a 9-degree-of-freedom manipulator, 16 FSR thin film pressure strain sensors, 3 cameras placed at different angles, PD controller hardware equipment, etc.

[0042] The following is combined with Figures 2 to 5 The present invention is described in further detail:

[0043] like Figure 2 As shown in Figure 2, the main steps of training dexterous hands for autonomous operation in a simulation environment through deep reinforcement learning based on complementary enhanced perception are as follows:

[0044] Step 1: Automatic domain randomization. To simplify the process of manually adjusting a large number of randomization settings and multiple iterations when migrating the model from simulation to reality, the automatic domain randomization method is used. First, define the distribution range of various physical factors and non-physical factors in the simulation environment, then automatically sample from this distribution and use it to generate training data. Evaluate the performance of the trained model on the current distribution, and decide the direction of updating the parameter distribution in the next step based on the evaluation results.

[0045] Step 2: Training the network for estimating the pose of the manipulated object. Use Isaac Sim’s Composer to generate a large amount of physically accurate 3D synthetic data of the manipulated object. Use the residual neural network as the backbone feature extraction network. The cameras placed at three different positions collect the image collection [img 1 ,img 2 ,img 3 ], the image set is used as the pose estimation network M l First, it passes through three ResNet50 networks with the same parameters to generate a feature map for each image.

[0046]

[0047]

[0048] Then, the three feature maps are flattened and spliced ​​together, and after a series of fully connected linear layers, the predicted position and posture of the controlled object are generated: pose = (x, y, z, α, β, γ):

[0049]

[0050] In formula (4), flatten(·) represents the matrix expansion operation, and concat(·) represents the matrix concatenation operation. This method generates large-scale training data with accurate annotations at low cost, while making the network robust to different camera parameters and visual effects that may appear in the scene.

[0051] Step 3: Multi-source information fusion and spatiotemporal feature extraction. Figure 3 The hierarchical attention recursive fusion method shown in the figure completes the controlled object pose, joint angle angle and binary pressure sensitive signal [F 1 ,F 2 ,...,F 16 Specifically, the whole fusion process can be described as the following steps:

[0052] Step 3.1: Align the original input data to solve the problem that the heterogeneous data collected by multiple sensors have different time and space distributions, and ensure that the time steps of different modal data are consistent.

[0053] Step 3.2, fuse the joint angle and binary pressure-sensitive signal to obtain the proprioceptive features of the dexterous hand. First, construct an adaptive channel attention module Att, which can aggregate the local and global contextual information of the joint angle angle and the pressure-sensitive signal F:

[0054]

[0055] In formula (6), represents the addition of corresponding elements, L(·) and g(·) are local channel context and global channel context respectively, Att(angle) and Att(F) are joint angle attention features and pressure sensitive signal attention features generated by the adaptive channel attention module respectively. Based on this module, the joint angle and pressure sensitive signal are adaptively fused to obtain the dexterous hand proprioception feature Z:

[0056]

[0057] In formula (7), To integrate the initial input features, another attention module is used to fuse the input features and perform iterative fusion of attention features. Specifically, G(angle,F) in formula (7) is:

[0058]

[0059] In step 3.3, the method described in step 3.2 is used to further fuse the proprioceptive feature Z of the dexterous hand with the pose of the controlled object to obtain the global perception feature U of the task. Finally, the feature vector U after hierarchical attention recursive fusion is passed through the convolutional neural network and recurrent neural network modules to extract the spatial and temporal correlation and obtain the global perception spatiotemporal feature.

[0060] Step 4: Proximal policy optimization. Take the multimodal global perception spatiotemporal features of step 3 as input and train the reinforcement policy network. The dexterous hand intelligent operation task is modeled as a partially observable Markov decision process I = (S, A, R, P) in discrete time, where S is the state space, A is the action space, R is the reward function, and P is the state transition matrix. In the dexterous hand intelligent operation task, the dexterous hand, as an intelligent agent, obtains the observed state s from the environment at time t. t , as shown in formula (9):

[0061] s t ={pose t ,angle t ,[F 1,t ,F 2,t ,...,F 3,t ]} (9)

[0062] The action a at the current moment is calculated through the strategy function π t , in this design, the action is defined as the control target value of the PD controller of each joint of the dexterous hand, as shown in formula (10):

[0063] a t =π(s t )=pose target (10)

[0064] Then, the reward function R(·) gives the reward value r at the current step t , as shown in formula (11):

[0065] r t =R(s t ,a t ,s t+1 ) (11)

[0066] During the whole process, the dexterous hand interacts with the environment as an intelligent agent to maximize the sum of the reward function R, as shown in formula (12), where γ∈[0,1] is the discount factor:

[0067]

[0068] Here we use a proximal policy optimization algorithm to learn a parameterized stochastic policy from observation S to action A, and a value function to score the policy.

[0069] Step 5: Iterative model update. During model training, the reward function R(·) is set as follows: a larger discrete reward is set when the manipulated object reaches the desired position; a continuous reward is set according to the rotation distance when the manipulated object approaches the desired position; different discrete penalties are set for situations where the current position is too far from the desired position and when the action changes too much. According to the reinforcement learning loss function L r The spatiotemporal feature extraction network M e , feature fusion network M f and proximal strategy optimization network M p Update the parameters according to the supervised learning loss function L s For the pose estimation network M l Update the parameters.

[0070] like Figure 4 As shown, the structure of the anthropomorphic dexterous hand includes: a thumb component 1, an index finger component 2, a middle finger component 3, a ring finger component 4, a little finger component 5, a FSR thin film pressure strain sensor 6, and a motor screw component 7.

[0071] like Figure 5 As shown in the figure, the autonomous operation model is finally deployed on the actual dexterous hand operation platform, and the specific control process of realizing autonomous operation is as follows: three cameras placed at different positions and angles above the dexterous hand operation platform collect images in real time during the experiment, and input the trained posture estimation network to obtain the posture information of the operated object in each frame; the FSR film pressure strain sensor 6 is connected to a microcontroller unit, which collects the analog voltage signal and converts it into a digital quantity and transmits it to the host, which binarizes the signal with a certain threshold and inputs it into the feature fusion and extraction network; the feature fusion and extraction network further processes the received posture information, pressure sensing information, and dexterous hand proprioception joint data information to obtain the fused multimodal features (global perception spatiotemporal features); the autonomous strategy network gives the optimal action sequence in the current state based on the multimodal features, that is, the control target value of the PD controller.

[0072] This embodiment also provides an electronic device, including: a memory: for storing computer-readable instructions; and a processor: for running the computer-readable instructions to execute an autonomous operation training method for a humanoid robot's dexterous hand.

[0073] This embodiment proposes a reinforcement learning algorithm that integrates a proximal strategy optimization model, a target segmentation model, and a multivariate spatiotemporal sequence feature extraction model, and trains an end-to-end network that can autonomously complete the intelligent operation tasks of dexterous hands; this embodiment constructs a hierarchical attention recursive fusion module, which iteratively integrates sensor information of different modalities through a multi-scale adaptive channel attention mechanism, which is more conducive to the fusion of multi-source heterogeneous inputs with inconsistent semantics and scales; this embodiment trains a posture estimation network, which can process image information collected from multiple angles in real time, obtain the position and posture of the controlled object, and provide accurate visual assistance for the fine operation of dexterous hands Information; This embodiment establishes a geometric simulation model for the actual mechanical structure of the dexterous hand applied to a humanoid robot, which solves the problems of difficulty in data collection of actual physical models and high trial and error costs; This embodiment introduces an automatic domain randomization method, which automatically defines physical factors such as mass and friction and non-physical factors such as Gaussian noise in the training environment in a course learning mode, and promotes the migration of algorithms from simulation to reality; This embodiment realizes a fully autonomous end-to-end control method applied to the dexterous hand of a humanoid robot. Compared with the traditional model-based control method, there is no need to obtain the physical characteristics of the operated object in advance, which avoids the construction of a complex hand dynamics model.

[0074] This embodiment takes into account the absolute spatial posture of the operated object and the multi-finger contact mechanical state representation of the dexterous hand, and proposes a hierarchical attention recursive fusion module for large-scale complex heterogeneous data. By gradually fusing the posture estimation information with the distributed pressure sensing spatiotemporal feature information, a globally consistent feature representation is obtained, complementary enhanced active perception is achieved, and the problem of robust intelligent control of dexterous hands in the absence of key modal information such as joint angles and object postures is solved.

[0075] Although the present invention has been disclosed as above in the form of a preferred embodiment, it is not intended to limit the present invention. Any person skilled in the art may make possible changes and modifications to the technical solution of the present invention by using the methods and technical contents disclosed above without departing from the spirit and scope of the present invention. Therefore, any simple modifications, equivalent changes and modifications made to the above embodiments based on the technical essence of the present invention without departing from the content of the technical solution of the present invention shall fall within the protection scope of the technical solution of the present invention.

Claims

1. An autonomous operation training system for humanoid robot dexterous hands, characterized in that include: Image acquisition unit, posture estimation network unit, humanoid dexterous hand structure, micro control unit, host, feature fusion and extraction network unit, autonomous strategy network unit and PD controller; Among them, The image acquisition unit acquires image set information of the operated object and inputs the image set information into the posture estimation network unit; The posture estimation network unit obtains posture information of the operated object according to the image set information, and transmits the posture information to the feature fusion and extraction network unit; The FSR thin film pressure strain sensor of the humanoid dexterous hand structure is connected to the micro control unit; The FSR thin film pressure strain sensor with the humanoid dexterous hand structure transmits the collected analog voltage signal to the micro control unit; The micro control unit receives the analog voltage signal, performs analog-to-digital conversion on the analog voltage signal to obtain a voltage digital signal, and transmits the voltage digital signal to the host; The host receives the voltage digital signal, performs binarization processing on the voltage digital signal according to a preset threshold value to obtain pressure sensing information, and transmits the pressure sensing information to the feature fusion and extraction network unit; The joint angle information of the humanoid dexterous hand structure is transmitted to the feature fusion and extraction network unit; The feature fusion and extraction network unit receives posture information, pressure sensor information and joint angle information, performs feature fusion on the posture information, pressure sensor information and joint angle information to obtain global perception spatiotemporal features, and transmits the global perception spatiotemporal features to the autonomous strategy network unit; The autonomous strategy network unit receives the global perception spatiotemporal features, obtains an optimal action sequence according to the global perception spatiotemporal features, and transmits the optimal action sequence to the PD controller; The PD controller receives the optimal action sequence, generates control instructions according to the optimal action sequence, and controls the action of the humanoid dexterous hand structure through the control instructions.

2. The autonomous operation training system for the dexterous hand of a humanoid robot according to claim 1, characterized in that: The image acquisition unit acquires image set information of the operated object including: Three cameras are placed above the humanoid dexterous hand structure; Each camera collects a set of image information, and the three sets of image information are combined into image set information.

3. The autonomous operation training system for the dexterous hand of a humanoid robot according to claim 2, characterized in that: The posture estimation network unit obtains the posture information of the operated object according to the image set information, including: The ResNet50 network of the pose estimation network unit generates a feature map for each image information; The posture estimation network unit expands and splices the three feature maps together, and generates the posture information of the operated object after passing through the fully connected layer Linear.

4. The autonomous operation training system for the dexterous hand of a humanoid robot according to claim 3 is characterized in that: The posture information of the operated object is obtained by the following formula: pose = (x, y, z, α, β, γ); Among them, pose is the posture information of the operated object, flatten(·) is the matrix expansion operation, concat(·) is the matrix concatenation operation, x is the x-axis coordinate value of the object position in the three-dimensional space coordinate system, y is the y-axis coordinate value of the object position in the three-dimensional space coordinate system, z is the z-axis coordinate value of the object position in the three-dimensional space coordinate system, α is the angle of the object posture relative to the x-axis, β is the angle of the object posture relative to the y-axis, γ is the angle of the object posture relative to the z-axis, Linear() is a series of fully connected layers, is the first feature map, is the second feature map, is the third feature map.

5. The autonomous operation training system for the dexterous hand of a humanoid robot according to claim 1, characterized in that: The global perception spatiotemporal features obtained by fusing posture information, pressure sensor information and joint angle information include: The pressure sensing information and joint angle information are fused to obtain the proprioceptive features of the dexterous hand; The proprioceptive features of the dexterous hand are fused with the posture information to obtain the global perception spatiotemporal features.

6. The autonomous operation training system for the dexterous hand of a humanoid robot according to claim 1, characterized in that: The proprioceptive characteristics of the dexterous hand are obtained by the following formula: Among them, Z is the proprioceptive feature of the dexterous hand, angle is the joint angle information, and F is the pressure sensing information. is multiplication, G(angle,F) is a nested attention module, and Att() is an adaptive channel attention module.

7. An autonomous operation training method for a humanoid robot's dexterous hand, characterized in that include: Collect image set information of the operated object and input the image set information into the pose estimation network; The pose estimation network obtains the pose information of the operated object based on the image set information and transmits the pose information to the feature fusion and extraction network; Collect analog voltage signals; The analog voltage signal is converted into a digital voltage signal; The voltage digital signal is binarized at a preset threshold to obtain pressure sensing information, and the pressure sensing information is transmitted to a feature fusion and extraction network; The joint angle information of the humanoid dexterous hand structure is transmitted to the feature fusion and extraction network; The feature fusion and extraction network receives the posture information, the pressure sensing information and the joint angle information, performs feature fusion on the posture information, the pressure sensing information and the joint angle information to obtain the global perception spatiotemporal features, and transmits the global perception spatiotemporal features to the autonomous strategy network; The autonomous policy network receives the global perception spatiotemporal features and obtains the optimal action sequence based on the global perception spatiotemporal features; Control instructions are generated according to the optimal action sequence, and the actions of the humanoid dexterous hand structure are controlled by the control instructions.

8. The autonomous operation training method for the dexterous hand of a humanoid robot according to claim 7, characterized in that: The image collection information of the operated object includes: Three cameras are placed above the humanoid dexterous hand structure; Each camera collects a set of image information, and the three sets of image information are combined into image set information.

9. The autonomous operation training method for the dexterous hand of a humanoid robot according to claim 8, characterized in that: The pose estimation network obtains the pose information of the operated object based on the image collection information, including: The ResNet50 network layer of the pose estimation network generates a feature map for each image information; The pose estimation network expands the three feature maps and splices them together, and generates the pose information of the operated object after passing through the fully connected Linear layer.

10. An electronic device, characterized in that: include: Memory: used to store computer-readable instructions; and Processor: used to run the computer-readable instructions to perform the method according to any one of claims 7 to 9.

Images