Dexterous manipulator integrated with body visual perception function

Abstract

The invention provides a dexterous manipulator fused with an ontology visual perception function, and belongs to the technical field of electromechanical control. The depth camera is arranged in the palm, the size of the device is remarkably reduced, the size of the device is closer to the size of a human hand, and therefore flexibility and naturalness are improved, the depth camera and a control system are highly integrated, the structure is simplified, synchronism of visual data and hand actions is improved, operation precision is improved, the dexterous manipulator has 16 degrees of freedom, and the flexibility and the reliability of the device are improved. The system is completely driven by a steering engine, the operation flexibility and the fine action execution capability are greatly improved, the requirements of complex tasks are met, the production and maintenance cost is effectively reduced through optimization design and reduction of external accessories, multiple objects are positioned at the same time through a developed instance-level multi-target positioning method and by using a neural network, pre-training is not needed, and the operation efficiency is improved. The space perception capability of the dexterous hand is further improved, and the autonomous operation capability of the dexterous hand in a complex environment is enhanced.

Description

Technical Field

[0001] The present invention relates to a dexterous manipulator integrating a body's visual perception function, and particularly belongs to the technical field of electromechanical control. Background Art

[0002] Dexterous hands, as mechanical devices that highly mimic the functions of human hands, play a vital role in modern robotics and automation. These devices often integrate multiple sensors and actuators to achieve complex operations and interactive tasks, such as fine object manipulation and advanced perception capabilities;

[0003] The defects of the existing technology are mainly reflected in the size and complexity, insufficient integration of visual perception, limited degrees of freedom and flexibility, and high cost. The existing dexterous hand designs generally use external cameras, which not only increases the overall size of the dexterous hand, but also affects the flexibility. The external camera requires additional brackets and mechanical structures to maintain stability, which also increases the complexity of the design and potential failure points. Although the miniature built-in cameras used in the existing technology can reduce the size and embed the camera inside the hand, they usually face resolution and performance limitations. It is difficult to develop related visual recognition algorithms, which limits the intelligence level of the dexterous hand. At present, most existing dexterous hands have between 6 and 10 degrees of freedom, which also limits their ability to perform complex or fine movements and cannot meet the needs of advanced operations. Existing dexterous hands often require a variety of expensive sensors and complex manufacturing processes, which increases costs and makes maintenance and troubleshooting more difficult. Summary of the invention

[0004] The purpose of the present invention is to provide a dexterous manipulator that integrates the proprioceptive visual perception function to ensure that the device can simultaneously locate multiple objects without pre-training, thereby improving the spatial perception of the dexterous manipulator and further enhancing its autonomous operation capability in complex environments.

[0005] In order to solve the above technical problems, the technical solution adopted by the present invention is: the invention includes a dexterous manipulator hand assembly and a controller, and also includes a depth camera; the dexterous manipulator hand assembly includes a thumb, index finger, middle finger and ring finger, the controller is fixed to the back of the dexterous manipulator hand assembly through a copper column, the controller is connected to the depth camera through a camera signal line, and the controller and the dexterous manipulator hand assembly driving servo are connected through the servo signal line for serial communication.

[0006] Furthermore, the dexterous manipulator assembly is controlled by the controller, and information is collected by the depth camera to drive the controller.

[0007] The thumb comprises a first side swing joint driven servo, a self-spin joint driven servo, a first bending joint driven servo and a second bending joint driven servo; the thumb has four degrees of freedom, one end of the first side swing joint driven servo is connected to a first connecting member, a second connecting member, a third connecting member and a fourth connecting member, the other ends of the first connecting member, the second connecting member, the third connecting member and the fourth connecting member are connected to a self-spin joint driven servo, one end of the self-spin joint driven servo is connected to a fifth connecting member and a sixth connecting member, the other ends of the fifth connecting member and the sixth connecting member are connected to the first bending joint driven servo, one end of the first bending joint driven servo is connected to an eighth connecting member and a ninth connecting member, the other ends of the eighth connecting member and the ninth connecting member are connected to the second bending joint driven servo, and one side of the first side swing joint driven servo, the self-spin joint driven servo, the first bending joint driven servo and the second bending joint driven servo is installed with a first baffle, a second baffle and a third baffle;

[0008] Furthermore, the connections between the thumb servos are double-ended to enhance the stability of the overall structure. The thumb side swing joint and bending joint imitate the joints of the human hand to enhance the anthropomorphic effect. The spin joint simulates the functionality of the human hand and enhances the operation function of the dexterous operator. The thumb connectors are connected by interference fit between bolts and holes, without nuts, reducing size and weight. Silicone parts are installed on the thumb baffle to ensure that the surface in contact with the object has elasticity and good friction, so that a better grip can be formed on the object.

[0009] The index finger includes a second side swing joint driving servo, a third bending joint driving servo, a fourth bending joint driving servo and a fifth bending joint driving servo; one end of the second side swing joint driving servo is connected to a ninth connecting member and a tenth connecting member, the other ends of the ninth connecting member and the tenth connecting member are connected to the third bending joint driving servo, one end of the third bending joint driving servo is connected to an eleventh connecting member and a twelfth connecting member, the other ends of the eleventh connecting member and the twelfth connecting member are connected to the fourth bending joint driving servo, one end of the fourth bending joint driving servo is connected to a thirteenth connecting member and a fourteenth connecting member, the other ends of the thirteenth connecting member and the fourteenth connecting member are connected to the fifth bending joint driving servo, and a fourth baffle, a fifth baffle, a sixth baffle, a seventh baffle and an eighth baffle are installed on one side of the second side swing joint driving servo, the third bending joint driving servo, the fourth bending joint driving servo and the fifth bending joint driving servo;

[0010] The structures of the index finger, middle finger and ring finger are the same, and all of them have four degrees of freedom;

[0011] Furthermore, the connections between the finger servos are all double-ended to enhance the stability of the overall structure. The mechanical limit angles of the three bending joints of the fingers are all 90°. Silicone parts are added to the baffles of the fingers to ensure that the surface in contact with the object has elasticity and good friction, so that the object can be better grasped.

[0012] The palm is used to connect the fingers and provide support for the grasping action. The palm of the dexterous hand is designed to imitate anthropomorphism, so that the palm is not a simple plane, but has a certain curved shape, which further improves the anthropomorphism of the dexterous hand. The whole palm is composed of two parts, the back of the hand and the front of the hand, which is easy to install. The side swing joints of the four fingers are installed on the back of the hand part. The front hand part is divided into a front grasping area and a rear functional area. The front grasping area of ​​the front hand part is the area where the dexterous hand contacts the object when grasping the object. The left side of the rear functional area is grooved for the side swing joint movement of the thumb, and the right side adopts a curved shape to improve the aesthetics of the dexterous hand. Two small slots are opened on the right side for the depth camera to recognize objects. The back of the hand part is connected to the front hand part by eight bolts and connected to the controller by four bolts, making the structure of the whole hand simple and easy to install and maintain.

[0013] The dexterous manipulator hand assembly also includes a hand surface part and a hand back part; the hand surface part is installed at the wrist position of the dexterous manipulator hand assembly, and the hand back part is installed at the hand back position of the dexterous manipulator hand assembly. The side swing joints of the thumb, index finger, middle finger and ring finger are all installed on the hand back part. The hand surface part is divided into a front gripping area and a rear functional area.

[0014] Furthermore, the hand surface parts and the hand back parts can be used to add connectors or make further modifications on the part plane, such as adding connectors for connecting to a robotic arm, so that the dexterous manipulator can be deployed on a suitable platform to work.

[0015] The dexterous manipulator assembly has a thumb mounting position, an index finger mounting position, a middle finger mounting position, a ring finger mounting position and a depth camera mounting position fixedly installed on the back of the hand; the thumb is mounted and fixed via the thumb mounting position, the index finger is mounted and fixed via the index finger mounting position, the middle finger is mounted and fixed via the middle finger mounting position, the ring finger is mounted and fixed via the ring finger mounting position, and the depth camera is mounted and fixed via the depth camera mounting position;

[0016] One end of the hand surface part and the back of the hand part is connected to a robotic arm, and the other end of the robotic arm is fixedly installed with a desk. A box is fixedly set on the top of the desk, and a target object is placed on the top of the box. The dexterous manipulator assembly cooperates with the controller to complete the grasping of the target object.

[0017] Furthermore, the images captured by the depth camera on the dexterous manipulator assembly are analyzed through the visual recognition algorithm in the controller; the visual recognition algorithm identifies the type, size and spatial position of the target object, and the robotic arm moves to the side of the target object according to the results of the solution, thereby completing the preliminary positioning of the target object.

[0018] The beneficial effects of the present invention are:

[0019] 1. By embedding the depth camera into the palm of the hand, the overall size of the device is significantly reduced, making its appearance and functions closer to the size of a human hand, thereby improving the flexibility and portability of dexterous hands. It is particularly suitable for delicate operations and space-constrained application scenarios, meeting the demand for compact and efficient design.

[0020] 2. By directly integrating the depth camera with the dexterous hand control system, the mechanical structure of the equipment is simplified, the synchronization between visual data and hand movements is improved, and higher operating accuracy and stability are achieved. The system enhances the spatial perception and target recognition capabilities of the dexterous hand through the depth perception function.

[0021] 3. The dexterous manipulator assembly is driven entirely by the servos, so that the dexterous manipulator assembly has a 16-degree-of-freedom setting, which greatly improves the dexterous manipulator's operational flexibility and the ability to perform complex movements. It can simulate human hand movements more accurately, thereby meeting higher-level task requirements.

[0022] 4. Through the design of the optimized structure of the dexterous manipulator component and the design of the key functional modules integrated in the controller, the dependence on expensive sensors and complex manufacturing processes is effectively reduced, the production and maintenance costs are significantly reduced, and the external accessories and potential failure points are reduced, making the maintenance of the equipment simpler and more convenient.

[0023] 5. A general instance-level multi-target localization method was developed through the cooperation of the dexterous manipulator component and the controller. Multiple target objects can be located simultaneously using a single neural network without the need for pre-training for each object. This improves the spatial perception ability of the dexterous manipulator, enabling it to achieve autonomous operation and precise target positioning in complex environments, further enhancing the intelligence level of the equipment. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 It is a schematic diagram of the three-dimensional structure of the dexterous manipulator assembly of the present invention;

[0025] Figure 2 It is a front view schematic diagram of the structure of the dexterous manipulator assembly of the present invention;

[0026] Figure 3 is a schematic diagram of the three-dimensional structure of the thumb of the present invention;

[0027] Figure 4 It is a schematic diagram of a three-dimensional structure in the present invention;

[0028] Figure 5 It is a schematic diagram of the three-dimensional structure of the hand surface parts of the present invention;

[0029] Figure 6 It is a schematic diagram of the three-dimensional structure of the hand back part of the present invention;

[0030] Figure 7 It is a schematic diagram of the overall three-dimensional structure of the present invention;

[0031] Figure 8 It is a schematic diagram of the three-dimensional structure of the mechanical arm adjustment of the present invention;

[0032] Fig. 9 It is a schematic diagram of the three-dimensional structure grasped by the dexterous manipulator assembly of the present invention.

[0033] 1. Dexterous manipulator assembly; 1.1. Thumb; 1.1.1. First side swing joint drives servo; 1.1.2. First connecting piece; 1.1.3. Second connecting piece; 1.1.4. Third connecting piece; 1.1.5. Fourth connecting piece; 1.1.6. Fifth connecting piece; 1.1.7. Spin joint drives servo; 1.1.8. Sixth connecting piece; 1.1.9. First bending joint drives servo; 1.1.10. Eighth connecting piece; 1.1.11. Ninth connecting piece; 1.1.12. First baffle; 1.1.13. Second bending joint drives servo; 1.1.14. Second baffle; 1.1.15. Third baffle; 1.2. Index finger; 1.3. Middle finger; 1.3.1. Second side swing joint drives servo; 1.3.2. Ninth connecting piece; 1.3.3. Tenth connecting piece; 1.3.4. Third bending joint Driving servo; 1.3.5, eleventh connecting piece; 1.3.6, twelfth connecting piece; 1.3.7, fourth baffle; 1.3.8, fourth bending joint driving servo; 1.3.9, fifth baffle; 1.3.10, sixth baffle; 1.3.11, thirteenth connecting piece; 1.3.12, fourteenth connecting piece; 1.3.13, fifth bending joint driving servo; 1.3.14, seventh baffle; 1.3.15, eighth baffle; 1.4, ring finger; 1.5, depth camera; 1.6, hand parts; 1.7, back of hand parts; 1.7.1, thumb installation position; 1.7.2, index finger installation position; 1.7.3, middle finger installation position; 1.7.4, ring finger installation position; 1.7.5, depth camera installation position; 2, controller; 3, robotic arm; 4, target object; 5, box; 6, desk. DETAILED DESCRIPTION

[0034] The following will be combined with the attached Figure 1-9 , clearly and completely describe the technical solutions in the embodiments.

[0035] Specific implementation method 1: Figure 1-2As shown, the dexterous manipulator as a whole includes a dexterous manipulator component 1 and a controller 2. The dexterous manipulator component 1 as a whole has 16 active degrees of freedom. The controller 2 is fixed to the back of the dexterous manipulator component 1 through a copper column, and the depth camera 1.5 is built-in and fixedly installed on the palm of the dexterous manipulator component 1, thereby significantly reducing the overall volume of the device, making its appearance and function closer to the size of a human hand, thereby improving the flexibility and portability of the dexterous manipulator component 1, and is particularly suitable for delicate operations and space-constrained application scenarios, meeting the needs for compact and efficient design;

[0036] The controller 2 is connected to the depth camera 1.5 through a camera signal line, and the controller 2 and the dexterous manipulator assembly 1 drive the servo through the servo signal line for serial communication; the system integration method of the depth camera 1.5 directly controlling the system with the dexterous manipulator assembly 1 simplifies the mechanical structure of the equipment, improves the synchronization of visual data and hand movements, and achieves higher operation accuracy and stability. The system enhances the spatial perception and target recognition capabilities of the dexterous manipulator assembly 1 through the depth perception function;

[0037] The controller 2 runs a visual recognition algorithm, uses the image data collected by the depth camera 1.5 to detect and locate the target object, and based on the deep learning model, processes the collected three-dimensional point cloud and RGB image to identify the type, size and spatial position of the target object, and generates the three-dimensional coordinate information of the target object;

[0038] Controller 2 can monitor the execution status of each servo in real time to ensure the accuracy of the grasping action. During the execution process, the real-time position, speed and other information of the servo are fed back to the user through the display interface of controller 2. The user can further adjust the position and speed parameters of each servo according to the feedback information. Controller 2 has an open design to support the expansion of target detection algorithms and servo control logic to adapt to different grasping tasks and operation requirements. It can also add a detection function for the target object category or adjust the servo position control mode to meet specific application scenarios.

[0039] The visual recognition algorithm run by controller 2 is efficient and flexible, and can handle target detection tasks in a variety of complex scenarios, including different lighting conditions and background environments. During the operation of the algorithm, it provides accurate target position data by comprehensively analyzing the multi-dimensional information of the target object; the control algorithm of controller 2 movement supports the mode of manual input of the servo position by the user to maximize the operational flexibility and safety of the system, which is particularly suitable for scenarios that require precise adjustment. For example, when handling fragile objects, the user can gradually adjust the position of the servo to ensure the stability and safety of the grasping action;

[0040] Specific implementation method 2: Figure 3-6As shown, the thumb 1.1 includes four sets of joint servo drives, namely, a first side swing joint driving servo 1.1.1, a self-spin joint driving servo 1.1.7, a first bending joint driving servo 1.1.9 and a second bending joint driving servo 1.1.13, so that the thumb 1.1 has four degrees of freedom. The thumb is fixedly mounted on the back of the hand of the dexterous manipulator assembly 1 through the first side swing joint driving servo 1.1.1. The mechanical limit rotation angle of the first side swing joint driving servo 1.1.1 is 90°. The first side swing joint driving servo 1.1.1 and the self-spin joint driving servo 1.1.7 are connected by a first connecting member 1.1.2 and a second connecting member 1. 1.3, the third connecting piece 1.1.4 and the fourth connecting piece 1.1.5 are connected by four sets of connecting pieces, the mechanical limit rotation angle of the spinning joint driving servo 1.1.7 is 60°, the spinning joint driving servo 1.1.7 and the first bending joint driving servo 1.1.9 are connected by the fifth connecting piece 1.1.6 and the sixth connecting piece 1.1.8, the first bending joint driving servo 1.1.9 and the second bending joint driving servo 1.1.13 are connected by the eighth connecting piece 1.1.10 and the ninth connecting piece 1.1.11, the first bending joint driving servo 1.1. The mechanical limit rotation angles of the first side swing joint driving servo 1.1.1, the spin joint driving servo 1.1.7, the first bending joint driving servo 1.1.9 and the second bending joint driving servo 1.1.13 are all 90°. The first baffle 1.1.12, the second baffle 1.1.14 and the third baffle 1.1.15 are installed on the four driving servos of the first side swing joint driving servo 1.1.1, the spin joint driving servo 1.1.7, the first bending joint driving servo 1.1.9 and the second bending joint driving servo 1.1.13. The contact surfaces of the first baffle 1.1.12, the second baffle 1.1.14 and the third baffle 1.1.15 with the object are flat, so as to enhance the anthropomorphic effect and facilitate grasping; the connections between the four driving servos of the thumb 1.1 are double-ended The thumb 1.1 is connected to the ground, thereby enhancing the stability of the overall structure; the first side swing joint driving servo 1.1.1, the first bending joint driving servo 1.1.9 and the second bending joint driving servo 1.1.13 of the thumb 1.1 imitate the joints of the human hand to enhance the anthropomorphic effect; the spin joint driving servo 1.1.7 realizes the functional simulation of the human hand and enhances the operation function of the dexterous manipulator; the connecting parts of the thumb 1.1 are connected by the interference fit of the bolts and the holes, without the need for nuts, to reduce the size and weight; silicone parts are installed on the three sets of baffles of the thumb 1.1 to ensure that the surface in contact with the object has elasticity and good friction, so as to form a better grip on the object;

[0041] The mechanical limit of the lateral swing joint of the index finger 1.2 and the ring finger 1.4 is 45°, the mechanical limit of the lateral swing joint of the middle finger 1.3 is 30°, and the axis of the index finger 1.2 and the ring finger 1.4 has an angle of 5° with the axis of the middle finger 1.3. Since the structures of the index finger 1.2, the middle finger 1.3 and the ring finger 1.4 are the same, the middle finger 1.3 is used as an example;

[0042] The middle finger 1.3 has four degrees of freedom, and the middle finger 1.3 is composed of a second side swing joint driven servo 1.3.1, a third bending joint driven servo 1.3.4, a fourth bending joint driven servo 1.3.8 and a fifth bending joint driven servo 1.3.13 from the root to the fingertip; the middle finger 1.3 is fixedly installed on the back of the hand of the dexterous operating hand assembly 1 through the second side swing joint driven servo 1.3.1, the second side swing joint driven servo 1.3.1 and the third bending joint driven servo 1.3.4 are connected by two sets of connecting parts, the ninth connecting part 1.3.2 and the tenth connecting part 1.3.3, the third bending joint driven servo 1.3.4 adopts an upright installation method different from other servos, so as to reduce the overall length of the finger, the third bending joint driven servo 1.3.4 and the fourth bending joint driven servo 1.3.8 are connected by the eleventh connecting part 1.3. The connecting piece 1.3.5 and the twelfth connecting piece 1.3.6 are connected by cooperation, the fourth bending joint driving servo 1.3.8 and the fifth bending joint driving servo 1.3.13 are connected by cooperation of the thirteenth connecting piece 1.3.11 and the fourteenth connecting piece 1.3.12, the second side swing joint driving servo 1.3.1, the third bending joint driving servo 1.3.4, the fourth bending joint driving servo 1.3.8 and the fifth bending joint driving servo 1.3.13 are equipped with a fourth baffle 1.3.7, a fifth baffle 1.3.9, a sixth baffle 1.3.10, a seventh baffle 1.3.14 and an eighth baffle 1.3.15, and the contact surface of the fourth baffle 1.3.7, the fifth baffle 1.3.9, the sixth baffle 1.3.10, the seventh baffle 1.3.14 and the eighth baffle 1.3.15 with the object is a plane;

[0043] In order to enhance the anthropomorphic effect and facilitate grasping, the connections between the four sets of driving servos are all double-end connections, thereby enhancing the stability of the overall structure. The mechanical limit angles of the three bending joints of the finger's third bending joint driving servo 1.3.4, the fourth bending joint driving servo 1.3.8 and the fifth bending joint driving servo 1.3.13 are all 90°. Silicone parts are added to the baffles of the fingers to ensure that the surface in contact with the object has elasticity and good friction, so that the object can be better grasped;

[0044] By setting each joint to be driven by a servo, the size of the dexterous manipulator 1 is close to that of a human hand, and all parts are prepared by 3D printing, with a simple structure, easy to manufacture and assemble, and convenient and quick maintenance, thus greatly reducing costs. The servos of the dexterous hand 1 are connected by servo connecting wires, which pass through the slots on the connectors and are fixed inside the baffle parts without being exposed, making it beautiful and not affecting the grasping.

[0045] The palm part of the dexterous manipulator assembly 1 is composed of a hand surface part 1.6 and a hand back part 1.7. The palm is designed to imitate human beings, so that the palm is not a simple plane, but has a certain curved surface shape, which further improves the anthropomorphism of the dexterous hand. The composition of the hand surface part 1.6 and the hand back part 1.7 makes the overall structure of the dexterous manipulator assembly 1 more convenient to install. The side swing joints of the four fingers are installed on the hand back part 1.7. The hand surface part 1.6 is divided into a front gripping area and a rear functional area. The front gripping area of ​​the hand surface part 1.6 is the area where the dexterous hand contacts the object when grasping the object. The left side of the rear functional area is grooved for the side swing joint movement of the thumb 1.1. The right side adopts a curved surface shape to improve the aesthetics of the dexterous hand 1. Two small slots are opened on the right side for the depth camera to recognize objects.

[0046] The back of hand part 1.7 includes a thumb mounting position 1.7.1, an index finger mounting position 1.7.2, a middle finger mounting position 1.7.3, a ring finger mounting position 1.7.4 and a depth camera mounting position 1.7.5; the thumb 1.1 is mounted and fixed through the thumb mounting position 1.7.1, the index finger 1.2 is mounted and fixed through the index finger mounting position 1.7.2, the middle finger 1.3 is mounted and fixed through the middle finger mounting position 1.7.3, the ring finger 1.4 is mounted and fixed through the ring finger mounting position 1.7.4, and the depth camera 1.5 is mounted and fixed through the depth camera mounting position 1.7.5; the back of hand part 1.7 is connected to the hand surface part 1.6 through eight bolts, and the back of hand part 1.7 is connected to the controller 2 through four bolts. This makes the overall dexterous manipulation hand assembly 1 simple in structure and easy to install and maintain;

[0047] Specific implementation method three: Figure 7-9 As shown, the hand face part 1.6 and the hand back part 1.7 are respectively located at the wrist and the back of the dexterous manipulator assembly 1, so as to be used for adding connectors or making further modifications on the plane of the parts, such as adding connectors for connecting the robot arm 3, so as to realize the deployment of the dexterous manipulator assembly 1 on a suitable platform for working;

[0048] One end of the hand surface part 1.6 and the back of the hand part 1.7 are connected to a mechanical arm 3, and the other end of the mechanical arm 3 is fixedly installed with a desk 6, and a box 5 is fixedly set on the top surface of the desk 6. The top surface of the box 5 is placed with a target object 4. The dexterous operating hand assembly 1 cooperates with the controller 2 to complete the grasping of the target object 4;

[0049] The control system of the dexterous manipulator specifically comprises the following steps:

[0050] Step 1: Analyze the image captured by the depth camera 1.5 on the dexterous manipulator assembly 1 through the visual recognition algorithm in the controller 2. The visual recognition algorithm identifies the type, size and spatial position of the target object 4 and generates the three-dimensional coordinate information of the target object 4;

[0051] Step 2: The controller 2 transmits the target position data to the control module of the robot arm 3 according to the acquired spatial position data of the target object 4. The control module of the robot arm 3 performs kinematics calculation according to the target position data to obtain the position parameters to which each joint of the robot arm 3 should move;

[0052] Step 3: The robot arm 3 moves to the side of the target object 4 according to the calculated result, and completes the preliminary positioning of the target object 4. The controller 2 monitors the motion state of the robot arm 3 in real time and provides real-time feedback information on the display interface so that the user can confirm the correctness of the positioning result.

[0053] Step 4: After the robot arm 3 moves to the side of the target object, the user inputs the target position command of the servo through the controller 2 to control the movement of each servo of the dexterous hand 1;

[0054] Step 5: The controller 2 transmits the servo position command input by the user to each servo of the dexterous manipulator assembly 1 through serial communication, and reads the position information of each servo in real time. The controller 2 feeds back the real-time position information of the servo to the user to confirm the accuracy of the grasping posture;

[0055] Step 6: The user can further adjust the servo position according to the feedback information until the grasping posture is correct. When the posture adjustment of the dexterous manipulator assembly 1 is completed and the grasping posture is confirmed to be correct, the controller 2 executes the final grasping instruction and drives the dexterous manipulator assembly 1 to grasp the target object;

[0056] Step 7: After the grasping action is completed, the controller 2 reads the position information of each servo again, confirms that the dexterous manipulator assembly 1 maintains a stable grasping posture, and feeds back the grasping result on the display interface.

[0057] The above is only a preferred embodiment of the present invention and does not limit the present invention in any form. Although the present invention has been disclosed as a preferred embodiment as above, it is not used to limit the present invention. Any technician familiar with this profession can make some changes or modify the technical contents disclosed above into equivalent embodiments without departing from the scope of the technical solution of the present invention. However, any simple modification, equivalent replacement and improvement made to the above embodiments without departing from the content of the technical solution of the present invention, based on the technical essence of the present invention, within the spirit and principles of the present invention, still fall within the protection scope of the technical solution of the present invention.

Claims

1. A dexterous manipulator integrating a body's visual perception function, comprising a dexterous manipulator component (1) and a controller (2), characterized in that: Also includes a depth camera (1.5); The dexterous manipulator component (1) comprises a thumb (1.1), an index finger (1.2), a middle finger (1.3) and a ring finger (1.4); a controller (2) is fixed to the back of the dexterous manipulator component (1) via a copper column; the controller (2) is connected to a depth camera (1.5) via a camera signal line; and the controller (2) and a driving servo of the dexterous manipulator component (1) are connected via a servo signal line for serial communication.

2. The dexterous manipulator integrating the proprioceptive visual perception function according to claim 1 is characterized in that: The thumb (1.1) comprises a first side swing joint driving servo (1.1.1), a spin joint driving servo (1.1.7), a first bending joint driving servo (1.1.9) and a second bending joint driving servo (1.1.13); The thumb (1.1) has four degrees of freedom, one end of the first side swing joint driving servo (1.1.1) is connected to a first connecting piece (1.1.2), a second connecting piece (1.1.3), a third connecting piece (1.1.4) and a fourth connecting piece (1.1.5), the other ends of the first connecting piece (1.1.2), the second connecting piece (1.1.3), the third connecting piece (1.1.4) and the fourth connecting piece (1.1.5) are connected to a spin joint driving servo (1.1.7), one end of the spin joint driving servo (1.1.7) is connected to a fifth connecting piece (1.1.6) and a sixth connecting piece (1.1.8), the other ends of the fifth connecting piece (1.1.6) and the sixth connecting piece (1.1.8) are connected to the first connecting piece (1.1.2), the second connecting piece (1.1.3), the third connecting piece (1.1.4) and the fourth connecting piece (1.1.5) are connected to the spin joint driving servo (1.1.7), A first bending joint driving servo (1.1.9) is connected; one end of the first bending joint driving servo (1.1.9) is connected to an eighth connecting piece (1.1.10) and a ninth connecting piece (1.1.11); the other ends of the eighth connecting piece (1.1.10) and the ninth connecting piece (1.1.11) are connected to a second bending joint driving servo (1.1.13); a first baffle (1.1.12), a second baffle (1.1.14) and a third baffle (1.1.15) are installed on one side of the first side swing joint driving servo (1.1.1), the spin joint driving servo (1.1.7), the first bending joint driving servo (1.1.9) and the second bending joint driving servo (1.1.13).

3. The dexterous manipulator integrating the proprioceptive visual perception function according to claim 1 is characterized in that The middle finger (1.3) includes a second side swing joint driving servo (1.3.1), a third bending joint driving servo (1.3.4), a fourth bending joint driving servo (1.3.8) and a fifth bending joint driving servo (1.3.13); One end of the second side swing joint driving servo (1.3.1) is connected to the ninth connecting member (1.3.2) and the tenth connecting member (1.3.3), the other ends of the ninth connecting member (1.3.2) and the tenth connecting member (1.3.3) are connected to the third bending joint driving servo (1.3.4), one end of the third bending joint driving servo (1.3.4) is connected to the eleventh connecting member (1.3.5) and the twelfth connecting member (1.3.6), the other ends of the eleventh connecting member (1.3.5) and the twelfth connecting member (1.3.6) are connected to the fourth bending joint driving servo (1.3.8), one end of the fourth bending joint driving servo (1.3.8) is connected to the tenth connecting member (1.3.4), and the other ends of the eleventh connecting member (1.3.5) and the twelfth connecting member (1.3.6) are connected to the fourth bending joint driving servo (1.3.8). The third connecting piece (1.3.11) and the fourteenth connecting piece (1.3.12), the other ends of the thirteenth connecting piece (1.3.11) and the fourteenth connecting piece (1.3.12) are connected with the fifth bending joint driving steering gear (1.3.13), and the second side swing joint driving steering gear (1.3.1), the third bending joint driving steering gear (1.3.4), the fourth bending joint driving steering gear (1.3.8) and the fifth bending joint driving steering gear (1.3.13) are installed with the fourth baffle (1.3.7), the fifth baffle (1.3.9), the sixth baffle (1.3.10), the seventh baffle (1.3.14) and the eighth baffle (1.3.15) on one side.

4. The dexterous manipulator integrating the proprioceptive visual perception function according to claim 1 is characterized in that: The structures of the index finger (1.2), the middle finger (1.3) and the ring finger (1.4) are identically configured, and the index finger (1.2), the middle finger (1.3) and the ring finger (1.4) all have four degrees of freedom.

5. The dexterous manipulator integrating the proprioceptive visual perception function according to claim 1 is characterized in that: The dexterous manipulator hand assembly (1) further comprises a hand surface part (1.6) and a hand back part (1.7); A hand surface part (1.6) is installed at the wrist position of the dexterous operating hand assembly (1), and a hand back part (1.7) is installed at the hand back position of the dexterous operating hand assembly (1). The hand surface part (1.6) is divided into a front gripping area and a rear functional area.

6. The dexterous manipulator integrating proprioceptive visual perception function according to claim 5, characterized in that: The back of hand part (1.7) includes a thumb mounting position (1.7.1), an index finger mounting position (1.7.2), a middle finger mounting position (1.7.3), a ring finger mounting position (1.7.4) and a depth camera mounting position (1.7.5); The thumb (1.1) is mounted and fixed via the thumb mounting position (1.7.1), the index finger (1.2) is mounted and fixed via the index finger mounting position (1.7.2), the middle finger (1.3) is mounted and fixed via the middle finger mounting position (1.7.3), the ring finger (1.4) is mounted and fixed via the ring finger mounting position (1.7.4), and the depth camera (1.5) is mounted and fixed via the depth camera mounting position (1.7.5).

7. The dexterous manipulator integrating proprioceptive visual perception function according to claim 1 is characterized in that: The back of hand part (1.7) is connected to the surface of the hand part (1.6) via eight bolts, and the back of hand part (1.7) is connected to the controller (2) via four bolts.

8. The dexterous manipulator integrating the proprioceptive visual perception function according to claim 1 is characterized in that: One end of the hand surface part (1.6) and the back of the hand part (1.7) is connected to a mechanical arm (3), and the other end of the mechanical arm (3) is fixedly mounted with a desk (6). A box (5) is fixedly arranged on the top surface of the desk (6), and a target object (4) is placed on the top surface of the box (5). The dexterous operating hand assembly (1) cooperates with the controller (2) to complete the grasping of the target object (4).

Images